Determine if system shutdown was clean

[johnlocke]

Hi.

I cannot find a way to see if the system shutdown was clean or not. The system is on UPS, NUT is configured to shut it down. I was away when power was lost and found TrueNAS in off state.
/var/log/messages contains only last boot logs, there is nothing from previous time.
/var/log/nut/ups.log is the same, only today's boot
dmesg -a starts with ---<<BOOT>>--- and nothing before it.

How do I tell for sure that system shut down nicely? How to make /var/log/messages contain data for last 7 days, for example?

Cheers

 

[artlessknave]

did you have an alert that the shutdown was unexpected? if you don't. then the shutdown was expected.
also, hardware is part of the forum rules.

 

[johnlocke]

Thanks.
a) If shutdown was unexpected, where and when should I've seen the alert?
b) I did not see any alerts. Why there is nothing in logs about last shutdown and shutdown reason?

Hardware is Odroid H2+, Intel J4115.

 

[artlessknave]

the alert will be literally be an alert when you log in, like the alerts for everything else, like bad disks, failed jobs, etc. its very hard to miss, by design.

I don't know that there is a way to set a shutdown reason. you could put in an "on startup" "on shutdown" script, i guess? have it put an entry into your own log file? something like:
"on shutdown"; echo shutdown server @ `date` > /root/powerstatus.log

 

[johnlocke]

"Shutting down on UPS power loss" should be an alert too, I think. I have no alerts whatsoever in GUI, except three old "Scrub of pool finished". Something is not right here.

 

[artlessknave]

I do not believe the UPS subsystem in use (NUT?) has that ability. if you want that, I think you have to make it.
only a failed shutdown matters. if the power died, and it shutdown normally anyway, then it's literally functioning as it should, as that can ONLY happen if the UPS shut it down cleanly.
also, even having that isn't always ideal, since if it shuts it down, it probably wont power itself back on.
a UPS is more about staying on during brownouts or short power loss, and cleaning the line power during those problems. power loss doesn't really affect ZFS, the way it works is so designed around data integrity that you have to REALLY work at it to break it. people have tried to corrupt pools by messing around with power and failed.
it seems to me like you are just stressing yourself out over a non issue, but, again, if you really want it, you should be able to make a few scripts

 

[johnlocke]

I do not believe the UPS subsystem in use (NUT?) has that ability.

Not sure what ability you meant. In Services / UPS I have configured
Shutdown Mode: UPS goes on battery
Shutdown Timer: 120
And it worked last time I tested it. I just re-tested, cut power to UPS, got alert in GUI "ups is on battery power", system shut down itself in 2mins.

 

[johnlocke]

But on reboot, I do not see this alert anymore! I, again, only see three old "Scrub of pool finished" ones.
Same goes for dmesg and /var/log/messages - no logs around shutdown, only last boot.

Why the "ups is on battery power" alert disappeared? How can see it again? Something like Alert History.

In short, there no way to see what previously happened to the box, power wise. All logs and alerts get cleaned on boot.

 

[artlessknave]

if you have no alert for an unscheduled shutdown...it means it shutdown correctly, whether that was because you shut it down, or because the UPS scripts triggered and shut it down. as far as TrueNAS is concerned, nothing is wrong, everything is working as intended (TM), and so there are no alerts, because nothing is wrong.

you will see alerts for UPS is on battery power only while it is on battery power. again, dont think there is a log for that.

I just remembered that you can configure email alerts. you will get the alert when the condition triggers and when it clears. it can get very annoying very fast, but maybe this is what you are looking for?

 

[johnlocke]

I do have email alerts, thing is, when they cut power to my house, emails stop working too :)

I believe having Alert History is very useful feature and needs to be added.
Also, is there a way to not erasing /var/log/messages on every boot? Log rotation?

 

[artlessknave]

/var/log/messages is not reset on boot. when it gets over a certain size, its rotated to an achive iirc. messages.tz1, messages.tz2, etc.
a boot does tend to send a lot to messages, and can roll it over quickly if you are rebooting often.

I don't see it getting much traction, but you certainly can submit a feature request. nobody else has chimed in and I can't think of anything else to contribute, as I just don't see it being useful. if the power dies, email dies, but you have no bad shutdown alert on startup, then nothing is wrong, it shut down as programmed, beyond maybe getting an ARC generator for infinite power?

 

[johnlocke]

/var/log/messages is not reset on boot. when it gets over a certain size, its rotated to an achive iirc. messages.tz1

I'll need to find more about it. /var/log contains only messages, no messages.tz1 or messages.0.bz2 ...

I don't see it getting much traction

I searched before posting, and saw quite a few threads with "why my system shuts down" or "how to troubleshoot shutdowns". There was no clear way to get whats going on in ones that I read.

 

[johnlocke]

/var/log/messages is not reset on boot. when it gets over a certain size, its rotated to an achive iirc. messages.tz1, messages.tz2, etc.

I'm trying to dig in. It looks like /var/log/messages gets erased on my system every reboot. I run :

# cat /var/log/messages Dec 21 17:14:51 mysystem newsyslog[1579]: logfile first created Dec 21 17:14:51 mysystem syslog-ng[1689]: syslog-ng starting up; version='3.35.1' Dec 21 17:14:51 mysystem ---<<BOOT>>--- ... # reboot (login again) # cat /var/log/messages Dec 21 23:23:45 mysystem newsyslog[1579]: logfile first created Dec 21 23:23:46 mysystem syslog-ng[1689]: syslog-ng starting up; version='3.35.1' Dec 21 23:23:46 mysystem Waiting (max 60 seconds) for system process `vnlru' to stop... done Dec 21 23:23:46 mysystem Waiting (max 60 seconds) for system process `syncer' to stop... Dec 21 23:23:46 mysystem Syncing disks, vnodes remaining... 0 0 0 0 0 0 0 0 done [COLOR=rgb(20, 20, 20)][/COLOR]

No signs of events from previous boot (Dec 21 17:14:51). How could that be, please?
What is the meaning of "newsyslog[1579]: logfile first created" - did it just erased existing messages file?

Hmm ... all the files in /var/log are fresh-after-reboot files ... as if they were recreated from scratch. Is it about system dataset location (boot-pool, eMMC drive)?

 

[Patrick M. Hausen]

The last command tells you if the last shutdowns/reboots etc. were clean or not. A typical output here looks like this:

Code:

ry93@truenas-ka:~ $ last
ry93       pts/4    2003:a:827:2a00:a4a6:8 Wed Dec 21 13:59   still logged in
[...]
boot time                                  Thu Dec  8 09:32
shutdown time                              Thu Dec  8 09:29
[...]
boot time                                  Thu Dec  8 09:06
shutdown time                              Thu Dec  8 09:03
[...]
boot time                                  Thu Dec  8 08:35
[...]
boot time                                  Wed Dec  7 17:03
shutdown time                              Wed Dec  7 17:00
boot time                                  Wed Dec  7 16:39

utx.log begins Wed Dec  7 16:39:14 CET 2022

 

[johnlocke]

OK :) The fix was to check System / System Dataset / Syslog checkbox. All logs are now persistent between reboots.

Still unclear why they weren't, unchecked Syslog means logs are stored on "operating system device", which is same device that holds System dataset ...

The last command tells you if the last shutdowns/reboots etc. were clean or not.

The above issue affected utx.log too, before the fix last gave me no output about previous shutdowns/reboots, just the current session, like this

# last root pts/0 10.1.1.1 Wed Dec 21 14:32 still logged in boot time Wed Dec 21 14:25