Spontaneous reboots TrueNAS Core 13

[keithg]

I cannot be 100% certain that it is only TrueNAS Core 13, but I do not recall ever having spontaneous reboots on anything from True NAS 9-12. I am running this on a Dell Xeon with 32 Gb of ECC ram and have been for a few years. It is a 4T NS and the only thing running in the 'background' is a jail with Plex running. The reboots do not appear to be time of day related nor with any heavy load. In fact It has never rebooted on me when I was actively using any of the shares or Plex in the jail. It generally reboots overnight or when I am at work.

The console log shows nothing before the reboot.

Console just shows something from hours before the reboot showing when I typed in the password incorrectly then shows bootup messages at the time of reboot.

There are crash logs, should I upload one?

I just tonight updated to the latest Core 13 U3.

 

[NugentS]

Do you have a UPS?

 

[keithg]

Yes, I have a cyberpower UPS. Pretty recent vintage. I had it with the 12.0 as well.

Code:

#  upsc ups
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 20
battery.mfr.date: CPS
battery.runtime: 1136
battery.runtime.low: 300
battery.type: PbAcid
battery.voltage: 13.7
battery.voltage.nominal: 12
device.mfr: CPS
device.model:  CP 1500D
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: CyberPower HID 0.4
driver.version.internal: 0.41
input.transfer.high: 140
input.transfer.low: 90
input.voltage: 120.0
input.voltage.nominal: 120
output.voltage: 120.0
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.delay.start: 30
ups.load: 24
ups.mfr: CPS
ups.model:  CP 1500D
ups.productid: 0501
ups.realpower.nominal: 388
ups.status: OL
ups.test.result: Done and passed
ups.timer.shutdown: -60
ups.timer.start: 0
ups.vendorid: 0764

 

[NugentS]

Try turning off the UPS Service and see if the reboots stop

 

[keithg]

Ok. Is this a known issue? Is it with TrueNAS Core 13 or was it previously apparent?
For November, I had reboots on Nov 11,14, 16, 18, 23, 24, 26. Will see how the month of December fares with this change. Thanks for the suggestion,

 

[keithg]

Just had a reboot last night.

Code:

* NAS.xxxxx had an unscheduled system reboot.
The operating system successfully came back online at Mon Dec 5 03:23:00 2022.

The UPS service was not running when it had the unscheduled reboot. I do not believe there was a power outage. If one did occur, it was not long enough for the UPS to be depleted.

 

[NugentS]

Well that my idea gone.
I would suggest running memtest for a couple of days to check the memory and prime for a day or so to test the CPU. See if anything is overheating

 

[winnielinnie]

No replication tasks configured?

 

[keithg]

Will start looking at memtest and such. I will bring it down tonight and vacuum out any accumulated dust, but when it reboots, it is just idling. The last checks of the file system (boot and zfs pool) completed normally and that is usually the highest usage the system sees. Not doing a replication I do not think, but will check for sure, tonight. It might be hung, thouugh, as I cannot get to the login page today over VPN, so I will have to check.

Like I intimated, this did not occur with TrueNAS 12. If it did reboot, it was usually due to a longer term power outage. Never happened such that I was moved to question it or post about it. Now it is weekly or more.

I cannot easily go back to 12 to verify that it is a 13 vs 12 thing as both my jails are now at 13.0. I guess I could re-create them under 12. One is pretty easy, the other a bit more challenging but do-able. I am at work, so when I get back this evening, I will see if this log shows anything else.

 

[keithg]

Well, I confirmed nothing was going on. When I got home, I could log in without issue. I also got another reboot at 5pm today. Turning off the UPS did not seem to change anything. I have not shut the NAS down for cleaning, but there is nothing untoward on the main page when it appears to go down (at 14:35 the data streams stop). It is strange that it does not come back until 5pm:

Code:

* NAS.xxxx had an unscheduled system reboot.
The operating system successfully came back online at Wed Dec  7 17:02:52 2022.

Code:

Memory :
31.9GiB total available (ECC)
Free: 27.4 GiB
ZFS Cache: 1.5 GiB
Services: 3.0 GiB

It looks like the CPU was at ~45C but at 800% 'idle' when it went down. It certainly looks like nothing going on. Is 45C bad? I have not yet run memtest, but that will require a bit of contortion as it runs headless and I'll have to get a monitor and keyboard over to it. Could it be that the clock gets out of sync? I do not think it went down at 14:35 then took until 17:02 to come back. A reboot usually takes on the order of seconds, not hours.

 

[ggiinnoo]

Do you have HW transcoding enabled on plex?

I had this issue with HW transcodingenabled.
I am running an intel 8086K. The solution is either buying a dummy plug, or for testing hooking up a display. It doesn't have to stay on, but it has to receive a signal for it to not crash.

 

[keithg]

No transcoding. I do not have the hardware for that, anyway. These reboots may have been due to periodic thermal excursions. I was heavily using it (making backups in preparation for shutting it down and upgrading the hardware) and I saw some excursions to 80C at times. It is an old system and draws some 250W at idle, so it is due for replacement. I am upgrading to scale with the new hardware sometime this week.

 

[Whattteva]

250W at idle, that is crazy high power use. If I were you, I would just replace that old clunker and replace it instead of wasting more time chasing a wild goose.
80C is a bit high, but still shouldn't induce a reboot. I say this because my gamer PC gets higher than that under load and has never rebooted or crashed. Of course, it's a different CPU type (I'm guessing since you never mentioned any specs), so it's possible that it has higher operating temperature range.