Shutdown Causes Reboot on TrueNAS Core 13.0-U1

[scout213]

My backup server is running TrueNAS Core 13.0-U1, and when I try to shut down the server using the Web UI and/or console, the server reboots 5 - 10 seconds after it powers off. The server specs are as follows:

Intel R2000 Server with S2600GZ motherboard
2 x E5-2630L v2
32GB ECC RAM
LSI HBA

I've installed the latest BIOS, BMC firmware, Intel ME firmware, and HBA firmware. I've also tried removing all PCIe devices, disconnecting all network cables, reseting the BIOS/BMC to factory defaults, turning off wake-on-lan, and yet the system still powers on seconds after shutting down. This issue also seems to be specific to TrueNAS, as the server powers off correctly under Linux (Debian 11.4).

I understand this issue is from a somewhat unusual use case, as most TrueNAS systems stay up 24/7, but this backup server is intended to only power on weekly for a few hours to perform replication tasks from the primary TrueNAS box and then shut down to save power.

One last thing, I've noticed the BMC/system status light blinks when TrueNAS shuts down the server, but remains solid when powered down from Linux. I suspect TrueNAS might be causing the BMC to reset just before shutdown, and that is then triggering the system to reboot instead of staying powered off, but I'm not sure if there is a way to confirm this theory.

Thanks in advance for any help.

 

[Samuel Tai]

Check your BIOS settings for anything like "Restore on AC/Power Loss", and change it to Off or Disabled. The BMC may be interpreting the shutdown as loss of wall power.

 

[jlpellet]

As a test, you could see what happens from the console (i.e., shutdown -h now) & see if a reboot occurs. If so, that would seem to indicate the issue is outside the OS. If the cl halts the system, we can look further about why the GUI is different. Good luck.

 

[scout213]

Check your BIOS settings for anything like "Restore on AC/Power Loss", and change it to Off or Disabled. The BMC may be interpreting the shutdown as loss of wall power.

Thanks, I went into the bios and verified that Restore on Power Loss was set to off.

As a test, you could see what happens from the console (i.e., shutdown -h now) & see if a reboot occurs. If so, that would seem to indicate the issue is outside the OS. If the cl halts the system, we can look further about why the GUI is different. Good luck.

If from the console I choose option 11 for reboot, it behaves the same as the web interface, and the server powers back on within a few seconds of shutting down. Followed your recommendation, from the console, I selected option 9 for a shell, and then ran 'shutdown -h now'. The system however never shuts down, and instead get stuck with my network adapters alternating between up and down states. Below is the tail end of the console output once the 'shutdown -h now' command is run. The interface up/down continues to repeat indefinitely (I let it run for 5 minutes and it didn't stop). I'm also no longer able to interact with the system at that point (ctrl+c, ctrl+alt+del, etc. don't result in any system change/behavior).

Code:

Uptime: 17m425
GEOM_MIRROR: Device swap3: provider destroyed.
GEOM_MIRROR: Device swap3 destroyed
GEOM_MIRROR: Device swap2: provider
destroyed.
GEOM_MIRROR: Device swap2 destroyed.
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
GEOM MIRROR: Device swapo: provider destroyed.
GEOM_MIRROR: Device swap@ destroyed
ix1: link state changed to DOWN
lagg8: link state changed to DOWM
ix1: link state changed to UP
lagg8: link state changed to UP
uhube: detached
ukbdo: detached
ums0: detached

The operating system has halted.
Please press any key to reboot.

ukbd1: detached
ums1: detached
uhido: detached
uhub3: detached
uhubi: detached
ix1: link state changed to DOWN
laggo: link state changed to DOWN
ixl: link state changed to UP
lagg8: link state changed to UP
xl: link state changed to DOWN
lagg0: link state changed to DOWN
ixl: link state changed to UP
lagg0: link state changed to UP
ix1: link state changed to DOWN
lagg8: link state changed to DOWN
ix1: link state changed to UP
lagg0: link state changed to UP
ix1: link state changed to DOWN
lagg0: link state changed to DOWN
ix1: link state changed to UP
lagg8: link state changed to UP
ix1: link state changed to DOWN
lagg8: link state changed to DOWN
ix1: link state changed to UP
lagg0: link state changed to UP
ix1: link state changed to DOWN
lagg0: link state changed to DOWN

 

[Samuel Tai]

Ah, does your BIOS have an option for Restore on PCI or Restore on network?

 

[scout213]

Ah, does your BIOS have an option for Restore on PCI or Restore on network?

I just looked, and I don't see any options like that. Intel provides a BIOS simulator for my board at this link if you would like to see what my BIOS options are:

https://www.intel.com/content/www/u...-1600-2400-2600-server-platform-families.html

 

[winnielinnie]

Intel provides a BIOS simulator for my board

That's pretty cool, I must admit!

I wish more manufacturers did that, but preferably web-based (rather than an .exe).

EDIT: On second thought, it's not that cool, since it's more of a "tour" than an interactive simulator.

 

[Samuel Tai]

Is Plug & Play BMC Detection enabled? TrueNAS's ipmitool can access the BMC. Also try disabling PXE boot.

 

[scout213]

Is Plug & Play BMC Detection enabled? TrueNAS's ipmitool can access the BMC. Also try disabling PXE boot.

Yes, I've tried it with that feature both enabled and disabled, and the system still reboots after shutdown. Is there an option or tunable to disable TrueNAS IPMI integration to see if that might be causing the restart behavior?

 

[Samuel Tai]

Yes, I've tried it with that feature both enabled and disabled, and the system still reboots after shutdown. Is there an option or tunable to disable TrueNAS IPMI integration to see if that might be causing the restart behavior?

You could try setting a LOADER tunable ipmi_load="NO" to prevent the kernel from loading the IPMI module on the next boot. I don't think this will help, though, as kernel modules sometimes get loaded anyway after the system comes up.

There are other SYSCTL tunables to try:

Code:

hw.ipmi.on <- defaults to 1
hw.ipmi.wd_init_enable <- defaults to 1

 

[jlpellet]

Thanks, I went into the bios and verified that Restore on Power Loss was set to off.


If from the console I choose option 11 for reboot, it behaves the same as the web interface, and the server powers back on within a few seconds of shutting down. Followed your recommendation, from the console, I selected option 9 for a shell, and then ran 'shutdown -h now'. The system however never shuts down, and instead get stuck with my network adapters alternating between up and down states. Below is the tail end of the console output once the 'shutdown -h now' command is run. The interface up/down continues to repeat indefinitely (I let it run for 5 minutes and it didn't stop). I'm also no longer able to interact with the system at that point (ctrl+c, ctrl+alt+del, etc. don't result in any system change/behavior).

OK - Odd that the shell & menu behave differently. So, some ideas. Given the response you describe, I think it likely it is a OS issue.
1. Does anything diffferent happen if you execute the shutdown command from ssh or the gui shell?
2. Is this behavior new to 13U1 (i.e., did it work in 12.x)?
3. Does it do this from a fresh install (i.e., rule out corrupted install)?
Hope this helps.
John

 

[Cryolithe]

Same problem here with TrueNAS 13.0-U1. It works well on TruNAS 12. Intel i5-3550 on MSI P8H77-V LE mobo.
Did you open a bug ticket?

 

[scout213]

OK - Odd that the shell & menu behave differently. So, some ideas. Given the response you describe, I think it likely it is a OS issue.
1. Does anything diffferent happen if you execute the shutdown command from ssh or the gui shell?
2. Is this behavior new to 13U1 (i.e., did it work in 12.x)?
3. Does it do this from a fresh install (i.e., rule out corrupted install)?
Hope this helps.
John

I did some additional testing, and this problem does not occur in TrueNAS Core 12, or TrueNAS Scale 22. I also tested out FreeBSD, and the problem is present in FreeBSD 13, but not FreeBSD 12. I think this pretty strongly indicates something kernel related and that a change between v12 and v13 introduced this issue.

Same problem here with TrueNAS 13.0-U1. It works well on TruNAS 12. Intel i5-3550 on MSI P8H77-V LE mobo.
Did you open a bug ticket?

No, I have not yet filed a bug report. It seems at this point it's not just TrueNAS specific, but traces back to FreeBSD. Does anyone have a recommendation regarding filing this with TrueNAS or going to the FreeBSD folks?

You could try setting a LOADER tunable ipmi_load="NO" to prevent the kernel from loading the IPMI module on the next boot. I don't think this will help, though, as kernel modules sometimes get loaded anyway after the system comes up.

There are other SYSCTL tunables to try:

Code:

hw.ipmi.on <- defaults to 1
hw.ipmi.wd_init_enable <- defaults to 1

Thank you for the recommendations. I tried them, but unfortunately they did not change/resolve the issue.

 

[Redcoat]

Does anyone have a recommendation regarding filing this with TrueNAS or going to the FreeBSD folks?

I would do both, advising each that you have filed the issue with the other, in both cases referring also to this thread.

 

[scout213]

I would do both, advising each that you have filed the issue with the other, in both cases referring also to this thread.

Ok, thanks. I reported the issue to both.

 

[Redcoat]

Ok, thanks. I reported the issue to both.

I should also have suggested that you post links to the bug reports here to faciliate others following them.

 

[IvanVolkov]

I have the same issue. It happens after truenas 13 upgrade. Is there solution?

 

[scout213]

I have the same issue. It happens after truenas 13 upgrade. Is there solution?

Not that I know of. I ended up having to migrate that system to TrueNAS Scale. I reported the bug to FreeBSD and iXsystems. The TrueNAS devs said there isn’t anything they can do since it also happens in FreeBSD 13. Not much has happened at this point on the FreeBSD bug report. Here’s then link to the ticket if you want to try and add additional comments to the bug report.

265466 – Unexpected start after shutdown -p now, if hw.efi.poweroff is true

bugs.freebsd.org

 

[Patrick M. Hausen]

I did some additional testing, and this problem does not occur in TrueNAS Core 12, or TrueNAS Scale 22. I also tested out FreeBSD, and the problem is present in FreeBSD 13, but not FreeBSD 12. I think this pretty strongly indicates something kernel related and that a change between v12 and v13 introduced this issue.

Then I recommend contacting the freebsd-stable mailing list about this issue. There's little iXsystems can do.

FreeBSD Mailing lists: subscription for freebsd-stable

 

[IvanVolkov]

I've found solution. Set hw.efi.poweroff=0 in sysctl instead 1