TrueNAS 12 Bhyve, Pool, or Jail causing panic on boot (most of the time)

[Samuel Tai]

There's a couple of possibilities to try. The middlewared.log shows a thread exception right after the jail starts and tries to hook vnet. The messages log shows watchdog timeout. In your BIOS, try disabling HyperThreading, and also disabling any system watchdog. I know on my system, I had to disable the BIOS watchdog, or I would have random reboots. I don't think the HyperThreading is actually an issue, but it's worth simplifying your CPU for troubleshooting purposes.

 

[dfalke]

So I checked the BIOS settings, Watchdog default is "Disabled" so I left it off. HyperThreading was turned on, so I "Disabled". Thoughts since watchdog was already disabled.

 

[Samuel Tai]

Hmm. Both of those were right before the reboot, so they're the most likely candidates. Let's see if disabling HyperThreading improves stability.

 

[dfalke]

After disabling HyperThreading it still reboots. I created a new jail and put a static IP and didn't check VNET, DHCP, or anything else. Very basic, jail started. I wonder if it has todo with checking VNET. I will play around with settings and see if I can isolate which option is causing it.

 

[Samuel Tai]

I’ll be out for a couple of days. Good luck.

 

[dfalke]

Alright, getting back to figuring out the issue with starting a basic/new jail in TureNas 12-U2.1. setup, but I need some help!!

I am able to create new jail with basic settings, and as far as networking I select DHCP (which uses VNET) and upon starting the jail it causes a reboot. If I put in a static ip and deselect VNET it works fine, no reboot.

My test reboot occurred right around Mar 29 between 17:35 and 17:39 in the logs below, attached are full logs. There seems to be a reference of core dump at Mar 29 17:39:38, (e.g. Segmentation fault (core dumped)), but there are not any new files generated under /data/crash, let me know if there is another place to look.

From Console
Mar 29 17:32:02 nas kernel: arp: 00:25:90:71:52:4b attempts to modify permanent entry for 192.168.111.201 on em1
Mar 29 17:33:01 nas kernel[1596]: Last message 'arp: 00:25:90:71:52:' repeated 43 times, suppressed by syslog-ng on nas.sgcof.local
Mar 29 17:36:58 nas kernel: watchdog pre-timeout, WD_SOFT_LOG

From Messages
Mar 29 17:36:30 nas ipmi0: KCS: Failed to start write
Mar 29 17:36:51 nas ums0: at uhub2, port 2, addr 3 (disconnected)
Mar 29 17:36:51 nas ums0: detached
Mar 29 17:36:51 nas ukbd1: at uhub2, port 2, addr 3 (disconnected)
Mar 29 17:36:51 nas ukbd1: detached
Mar 29 17:36:52 nas usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_TIMEOUT, ignored)
Mar 29 17:36:52 nas usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_STALLED
Mar 29 17:36:53 nas usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_STALLED, ignored)
Mar 29 17:36:53 nas usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_STALLED
Mar 29 17:36:58 nas kernel: watchdog pre-timeout, WD_SOFT_LOG
Mar 29 17:36:58 nas kernel[1596]: Last message 'watchdog pre-timeout' repeated 1 times, suppressed by syslog-ng on nas.sgcof.local
Mar 29 17:36:58 nas watchdog pre-timeout, WD_SOFT_PRINTF
Mar 29 17:37:00 nas ipmi0: KCS: Failed to start write

Middlewared Log:
[2021/03/29 17:31:43] (INFO) iocage.callback():71 - TestJail successfully created!
[2021/03/29 17:31:50] (DEBUG) iocage.__start_jail__():248 - Grabbing IPv4 default route
[2021/03/29 17:31:50] (DEBUG) iocage.__start_jail__():250 - Default IPv4 Gateway: 192.168.111.1
[2021/03/29 17:31:50] (DEBUG) iocage.__start_jail__():253 - Grabbing IPv6 default route
[2021/03/29 17:31:50] (WARNING) iocage.callback():69 - No default gateway found for ipv6.
[2021/03/29 17:31:50] (DEBUG) iocage.__start_jail__():255 - Default IPv6 Gateway: none
[2021/03/29 17:31:50] (INFO) iocage.callback():71 - * Starting TestJail
[2021/03/29 17:31:51] (INFO) iocage.callback():71 - + Started OK
[2021/03/29 17:31:51] (INFO) iocage.callback():71 - + Using devfs_ruleset: 1000 (iocage generated default)
[2021/03/29 17:31:54] (INFO) iocage.callback():71 - + Configuring VNET OK
[2021/03/29 17:31:54] (INFO) iocage.callback():71 - + Using IP options: vnet
[2021/03/29 17:34:09] (INFO) iocage.callback():71 - + Starting services OK
[2021/03/29 17:34:09] (INFO) iocage.callback():71 - + Executing poststart OK
[2021/03/29 17:34:09] (INFO) iocage.callback():71 - + DHCP Address: 192.168.111.133/24

Thanks,
David

 

[Samuel Tai]

Can you supply the output of ifconfig -a? Usually, when there's a VNET issue, you have to clear out all your networking config, and then rebuild it.

 

[dfalke]

Attached is the output for ifconfig -a

 

[dfalke]

Also, did you see anything in the recent logs i posted. I was reading about the watch dog, its disabled in the bios but I was reading that on supermicro boards there was another watch dog as part of the IPMS(?sp?) and that you can disable watchdog integration with one of the following hints...

hint.wdwd.0.disabled = 1
watchdogd_enable = NO

Do you think watchdog is possibly an issue even though disabled in bios?

 

[Samuel Tai]

With respect to the watchdog, it can't hurt to use the first hint as a tunable. As for VNET, you may want to redo your networking so vnet0 and bridge0 are on em0, instead of em1. If you need to use em1 due to your network topology, then you may need to set the vnet_default_interface in the jail's Basic properties to em1 instead of auto.

 

[dfalke]

Sorry it might be a dump question, but how do i go about rebuilding the network interfaces? Delete network interface em1 and recreate? use the "Reset configuration"? Any pointers would be appreciated.

 

[Samuel Tai]

According to the revised documentation at https://www.truenas.com/docs/core/network/interfaces/settingstaticip/, you could use the console menu, select option 1, then answer n to Delete interface?, and then y to Remove interface settings?.