Reboot stuck at mp0: reinitializing

[CreeMichaud]

Hardware:
Supermicro X8DAL with 2x Intel x5660
64 GB ECC RAM
SAS9207-8i (LSISAS2308) BIOS: 7.39.02.00 FW: 20.00.07.00

Environment:
TrueNAS-12.0-U1 running as VM (this is a non-prod homelab)
Hosted on XCP-ng 8.2.0 (latest patches)
SAS HBA is PCI-Passthrough to guest OS (TrueNAS)

Symptom:
VM reboot gets stuck at "mp0: reinitializing" unless I set "verbose" mode from boot menu (opt 6) or I reboot entire XCP-ng host

Background:
This hardware has been running without issue as a FreeNAS 11.3-U4 guest on XCP-ng 8.1.0. I've completed multiple in place upgrades previously without issue.
No hardware changes, no BIOS changes, no Firmware changes in the environment. Only change in environment was migration to TrueNAS and apply XCP-NG patches.
Installation and first boot to TrueNAS occurred without incident. Attached disks and imported pool without issue. When restoring TrueNAS config, the reboot gets stuck.
Terminal won't get passed "mp0: reinitializing"
VM Boot never progresses no matter how long I wait. Restarting the VM yields the same symptom. If I Reboot XCP-ng host it resolves the issue. A subsequent VM reboot gets stuck again. Strangely, one work around I stumbled upon was to set "verbose" option from boot menu. Most of the time I can get it to boot without rebooting the XCP-ng host. Subsequent VM reboots get stuck again.
Once TrueNAS actually boots up, the system appears to be stable enough to complete a scrub on 10TB of encrypted raidz2 in 10hrs with no system activity. I have not found any suspicious log entries related to disk/HBA subsystems in either VM or HOST logs
I have reinstalled a fresh copy of TrueNAS instead of an in place upgrade. No change in behavior
I have also tested/found the same behavior on FreeNAS 11.3-U5 and TrueNAS 12.0-RC1

I have not tried to roll back XCP-ng to 8.1.0 or to try bare metal TrueNAS with same hardware config. Both of these options are a large investment in time. I unfortunately don't have a spare server/SAS HBA. A rollback would require a complete wipe / restore of all VMs
I find it odd the "verbose" boot option seems to affect the symptom. I'm not sure how to leverage this data point to find the solution.
Does anyone have a suggestion I can try before rolling it all back?
Any help/advise is appreciated, thanks in advance.

 

[sretalla]

You could just try creating a fresh VM and passing it the same hardware and see if it's able to get going. This will at least save you the messing around to see if it runs on bare metal or different versions of hypervisor (at first at least).

 

[CreeMichaud]

Thanks for the feedback sretalla however I have tried this step already with FreeNAS 11.3-u5 and TrueNAS 12.0-U1 no change in behaviour. My task for today is try to find my older FreeNAS ISO’s and try them. I wanted the newest ZFS features so I was hoping for a solution for the TrueNAS 12.0-U1 platform.

 

[CreeMichaud]

Update: Over the holidays I did a fresh install of XCP-NG 8.1 and confirmed FreeNAS 11.3-u5 --> TrueNAS 12.0-U1 all appear to work as designed.
Same hardware, same configuration. Fresh install of XCP-NG 8.2 and the problem is reproduced on multiple versions on FreeNAS / TrueNAS.
This problem appears to be related to an XCP-NG update.
A search in the XCP-NG forums got a similar hit and is still underinvestigation. Added link for reference for anyone encountering this issue

Passthough on 8.2 hangs on VM reboot and stops other VMs passthough devices working

Happy Christmas all!! I took the opportunity to upgrade my pool to 8.2 which went smoothly (other than the expect cephfs issue migrating from the test drive...

xcp-ng.org

 

[CreeMichaud]

Update: There is a kernel patch on XCP-NG 8.2 that seems to have caused this behaviour. The XCP-NG dev team provided a kernel with the patch removed that is a verified working solution. A ticket was opened with Xen Project for an official fix. If you are running TrueNAS Core 12+ in a VM on XCP-NG 8.2 with a SAS or NIC passed through to VM, please follow the forum link above for your work around.