TrueNAS Core and ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card = no boot

[MisterBaz]

Been running TrueNAS Core just fine for quite a while now. I decided I needed some faster flash storage (for VMs via iSCSI on hypervisors), so I plopped an ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card (with 4 x 1TB Samsung EVO NVMe drives) in and enabled the x4x4x4x4 bifurcation for the slot. The system sees 4 new 1TB NVMe devices (I can see it when TrueNAS is attempting to load), but now it won't boot TrueNAS. I've also set the bifurcation back to default with no change. So, there is something it doesn't like about seeing that ASUS expansion card.
I've received the following types of errors in my troubleshooting:

• I get the "error 2" when attempting to boot.
• mountroot: unable to remount devfs under /dev (error 2)
  mountroot: unable to unlink /dev/dev (error 2)
  init: not found in path /sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/initpanic: no init
• I get a "panic" and then a "db>" prompt.

This is on a Dell R730xd. I'm using the rear flexbay with two 512GB SATA SSDs as boot devices.
Very similar to this guy (added new disc and gets the same errors): https://www.truenas.com/community/t...with-error-2-after-adding-storage-disc.69575/
I've also tried reinstalling TrueNAS - it installs fine but boots right up to the same error as before.

I remove the card and everything works fine. Why does it fail to boot when I have the card installed?

 

[NickF]

Interesting.
Some random thoughts that may be barking up the wrong tree. I also have no idea what the impact of downtime is for you so this may be untenable.

Questions are in order of severity of impact.

Are you booting with BIOS or UEFI? Can you try changing it from one to the other?

Can you try reinstalling TrueNAS as a new new boot environment?

Can you try a fresh install of TrueNAS formatting your devices and then restoring your config from backup?

Can you try reinstalling TrueNAS on a usb drive and booting from that just to see if behavior changes?

 

[MisterBaz]

Interesting.
Some random thoughts that may be barking up the wrong tree. I also have no idea what the impact of downtime is for you so this may be untenable.

Questions are in order of severity of impact.

Are you booting with BIOS or UEFI? Can you try changing it from one to the other?

BIOS. No.

Can you try reinstalling TrueNAS as a new new boot environment?

Did, same effect.

Can you try a fresh install of TrueNAS formatting your devices and then restoring your config from backup?

I reinstalled TrueNAS to the boot drives and still had the error.

Can you try reinstalling TrueNAS on a usb drive and booting from that just to see if behavior changes?

No.

Weirdly enough, I got it working. I had the card removed so I could restore from backup (which succeeded). I then ran the ipmitool commands to disable the third party PCIe card algorithm thing (keep the fans from going to 100% of a Dell server if a non-Dell branded card is used). I also yanked a SAS drive that wasn't being used.

For kicks and giggles, I installed the card and powered on the system. It powered up, iDRAC was unresponsive for longer than normal (which was weird). It came up, then went back down. Then the whole system rebooted, which was REALLY strange. After that, everything came up normal. I could now see the 4 NVMe devices as usable drives.