New install fails. Reboots after kernel loads

[kev2011]

I have run out ideas to test. I am trying to do a new install of TrueNAS 12.0-Release onto a NVMe drive. I have tried multiple usb drives (and iso writers) and now a via DVD. I tried UEFI and legacy bios settings, I set the install device to the 1st boot option, network boot is disabled. Secure boot is disabled as far as I understand (see attached). It will show the boot screen and load the kernel but then it simply reboots without reaching the install screen. It does that in a loop. I was able to install Ubuntu 14.04 and Windows 7 with no issue. I was also able to install TrueNAS on a VM in Ubuntu. I am not seeing any errors. It just reboots without warning. I also tried removing all devices (HBA and drives, NVMe drive) and using a HDD but the same issue occurs. I also tried other versions for TrueNAS and FreeNAS, same problem.

The regular install seems to stop at the same point (attached: install_6 or install_7 with NVMe installed). Attached pictures are only for the regular install (not VM).

If I could get some insight as to where to look next, I would appreciate it.

Build details:
Motherboard: ASUS Z10PE-D16 WS
CPU: Intel Xeon E5-2683v4 (x2)
Ram: Samsung M386A4G40DM0-CPB 32GB DDR4 2133 LRDIMM (x4)
Boot drive: SK hynix Gold P31 1TB PCIe NVMe Gen3 M.2
HBA: LSI Broadcom SAS 9300-8i
Pool drives: 6TB WD Ultrastar (x8)
ODD: Asus DRW-24B1ST-N28

 

[Kris Moore]

NVME install will require installing / booting in UEFI mode, so you can scratch legacy booting.

Bios settings looks OK, but by any chance do you have CSM mode enabled still somewhere?

Also, are you on latest motherboard bios/firmware version?

 

[kev2011]

I tried to bypass the issue and transfer the VM image directly to the NVMe drive. It almost worked but it still has the issue of not loading. It starts the boot sequence (showing TrueNAS menu) but restarts before loading the OS. It seems like there is an issue with mounting the file system at boot time. The one difference I noticed from the VM is the root mounting sequence is not happening when trying to install on the physical drive or when booting the VM install I transferred to the NVMe.

I will need to test installing again. I think I tried disabling CSM but it thought it had the same issue. Plus, I did more research on secure boot. For it to be fully disabled I needed to delete the installed keys. ASUS bios is particular about OS it allows. The BIOS version shows it is newer than the one available from ASUS. But I think I'm going downgrade it.

Let me run more tests and I'll post results. Thank you.

 

[kev2011]

Sorry for the delay. I had to take break from testing. I wish I had positive results. It still will not work.

There is something weird about the BIOS on this motherboard. It does not like to have CSM disabled. Sometimes it will not save the change, plus re-load secure boot keys. When I'm able to disable CSM it removes the option to adjust boot priority. All the devices are recognized by the POST test at boot. But the NVMe drive will not show up in BIOS. Sometimes the optical drive will not show up. When it does load correctly I can load the install kernel but it stops at the same point as before. I have wiped the NVMe but maybe I need to remove the GPT table as well. There also seems to be a compatibility issue for devices with UEFI booting. Some characters show up as "?" and the keyboard has issues. The arrow keys cause the boot menu to exit into command line. I also did a confirmation test of secure boot. It is disabled, since leaving it enabled blocks the disk image before it even loads. I'm not sure how UEFI is implemented for this motherboard since its considered a workstation. The setup screen looks the same with CSM disabled. I don't have the fancy menu like on many new motherboards.

I downgraded the BIOS by 1 version, the issue is the same. I haven't tried another version. TrueNAS seems to have an issue with loading on this motherboard. Once the kernel loads, is BIOS even involved? Additional reading makes me think the BSD boot manager could be the issue. I have seen forum posts about the change from GRUB creating some issues. It could also be the ASUS UEFI configuration. I have read it will only load the boot file is it is \EFI\BOOT\BOOTx64.EFI. I don't know enough to determine if that is the case. But I will be trying efibootmgr to have a look.

Any other ideas to test would be great. I don't want to replace this motherboard if I can avoid it.

 

[kev2011]

I decided to try freeBSD 12.2 since it is similar and it worked the 1st time. My previous thought about mounting the file system seems to be correct. The output from the kernel before launching the installer is nearly the same except freeBSD is able to start the mounting process and TrueNAS does not. CSM was disabled and I am using a dvd to boot in both cases.

In picture 'TrueNAS_2' and 'freeBSD_2' you can see the "ahcich4: <AHCI channel ..." in both near the top (sorry about the blurriness), freeBSD continues after this but TrueNAS does not. That seems to be the point when the issue lives. I think a bios issue can be ruled out that this point.

Does anyone know what the difference is in the kernels that could be the cause? Thank you.