Unable to boot from NVMe SSD after a fresh install

[northbridg3]

Hello everyone!

I recently decided to go ahead and complete a home NAS project for myself. I haven't tried TrueNAS before, but after a lot of reading here and elsewhere, I decided to go with TrueNAS Core, then ordered a few parts and assembled the following build:

CPU: Intel Core i3-9100 3.6 GHz
Motherboard: Supermicro X11SCL-IF Mini ITX LGA1151
Memory: 2x Kingston KSM26ED8/16ME 16 GB DDR4-2666 ECC
Storage: 2x Western Digital Red SA500 2 TB 2.5" SSD
OS drive: Crucial P5 Plus 1 TB M.2-2280 PCIe 4.0 X4 NVME SSD

Then I went on to memtest86+, cpu-stress and the SMART tests, with no errors encountered so far.
Next on the list was install TrueNAS, so I got the 13.0-U6.1 iso and booted it via the IPMI. During the install I selected "Boot via UEFI" as I've read that a NVMe boot should use that option. The installation went on without any errors, but after reboot and selecting the (UEFI) NVMe, the system was unable to load the kernel with the following error:

Code:

Loading kernel...
/boot/kernel/kernel text=0x1667bd4 ZFS: i/o error - all block copies unavailable

elf64_loadimage: read failed
/boot/kernel/kernel text=0x1667bd4 ZFS: i/o error - all block copies unavailable

elf64_loadimage: read failed
Failed to load kernel 'kernel'
can't load 'kernel'

This is absolutely the same error as the one described in this thread: strange ZFS boot issue with new TrueNAS build

Thinking that I got a faulty NVMe SSD, I went on and redid the SMART test on that drive, along with a full capacity dd read/write test. No errors reported.
I tried playing with various settings in the BIOS related to the UEFI/BIOS and the SATA/M.2 drive, none of which helped. Pulling out the SATA cables for the other SSDs and disabling the SATA controller in the BIOS did not help either.
I tried an installation of FreeNAS 11.3, TrueNAS 12.0 and TrueNAS 13.1-master, and they're all unable to boot with the exact same error.
However, I was able to install and boot TrueNAS Scale 23.10 without any issues. Same goes for a Ubuntu MATE 23.10, and even a FreeBSD 13.2.

Does anyone have an idea what could be the source of this problem? There are a few suggestions around the net, but they're all about the BIOS not being able to see the NVMe drive, while my system is able to see it, but unable to boot, and only the TrueNAS Core installation. If nothing else works, I'll go with the TrueNAS Scale, but ideally I'd like to use TrueNAS Core as originally intended.

 

[joeschmuck]

I would recommend you use Legacy BIOS for the task, for CORE. UEFI is so problematic. See if that fixes the issue.

 

[northbridg3]

Unfortunately it's still unable to boot properly after a reinstall with selected BIOS mode, although with a different error that suggests it doesn't even reach the bootloader.

Some of the Supermicro FAQ's suggest that to boot from a NVMe drive it needs to do so in UEFI mode. I'm curious, what's so problematic about it?

 

[sretalla]

I tried an installation of FreeNAS 11.3, TrueNAS 12.0 and TrueNAS 13.1-master, and they're all unable to boot with the exact same error.

Common to all those which didn't work is the FreeBSD bootloader.

FreeNAS 11.2 was the last version to use GRUB (same as SCALE... although obviously on older release).

 

[sretalla]

and even a FreeBSD 13.2.

Although this is confounding... should also be the FreeBSD bootloader, but works.

 

[northbridg3]

Common to all those which didn't work is the FreeBSD bootloader.

FreeNAS 11.2 was the last version to use GRUB (same as SCALE... although obviously on older release).

FreeNAS 11.2-U8 still had the FreeBSD bootloader, but I tried with FreeNAS 11.1-release as well which did have GRUB. Guess what...it worked like a charm. So apparently it's indeed an issue with the bootloader.

So I suppose that limits my choices to either wait for a TrueNAS Core release which uses a working bootloader, or go with TrueNAS Scale. If I'm not mistaken, the current version (TrueNAS 13.0-U6.1) is based on the corresponding FreeBSD version (which would be 13.0)? I'll also test the previous two FreeBSD releases (13.1 and 13.0) and see whether they generate the same error.

 

[sretalla]

FreeNAS 11.2-U8 still had the FreeBSD bootloader, but I tried with FreeNAS 11.1-release as well which did have GRUB

OK... my memory is failing... 11.2 was the first with the FreeBSD bootloader, 11.1 was the last with GRUB.

It's still odd that you can get the bootloader working on vanilla FreeBSD 13 though.

 

[joeschmuck]

This has nothing to do with your M.2 bootloader issue but exactly which PCIe to four M.2 card do you have?

Also, just to be clear for me,:
1) You are booting from what device to install TrueNAS CORE?
2) Do you get to the screen to select BIOS or UEFI for installation?
3) If you do get to the BIOS or UEFI screen, what configuration is your motherboard in (BIOS SETUP)? UEFI, DUAL, LEGACY?

If your motherboard is set to LEGACY then select BIOS during the installation, if it's in UEFI then select UEFI. Legacy is the most compatible selection (least amount of problems).

I'm asking these questions to be absolutely clear we understand exactly what you are doing and have configured. If you say to yourself "What the hell is he talking about?" when you read question 3, then this could be the problem.

It is odd that you can install an older version.

Have you checked to ensure you have the most current BIOS firmware?

 

[northbridg3]

Happy New Year! I finally managed to get some time after the holidays, and I did a few more tests with various OS versions. It turned out that the system can actually boot in BIOS mode from the NVMe drive after all. Here are the results, both for UEFI and BIOS modes. In the TrueNAS Core installs, the corresponding boot mode (UEFI or BIOS) has been selected during the installation process.

	UEFI	BIOS
FreeBSD 12.4	No (ZFS i/o error)	Yes
FreeBSD 13.0	Yes	Yes
FreeBSD 13.1	Yes	No (BTX halted)
FreeBSD 13.2	Yes	No (BTX halted)
TrueNAS Core 13.0-U6.1	No (ZFS i/o error)	No (BTX halted)
TrueNAS Core 13.1-master (20240101)	Yes	No (BTX halted)
TrueNAS Scale 23.10.1	Yes	Yes

I'm not sure what to make of these, especially considering that the latest master of TrueNAS Core 13.1 does work in UEFI mode, while the earlier build I tried did not. The best part is that the current stable TrueNAS Core (13.0) is not able to boot in both modes.

This has nothing to do with your M.2 bootloader issue but exactly which PCIe to four M.2 card do you have?

None. The X11SCL-IF has a built-in M.2 slot that I'm using.

Also, just to be clear for me,:
1) You are booting from what device to install TrueNAS CORE?
2) Do you get to the screen to select BIOS or UEFI for installation?
3) If you do get to the BIOS or UEFI screen, what configuration is your motherboard in (BIOS SETUP)? UEFI, DUAL, LEGACY?

1) An ISO file mounted via the IPMI
2) Yes, regardless of the TrueNAS Core version on the ISO file
3) It's currently set to DUAL. While booting I always select manually which device I want to boot and the mode (see screenshot below).

Have you checked to ensure you have the most current BIOS firmware?

FWIW, the firmware is 01.73.12 (latest is 01.74.13), and the BIOS version is 1.6 (latest is 2.2), but given that both FreeBSD 13.0 and TrueNAS Scale boot in both UEFI/BIOS modes, I don't think firmware/BIOS is the issue. Not excluding it 100% though, so I might try updating them after all.

 

[joeschmuck]

I guess the question is, do you want to run in CORE or SCALE? SCALE apparently works for your system, or you could wait for CORE 13.1 to come out.

If you want CORE, try to install TrueNAS 13.0 (no Update "U" version), see if that works. If it does, then clone the boot environment and change to the new boot environment to boot from. Now reboot. Download the "upgrade" version of TrueNAS 13.0-U6.1 and perform the upgrade from that download. Do not use the online update process, sometimes it just does not work well.

Hopefully this process will allow you to run 13.0-U6.1. Let's say TrueNAS CORE 13.0 fails, you could try the last version of 12.

Hope you are able to find a work around if you want CORE.

 

[weingeist]

Hey, did you solve this?

I am running into the same issue, but not for a fresh install but for a system running for 7 years. I switched from USB to USB-NVME mirror to NVME only and tried updating to 13.0.1-U6. Reading your table correctly, I am pretty screwed up, since neither BIOS nor UEFI is compatible with an NVME boot medium, is that correct?

Probably my best bet is to freshly install on a USB thumb drive until 13.1 is released?

 

[northbridg3]

Unfortunately I did not manage to resolve it. The last thing I tried was an install of the latest TrueNAS Core (20240101), but for some reason it resulted in the same ZFS i/o error as the other builds, so I gave up on it, and went with TrueNAS Scale, as none of the Scale builds I tried gave me any issues at all. Now I think that I'll stick with it, as it gives me the option to use Jellyfin, which is a nice bonus.

As for the rest, I think it you could try what @joeschmuck suggested, or stick to the USB thumb drive installation. If I insisted on using Core, then I'd probably go with the latter, as it's less overall hassle compared to the time I spent trying to figure out what's wrong.