TrueNAS on NVMe, bootloader on USB

[slotmaster]

Hi,
I have HP ProLiant G8 servers with bunch of drives in the hot swap bays and NVMe SSDs in PCIe slot. Unfortunately the ProLiant does not support booting from the drive bays (with Smart Array controller in HBA mode), neither it supports booting from the PCIe slots. It can however boot from the internal USB slot. So when I needed HBA mode I usually installed Debian on the NVMe SSDs and a bootloader on a USB stick, which worked fine.
Now I want to install TrueNAS SCALE the same way, but I hit a wall with the installer which basically has zero options. I was able to install TrueNAS onto the NVMe SSDs, but obviously that won't boot. I tried to install GRUB and move the /boot folder after install, but the boot drive is zfs which I never used before, so I just spent the day messing around and getting nowhere.
I tried using an adapter to plug the SSD into the internal USB port, but the speed is so low it takes around 15 minutes just to boot, so that's not an option.
Does anybody know about some guide/blogpost that covers this bootloader install issue? I wasn't able to find any...
Thanks.

 

[Arwen]

There is no supported way for TrueNAS SCALE to chain boot. Plus, future update(s) may cause any work around to fail.

That said, I have 2 laptops that won't boot SD card media. I use those SD cards as both backups on one partition, and alternate boot environment on another partition, in case I need something for recovery. I figured out how to get them to work, but it is clumsy.

I added something like this below to my "working" boot partition. The BIOS boots to the working disk's boot partition. Then I can select, (or make default), a chain loaded OS bootable through Grub.

Code:

####################
#
# Chainload SDXC drive
#
menuentry "SDXC chainloader+" {
    set root='(hd2,gpt3)'
    chainloader (hd2)+1
    configfile (hd2,gpt3)/boot/grub/grub.cfg
}

Please note that I can't help you trouble shoot the details. I don't have knowledge of all the details.

Basically the "hd2" would be the device of the target. Then, "gpt3" is the boot partition of of the target. How to get the NVMe name used by Grub, ("hd2" in my case), for your case, is beyond me. I think if you make Grub fall into command line mode, you can list the devices it sees. Last, Grub almost certainly needs to be built with NVMe & ZFS support on the working boot disk, (your USB device).

This is all I know;

Code:

Using command line mode, ("c" from menu):
-----------------------------------------
        First, set pager;

grub> set pager=1

        Listing disks & partitions;

grub> ls

        Listing files in a partition;

grub> ls (hd0,1)/

        Booting off an un-listed entry;

grub> set root=(hd0,1)
grub> linux /boot/vmlinuz-3.13.0-29-generic root=/dev/sda1
grub> initrd /boot/initrd.img-3.13.0-29-generic
grub> boot