Booting from USB to boot-pool on SSDs?

[MGYVR]

I have tried to research this for a bit now and haven't managed to find any documentation, or forum threads about this. It's kind of an oddly redundant group of search terms so hopefully I'm not just searching the wrong terms or missing something completely obvious... Or being a total moron... If by chance I am, then let me say sorry in advance!

In a nut shell:
1. Installed TrueNAS to a USB drive for basic testing and to familiarize myself with it.
2. Installed a PCIe to NVMe adapter, with two small SSDs in it that I was going to use as boot media.
3. Install TrueNAS from another (install media) USB drive onto the SSDs (create boot-pool and complete setup)
3. Turns out my hardware (or possibly the SSDs) won't let me boot from the SSDs.
4. Leave SSDs in, plug the original USB drive install back in (not the install media, the original install of TrueNAS that I was playing with).
5. Boot from USB but appear to load the boot-pool on the SSDs?
-Everything after the TrueNAS ascii boot options screen loaded WAAAAY faster than I recalled that first USB drive install loading.
-When I check on the boot-pool status it shows the boot-pool as the nvd0/nvd1 device pool which I setup before discovering I couldn't boot directly to it.
-I had already setup a pool of SAS drives in the original USB drive install, which didn't exist when I booted into what I believe to be the boot-pool on the SSDs.

Is this known behavior?
Is this a feasible work around for not being able to boot directly from the SSDs?
This USB drive has an activity indicator light, and it goes crazy for a minute or two during boot, but never blinks at all once everything is up and running...

My hardware is a Dell R730xd, I've only found a few documented (random forum posts) cases of people trying to boot to PCIe SSDs on that particular hardware, I guess the Intel PCIe AIO drives will work, but read conflicting reports about it working through an adapter...

It was just such an odd turn of events, I felt like I needed to consult the community...

 

[Samuel Tai]

You may have accidentally mirrored your boot pool onto the NVMe SSDs, so that the initial loading works off the USB thumb drive, but later portions of the boot run off the other members of the mirror.

 

[MGYVR]

You may have accidentally mirrored your boot pool onto the NVMe SSDs, so that the initial loading works off the USB thumb drive, but later portions of the boot run off the other members of the mirror.

Interesting, figured these caps might be instructive:

 

[Samuel Tai]

OK, then what's likely going on is you're using the USB da4's EFI partition only for booting, and after it loads the BTX bootloader, BTX is jumping to the actual boot pool on the SSD mirror.

 

[MGYVR]

OK, then what's likely going on is you're using the USB da4's EFI partition only for booting, and after it loads the BTX bootloader, BTX is jumping to the actual boot pool on the SSD mirror.

Is this a workable arrangement long term? I kinda presume it's not an officially supported arrangement, but is this kinda side-stepping the general recommendation of not booting from USB flash drives?

Many thanks for your input either way!

 

[Samuel Tai]

It's definitely not a supported configuration, and would probably break at the next upgrade.

 

[marcevan]

Funny, I've used FreeNAS since 9.2 and have same USB.

Only when I got to storage/pools I don't see boot pool but on CLI I have one that is FREENAS-BOOT

I'd like to transfer that to SSD I have on ADA0 but the "old" way of extending the boot USB to the SSD is not working for any of us.

Should I just grab the latest install from ix and put that on the SSD and save my config and put that to the SSD as well? Then switch boot device in BIOS?

 

[Newfoundland.Republic]

That seems like a plan.
0. Download TrueNAS 12.0-U3
1. Back up your config (to 2 devices ).
2. Shutdown and remove the USB boot drive
3. Set the BIOS to book from NVMe
4. Reinstall TrueNAS to the NVE drives
5. Restore your configuration.

I'm paranoid, so I would make sure that my backup of my data was good, too.

 

[MGYVR]

That seems like a plan.
0. Download TrueNAS 12.0-U3
1. Back up your config (to 2 devices ).
2. Shutdown and remove the USB boot drive
3. Set the BIOS to book from NVMe
4. Reinstall TrueNAS to the NVE drives
5. Restore your configuration.

I'm paranoid, so I would make sure that my backup of my data was good, too.

That's basically what I did, but on my hardware I simply could not boot to any drive/pool on the 4x/4x/4x/4x PCIe NVMe adapter. So I ended up getting the internals to use the two 2.5" rear SATA/SAS "flex drive bays". Found some small/cheap commonly supported SATA SSDs, then did exactly as your list states, restored the sys config, and was golden.

 

[marcevan]

I'll have to unrack it for this to work but can do that later today.

Anyway since it boots from USB right now, I can prepare 12.0-U3 on a fresh USB and boot from that to install on my SSD, import config file, flip BIOS to SSD and test with no USB.