Need Help Booting TrueNAS Scale from SSD in PCI Slot on Dell R730XD

[atom5ive]

Hello everyone,
I'm facing a challenging issue trying to boot TrueNAS from an SSD located in a PCI slot on my Dell R730XD server. I've been at this for a while and would greatly appreciate any insights or suggestions.
The Problem:

• I've installed TrueNAS on an SSD which is housed in a PCI slot.
• The Dell R730XD doesn't seem to recognize the PCI card holding the SSD with the TrueNAS OS on it.
• I've been using Clover (as it was recommended, as the Dell BIO's doesn't see the PCI drive) to try and boot from this SSD, but I'm running into issues.

What I've Done So Far:

1. Clover UEFI Shell: I've accessed the Clover UEFI shell to try and manually boot from the SSD. The mapping table does recognize two PCI devices, and I can see the path to the EFI bootloader (EFI\debian\grubx64.efi
   ). However, when I try to execute the bootloader, it just loops back to the Clover main menu.
2. Commands Tried: In the UEFI shell, I've tried various commands like dir, ls, cd, load, and bootx64, but they either don't work or don't produce the desired result.
3. Firmware Update: I've updated the UEFI firmware of the Dell R730XD to the latest version, hoping it might resolve any compatibility issues.
4. Research: I've consulted various resources, including Clover documentation, UEFI shell command references, and TrueNAS community posts to find a solution. I've also tried UEFI and using stand BIO's boot, neither seem to work.
5. I've been able to boot to USB and then install TrueNas Scale with absolutely zero issues, it sees the SSD in the PCI slot with no issues and installed with no errors. Clover also see the PCI card as well. When I reboot, it won't boot into TrueNas as Dell's BIO's does not see the drive to boot to, hence using clover.

My Setup:

• Server: Dell R730XD
• OS: TrueNAS Scale
• Boot Drive: SSD in PCI slot
• UEFI Firmware Version: 2.40

Has anyone encountered a similar issue or have any suggestions on how to proceed? I'm trying to avoid pulling out the drive if possible and would like to resolve this all via the UEFI shell or other means. Any workarounds, insights, or advice would be immensely helpful.

Also, here are some helpful outputs:


FSO: bcfg boot dump
Option: 00. Variable: Boot0006
Desc - TrueNAS-0
Dev Path
HD (2,GPT, 12463820-82B2-41D5-847A-F017DBFD160F,0x1800,0x100000) /\EFI\
debian\grubx64.efi
Optional- N
Option: 01. Variable: Boot0001
Desc - USB 3.0 FD
Dev Path PciRoot (0x0) /Pci (0x10, 0x0) /USB (0x0, 0x0) /USB (0x1,0x0)
Optional- N
Option: 02. Variable: Boot0000
Desc - OEMDRU
Dev Path - PciRoot (0x0) /Pci (0x1A, 0x0) /USB (0x0, 0x0) /USB (0x5,0x0) /USB (0x1,0x0) Optional- N
Option: 03. Variable: Boot0005
Desc - Integrated NIC 1 Port 1 Partition 1
Dev Path - VenHw (3A191845-5F86-4E78-8FCE-C4CFF59F9DAA) Optional N

As well, the mapping table.

UEFI Interactive Shell v2.205 EDK II
UEFI v2.40 (Dell Inc., 0x00110201)
Mapping table
FSO: Alias(s):HD0a0f0b0b::BLK1:
Pci Root (0x0)/Pci (0x1A,0x0) /USB (0x0, 0x0) /USB (0x5,0x0) /USB (0x1,0x0) /HD (1 MBR, 0x444D454F,0x1,0x99FFF)
FS1: Alias(s):HD1a0b0b::BLK3:
Pci Root (0x0) /Pci (0x1D, 0x0) /USB (0x0, 0x0) /USB (0x1,0x0) /HD (1,MBR, 0x00925C 52,0x3F,0x64000)

BLKO: Alias(s):
Pci Root (0x0) /Pci (0x1A, 0x0) /USB (0x0, 0x0) /USB (0x5,0x0) /USB (0x1,0x0) BLK2: Alias(s):
PciRoot (0x0) /Pci (0x1D, 0x0) /USB (0x0, 0x0) /USB (0x1,0x0) BLK4: Alias(s):
PciRoot (0x0) /Pci (0x1D, 0x0) /USB (0x0, 0x0) /USB (0x1,0x0) /HD (2,MBR. 0x00925C 52,0x6403F,0x395BFC1)

 

[HoneyBadger]

Hi @atom5ive

The R730 series (vanilla or XD) appears to be rather particular on its willingness to boot from PCIe/NVMe - the general Internet wisdom does lead towards "chain-loading" from a USB device, with or without the Clover UEFI shim.

The "Official Dell Supported" method would be to use the BOSS-S1 or -S2 card, some other users have reported success with these (although it requires some firmware updates, and UEFI-only booting):

ahci driver problems with Marvell 88SE9230 (Dell BOSS-S1)

Hi All, I'm copying the title of a bug report submitted to FreeBSD, which hasn't got much traction there, in the hope that someone here can help. I have three Dell PE R740xd2 servers with these BOSS cards in, all running FreeNAS 11.3, and can verify that the latest firmware for the BOSS...

www.truenas.com

And the final piece that I read is that apparently Intel Optane memory devices contain the necessary option ROM or special sauce needed to get the R730 to boot - although the source on that discussion is a Reddit post that is a little light on the details of the exact Optane model.

Reddit - Dive into anything

www.reddit.com

Hope some of this helps!

 

[Ericloewe]

and UEFI-only booting

By the time NVMe boot became relevant, the industry consensus was that UEFI was it and legacy CSM support for NVMe was pointless, so int 13h support for NVMe disks is atypical, if it exists at all. Windows doesn't boot from NVMe in BIOS mode, either, if you figured out a workaround. So the UEFI-only part really goes without saying.
Sidenote: the BOSS cards are actually SATA.

Further sidenote: Have you guys considered getting rid of GRUB in favor of something like ZFS Boot Menu? It's not a huge deal with TrueNAS, since the boot pool is separate anyway and zpool compatibility settings get rid of the dirty hack that used to be necessary to keep the boot pool from accidentally having newer features enabled.
Fun fact/sidenote to the sidenote: The other day I set up a test machine (Ubuntu 22.04) with an X10SLM+-LN4F and wanted to boot from NVMe. Since the X10SL* series does not have the NVMe driver in firmware, I figured I'd install ZBM to a USB flash drive. It was pleasantly painless.

Has anyone encountered a similar issue or have any suggestions on how to proceed?

Fundamentally, it sounds like something around Clover. Conceptually, what you need to do is quite simple in UEFI land:

1. Load an NVMe driver
2. Hand over execution to bootx64.efi on the NVMe device

Modern systems do step 1 themselves. For older systems, a simple shim is needed. With that in mind:

• rEFInd or something similar should be able to do what you need, but you need to provide the NVMe driver
• Clover is a bit more complex, but seems to have an NVMe driver, though not by default
• My side-sidenote above, using ZFS Boot Menu, was fairly painless because ZBM is really a Linux distro with a smallish kernel (with ZFS, console, networking and storage drivers, and few other bells and whistles), so it has the normal Linux NVMe driver included in the image that gets loaded by the system firmware.

 

[atom5ive]

Thank you for the detailed solutions. Here's where I currently stand:

1. R730 Series Booting from PCIe/NVMe:
   • I've been trying to use the Clover UEFI shim to "chain-load" from a USB device. However, I've been encountering issues with recognizing the correct boot path and executing it.
2. Dell Supported Method - BOSS-S1 or -S2 Card:
   • I'm not using the BOSS-S1 or -S2 card. Instead, I've opted for the ICY DOCK 2.5in SATA Removable Hard Drive Bay Adapter for the PC Expansion Card Slot. While it's not the BOSS card, the system does seem to recognize it, which leads me to believe it's a configuration or software issue rather than a hardware compatibility problem.
3. UEFI-only Booting:
   • I've ensured that the system is set to boot in UEFI-only mode. The challenge has been getting the system to recognize and boot from the NVMe drive correctly.
4. ZFS Boot Menu:
   • Given the challenges I've faced with Clover, I'm considering the ZFS Boot Menu as a potential solution. The project you mentioned, ZFS Boot Menu, seems promising. Do you think transitioning to this would simplify the process and increase the chances of a successful boot from the NVMe drive?

I appreciate all the insights and suggestions. If there's any additional information or steps you think might help, please let me know.

 

[atom5ive]

I've been trying to follow this article but have been running into issues where clover just reboots into itself when putting in the correct UUID in the config.

 

[atom5ive]

This has been solved using ZFS Boot.

 

[Ericloewe]

Did it work out of the box or did you have to configure additional kernel command line parameters?

 

[atom5ive]

Did it work out of the box or did you have to configure additional kernel command line parameters?

Out of the box. Zero setup, zero config. The only setup I did was ON the actual USB itself. The websites docs were lacking to my ability to understand how to make the bootable USB in the correct format and the directory structure setup right, but I was able to make something work of it, somehow, lol.