SOLVED Running TrueNAS Scale in Proxmox with physical disk passthrough with H710 in IT mode

[mishanw]

Hi Everyone, I have a Dell R720 and am new to homelabbing and TrueNAS including Proxmox.

I have been working with multiple challengers trying to pass my H710 controller in IT mode to TrueNAS Scale via Proxmox. I was able to get IOMMU turned on, I have Proxmox installed on a SSD thats connected to a SATA caddy that respectfully replaced the DVD drive. This allows me to have Proxmox and all VM's sit outside the H710, thus should give me the ability to pass the H710 as a PCI device to TrueNAS Scale. The issue I have seen is:

1) If I add the H710 as hardware to TrueNAS scale, then run the installer, it installs and goes on a loop to the install screen
2) If I install trueNAS and then add the PCI device, same issue.

I have checked the SeaBios to see the boot order and can not see an issue there. Do you have any thoughts on how to mitigate these issues?

Thank you kindly.

 

[Ericloewe]

SATA caddy that respectfully replaced the DVD drive.

That's okay, DVD drives don't have feelings, you can replace them with prejudice.

) If I add the H710 as hardware to TrueNAS scale, then run the installer, it installs and goes on a loop to the install screen

Are you trying to install to a disk connected to the HBA? That's not impossible, but it does involve extra hoops.

 

[mishanw]

That's okay, DVD drives don't have feelings, you can replace them with prejudice.


Are you trying to install to a disk connected to the HBA? That's not impossible, but it does involve extra hoops.

If I install on a disk on the HBA, everything goes smoothly. The issue is I want to install on the boot volume created for the VM by proxmox. Once I get through the installation. If the PCI hardware is passed as RAW, when booting into TrueNAS, it goes back to the install screen.

 

[Ericloewe]

So, let me see if I'm understanding this:

1. You setup a VM with an HBA via PCI pass-through, and a virtual disk to be used for booting.
2. The virtual disk shows up in installer and the installer seems to do its thing
3. The VM will not boot from the virtual disk
4. The VM is configured to boot from the virtual disk
5. If you remove the HBA from the VM, the boot process does work

 

[mishanw]

So, let me see if I'm understanding this:

1. You setup a VM with an HBA via PCI pass-through, and a virtual disk to be used for booting.
2. The virtual disk shows up in installer and the installer seems to do its thing
3. The VM will not boot from the virtual disk
4. The VM is configured to boot from the virtual disk
5. If you remove the HBA from the VM, the boot process does work

Yes. As long as the HBA is connected (In this case an H710 in IT mode) as a PCI device: See below

The VM does not boot into TrueNAS.The moment its removed, im able to boot in to the VM.

 

[Ericloewe]

Here's an idea: try wiping the Option ROMs on the card.

 

[mishanw]

Can you elaborate on that? The HBA is on IT mode.

 

[Ericloewe]

Besides the firmware, these controllers also take Option ROMs for BIOS and/or UEFI, which provide the needed functionality to manage the card (not super useful on a HBA) and allow for the card to be booted from by registering a hook for int13h (BIOS) or providing a suitable driver (UEFI).

Both OpROMs are optional on LSI SAS2 controllers, so you can just wipe them and remove a moving part that may be interfering with the boot process.

 

[mishanw]

Besides the firmware, these controllers also take Option ROMs for BIOS and/or UEFI, which provide the needed functionality to manage the card (not super useful on a HBA) and allow for the card to be booted from by registering a hook for int13h (BIOS) or providing a suitable driver (UEFI).

Both OpROMs are optional on LSI SAS2 controllers, so you can just wipe them and remove a moving part that may be interfering with the boot process.

Interesting. Whats the process to wipe the ROM? On Proxmox, I have setup TrueNAS to have SeaBIOS, the Dell its self has UEFI.

 

[Ericloewe]

It should be an option in sas2flash, check the manual.

 

[fohdeesha]

Interesting. Whats the process to wipe the ROM? On Proxmox, I have setup TrueNAS to have SeaBIOS, the Dell its self has UEFI.

replied to your email but putting it here for public visibility, to wipe option rms the card may have been pre-flashed with when cross flashed, boot into my freedos live ISO (https://fohdeesha.com/docs/perc.html#preparation) and run the following

sas2flsh.exe –o –e 5

the -o -e -5 means erase, option 5, which is bootloader. yes, sas2flsh should be missing the a

 

[Ericloewe]

I love how Software written for Hardware released long after 8.3 filenames stopped being a thing has to worry about 8.3 filenames because MS-DOS was the easiest environment to run said software in, despite the fact that it literally does not even support sufficient memory and it needs DOS4GW to work?

Though rants aside, would the erase not work the same from the EFI version of sas2flash?

 

[mishanw]

replied to your email but putting it here for public visibility, to wipe option rms the card may have been pre-flashed with when cross flashed, boot into my freedos live ISO (https://fohdeesha.com/docs/perc.html#preparation) and run the following

sas2flsh.exe –o –e 5

the -o -e -5 means erase, option 5, which is bootloader. yes, sas2flsh should be missing the a

Yup!!! Looks like that fixed the problem. One more hurdle overcome!! Thank you for the help.