• Currently using VMware at work? We want to hear from you.

    Thinking of making a switch from VMware? We'd love to hear your thoughts and feedback about which hypervisor you have been researching or already using. Click here to vote and share your thoughts! You can vote HERE!

SOLVED Running TrueNAS Scale in Proxmox with physical disk passthrough with H710 in IT mode

mishanw

Cadet
Joined
Feb 5, 2024
Messages
6
Hi Everyone, I have a Dell R720 and am new to homelabbing and TrueNAS including Proxmox.

I have been working with multiple challengers trying to pass my H710 controller in IT mode to TrueNAS Scale via Proxmox. I was able to get IOMMU turned on, I have Proxmox installed on a SSD thats connected to a SATA caddy that respectfully replaced the DVD drive. This allows me to have Proxmox and all VM's sit outside the H710, thus should give me the ability to pass the H710 as a PCI device to TrueNAS Scale. The issue I have seen is:

1) If I add the H710 as hardware to TrueNAS scale, then run the installer, it installs and goes on a loop to the install screen
2) If I install trueNAS and then add the PCI device, same issue.

I have checked the SeaBios to see the boot order and can not see an issue there. Do you have any thoughts on how to mitigate these issues?

Thank you kindly.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
SATA caddy that respectfully replaced the DVD drive.
That's okay, DVD drives don't have feelings, you can replace them with prejudice.

) If I add the H710 as hardware to TrueNAS scale, then run the installer, it installs and goes on a loop to the install screen
Are you trying to install to a disk connected to the HBA? That's not impossible, but it does involve extra hoops.
 

mishanw

Cadet
Joined
Feb 5, 2024
Messages
6
That's okay, DVD drives don't have feelings, you can replace them with prejudice.


Are you trying to install to a disk connected to the HBA? That's not impossible, but it does involve extra hoops.
If I install on a disk on the HBA, everything goes smoothly. The issue is I want to install on the boot volume created for the VM by proxmox. Once I get through the installation. If the PCI hardware is passed as RAW, when booting into TrueNAS, it goes back to the install screen.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
So, let me see if I'm understanding this:
  1. You setup a VM with an HBA via PCI pass-through, and a virtual disk to be used for booting.
  2. The virtual disk shows up in installer and the installer seems to do its thing
  3. The VM will not boot from the virtual disk
  4. The VM is configured to boot from the virtual disk
  5. If you remove the HBA from the VM, the boot process does work
 

mishanw

Cadet
Joined
Feb 5, 2024
Messages
6
So, let me see if I'm understanding this:
  1. You setup a VM with an HBA via PCI pass-through, and a virtual disk to be used for booting.
  2. The virtual disk shows up in installer and the installer seems to do its thing
  3. The VM will not boot from the virtual disk
  4. The VM is configured to boot from the virtual disk
  5. If you remove the HBA from the VM, the boot process does work
Yes. As long as the HBA is connected (In this case an H710 in IT mode) as a PCI device: See below

The VM does not boot into TrueNAS.The moment its removed, im able to boot in to the VM.
1707253662422.png
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
Here's an idea: try wiping the Option ROMs on the card.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
Besides the firmware, these controllers also take Option ROMs for BIOS and/or UEFI, which provide the needed functionality to manage the card (not super useful on a HBA) and allow for the card to be booted from by registering a hook for int13h (BIOS) or providing a suitable driver (UEFI).

Both OpROMs are optional on LSI SAS2 controllers, so you can just wipe them and remove a moving part that may be interfering with the boot process.
 

mishanw

Cadet
Joined
Feb 5, 2024
Messages
6
Besides the firmware, these controllers also take Option ROMs for BIOS and/or UEFI, which provide the needed functionality to manage the card (not super useful on a HBA) and allow for the card to be booted from by registering a hook for int13h (BIOS) or providing a suitable driver (UEFI).

Both OpROMs are optional on LSI SAS2 controllers, so you can just wipe them and remove a moving part that may be interfering with the boot process.
Interesting. Whats the process to wipe the ROM? On Proxmox, I have setup TrueNAS to have SeaBIOS, the Dell its self has UEFI.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
It should be an option in sas2flash, check the manual.
 

fohdeesha

Dabbler
Joined
Nov 14, 2016
Messages
14
Interesting. Whats the process to wipe the ROM? On Proxmox, I have setup TrueNAS to have SeaBIOS, the Dell its self has UEFI.
replied to your email but putting it here for public visibility, to wipe option rms the card may have been pre-flashed with when cross flashed, boot into my freedos live ISO (https://fohdeesha.com/docs/perc.html#preparation) and run the following

sas2flsh.exe –o –e 5

the -o -e -5 means erase, option 5, which is bootloader. yes, sas2flsh should be missing the a
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,048
I love how Software written for Hardware released long after 8.3 filenames stopped being a thing has to worry about 8.3 filenames because MS-DOS was the easiest environment to run said software in, despite the fact that it literally does not even support sufficient memory and it needs DOS4GW to work?

Though rants aside, would the erase not work the same from the EFI version of sas2flash?
 

mishanw

Cadet
Joined
Feb 5, 2024
Messages
6
replied to your email but putting it here for public visibility, to wipe option rms the card may have been pre-flashed with when cross flashed, boot into my freedos live ISO (https://fohdeesha.com/docs/perc.html#preparation) and run the following

sas2flsh.exe –o –e 5

the -o -e -5 means erase, option 5, which is bootloader. yes, sas2flsh should be missing the a
Yup!!! Looks like that fixed the problem. One more hurdle overcome!! Thank you for the help.
 
Top