TrueNAS Scale in Proxmox ignores install when passed through PCI-E LSI HBA

[Nuravictus]

I was running current TrueNAS Core Stable release 13.0-U3.1
I decided to try to TrueNAS Scale now that it is a stable release.
Loaded TrueNAS Scale 22.12 Stable version without issue. Restarted to confirm.
Shutdown and passed through HBA - LSI 9207-8i that is running the latest firmware.
When attempted to reboot the system will not recognize there is an active install and will boot hang or attempt to reinstall.
If I force shutdown the VM and remove the HBA and reboot, the system boots as normal.
If I attempt to reinstall it will put itself into an reinstall loop.
With this I also magically seem to loose the ability to roll back.

My server pc is a
CPU: Intel i7 12700k
MOBO: Asus TUF Gaming z690-Plus WIFI D4
Ram: 32GB DDR4 3200
HBA: LSI 9207-8i

 

[jgreco]

Set the boot ordering for the VM correctly. Just like your host system, the VM has its own expectation of where to find boot sources; if you fail to set this up correctly, it may simply be trying to boot FROM the HBA, which is an entirely reasonable thing to do, since the HBA may appear to be a preferential source over a mere SATA HDD or whatever you've defined the VM's virtual boot disk to be in Proxmox.

Also note that Proxmox only has experimental support for PCIe passthru, and you're attempting this on what appears to be a gaming board. There's no real reason to expect this to magically work out of the box.

See

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[Nuravictus]

Just thinking out loud here but could this have to do with my card booting in the bios? If so how would I turn that off?

 

[Nuravictus]

Set the boot ordering for the VM correctly. Just like your host system, the VM has its own expectation of where to find boot sources; if you fail to set this up correctly, it may simply be trying to boot FROM the HBA, which is an entirely reasonable thing to do, since the HBA may appear to be a preferential source over a mere SATA HDD or whatever you've defined the VM's virtual boot disk to be in Proxmox.

Also note that Proxmox only has experimental support for PCIe passthru, and you're attempting this on what appears to be a gaming board. There's no real reason to expect this to magically work out of the box.

I see your point and 100% valid.
I did ensure the drive that held the system was set for #1 in the boot order, I even tried to disable to hba to boot in the startup. This did not effect any change unfortunatly.

 

[jgreco]

The image included above shows the LSI's boot order as zero, which I would think offhand would be highest priority.

 

[Nuravictus]

jgreco - you are the .......ummmm......grinch? !!!! aaaa helll, you rock!!! So it was the LSI booting ahead of the drive and hanging everything. As you can see the main drive is first, then the iso for install, then random proxmox stuffs, then the hba that is deselected. This changed nothing for me.

The ticket was editing my PCI device, deselecting all functions and rom bar. I tried one and then the other and for shear giggles I tried both and it went through. I don't understand it, I really don't but it works!!!!

 

[Nuravictus]

I have re attempted this with 22.12 and it is working as well so some weird thing my HBA card picked up since before I had to use all functions and rom-bar as selected......Talk about a complete turn around lolz

 

[jgreco]

I don't understand it,

So I can probably get away with deadpanning "New to servers, are ya."

This is just one of those stupid things where various designers work towards default configurations that they think make sense for weird use cases. My guess would be that the HBA appears to be some sort of highly reliable boot source to the BIOS, and the BIOS prefers that.