Virtualize TrueNAS, SATA Controller passthrough question.

[testcb00]

Hi everyone, recently I get a 1U Supermicro X9 Xeon server. I would like to make a virtualization server using this hardware. TrueNAS will act as a storage server plus some Jails service in my design.

The VM Hypervisor will be ESXi or Proxmox VE. In the SATA Controller, I have two options. One is passthrough the normal Intel AHCI (SATA) controller, another is to passthrough the Intel SCU (S-SATA) controller.

I understand that I should not passthrough the on-board controller. However, I do not have any choice as I have to install Network Card on the only PCIe slot.
Any suggestions are welcome, thank you.

 

[Spearfoot]

Hi everyone, recently I get a 1U Supermicro X9 Xeon server. I would like to make a virtualization server using this hardware. TrueNAS will act as a storage server plus some Jails service in my design.

The VM Hypervisor will be ESXi or Proxmox VE. In the SATA Controller, I have two options. One is passthrough the normal Intel AHCI (SATA) controller, another is to passthrough the Intel SCU (S-SATA) controller.

I understand that I should not passthrough the on-board controller. However, I do not have any choice as I have to install Network Card on the only PCIe slot.
Any suggestions are welcome, thank you.

There's a good chance you can pass through the motherboard SATA controller, see my post from 2017:

Configure ESXi to pass-through the X10SL7-F motherboard SATA controller

By default, VMware ESXi 6.0 does not make the X10SL7-F motherboard's SATA controller available for VT-d passthrough. Because who in their right mind would do such a thing? :-) Well... I do! I've never used the motherboard's SATA controller; I boot from a Dell H200 HBA. Recently I wanted to add...

www.truenas.com

I've never tried passing through an SCU controller, but that might work too.

 

[Evertb1]

I've never tried passing through an SCU controller, but that might work too.

My lab server is running ESXi 7.0. The motherboard is a SuperMicro X9SRI-3F. It also contains a SCU (with four ports populated for SATA). Both the SATA AHCI controller and the SCU can be toggled for passthrough. See picture below of my ESXi hardware overview.
Now I don't know if the SCU behaves like a HBA or if you are able to boot from it (never used it) , but you have a choise what device you give to the TrueNAS VM. As you can see, on the picture both devices are black and thus not grayed out. In fact I have toggled the SCU for passthrough.

 

[jgreco]

I understand that I should not passthrough the on-board controller.

Where'd you get this tidbit from? If someone said that and actually meant it as a general rule, I suggest not listening to them in the future, they're an idiot.

What you DO need is to have some sort of storage controller for ESXi to run from. This is often an onboard controller, in which case passing it through will cause all sorts of badness, but you can potentially boot from other sources, in which case the onboard controller(s) are free for passthru. We run all ESXi datastores as LSI MegaRAID RAID1 on stuff like 9270CV's etc., so the onboard SATA and SCU's are always available for PCI passthru, and are often used as such here. The SCU's typically show up as isci.

isci0: <Intel(R) C600 Series Chipset SAS Controller (SATA mode)> port 0x6000-0x60ff mem 0xe7afc000-0xe7afffff,0xe7400000-0xe77fffff irq 16 at device 0.0 on pci10
[...]
da9 at isci0 bus 0 scbus4 target 0 lun 0
da9: <ATA WDC WD120EMFZ-11 0A81> Fixed Direct Access SPC-3 SCSI device
da9: Serial Number [blabla]
da9: 300.000MB/s transfers
da9: Command Queueing enabled
da9: 11444224MB (23437770752 512 byte sectors)

Do note that you can break yourself horribly if you pass thru a controller that you need to have available to boot/run from. ESXi runs "in memory" so when you set your SATA controller (that you are booting from) to PCI passthru, it will write that to your boot partition, tell you to reboot your hypervisor, and on reboot it will no longer see the boot partition, so you cannot undo the change and write it back to the boot partition.

The VM Hypervisor will be ESXi or Proxmox VE.

Proxmox's support for PCI passthru is still listed as experimental, and Proxmox itself may offer some challenges. ESXi has been rock solid for many years.