Supermicro X11SDV-4C-TP8F setup with ESXi and FreeNAS - multiple passthrough

francisaugusto

Contributor
Joined
Nov 16, 2018
Messages
153
Hi,

I've decided to move my FreeNAS setup to a SuperMicro X11SDV-4C-TP8F.

My goal is:

- Boot from ESXi via an M.2 SSD or SATA DOM, perhaps even having most VM's stored here as well;
- Passthrough the SATA disks from the on-board connectors to FreeNAS
- Passthrough other disks to additional VM's for storage.

Since I will (hopefully) passthrough the SATA controller to the FreeNAS board, is adding a HBA to one of the PCI slots my only hope to provide additional storage to the other VM's (besides using the FreeNAS as storage)?

It's a bit of a pity because this board has two other connectors (called I-SATA0-3 and I-SATA4-7) that can be used for additional SATA ports or two NVMe U.2 ports, but I imagine that since I will use passthrough so that FreeNAS can use the controller, I won't be able to use these ports on a different VM, right?

I need one HDD (or SSD) as storage for the other VM's. What could I do?

- Would I be able to use the M.2 connectors to add other HDD's that won't be assigned to FreeNAS when I set up passthrough of the SATA controller? If so, can I get SATA ports via some sort of adapter?
- Is adding a HBA to one of the PCIe slots my only hope to get other SATA ports?

I'd be happy if I could get any tips on this setup. I would rather get the TLN2F variant, but because I want to have the possibility of passing through more hard disks to different VM's other than the FreeNAS one, I guess the TP8F is my only hope, unless I use the FreeNAS as storage for the other VM's, which I guess would be ok anyway.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Are you familiar with how ESXi works?

Based on the chipset diagram in the MB manual it looks like you would pass through the (4) SATA ports in one group and the (2) Mini-SAS ports in a different group.
https://www.supermicro.com/manuals/motherboard/d/MNL-2007.pdf

PCIe passthru doesn't work based on how ports are grouped or what kind of connectors they have. It works on how the device presents on the PCIe bus. If that controller is showing up as a monolithic 12 port controller, then you can't split it up. If it is showing as a 4 port and an 8 port, then you can.

The manual doesn't offer any obvious insight into this, though admittedly I only glanced through it. The Intel docs for the part should indicate what the story is.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
The manual doesn't offer any obvious insight into this, though admittedly I only glanced through it. The Intel docs for the part should indicate what the story is.
I based it on this from the manual
supermicro manual snap.PNG
 

francisaugusto

Contributor
Joined
Nov 16, 2018
Messages
153
Are you familiar with how ESXi works?

Based on the chipset diagram in the MB manual it looks like you would pass through the (4) SATA ports in one group and the (2) Mini-SAS ports in a different group.
https://www.supermicro.com/manuals/motherboard/d/MNL-2007.pdf

Yes, I am familiar with ESXi.
It isn’t clear, as @jgreco said, if the PCI controller will present itself as one, specially because elsewhere it’s said that there are three CPU PCIe, which made me wonder if the other connectors were under that “root” PCIe and therefore not subject to passthroigh individually.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I based it on this from the manual
View attachment 32061

Right, that's what I saw too. So it depends on what is providing "Flexible I/O". If that's a monolithic controller on the PCH, it'll be an all-or-nothing proposition. In some previous cases, Intel has also integrated an SCU into the PCH in addition to AHCI ("SATA") which would present as a separate PCIe device. Because this seems to be presenting as a conventional AHCI along with a multimode AHCI/NVMe interface, it isn't clear what's going on here. However, since it isn't offering SAS, the simple way to do a combined interface would be to have a bunch of AHCI interfaces and switching logic to make it able to switch to U.2. Therefore it's a very reasonable possibility it's a high portcount AHCI controller. This is the point where if I was doing this for a customer, I'd pull the Intel engineering docs and see if I could ferret it out, but that can be a lot of work.
 

francisaugusto

Contributor
Joined
Nov 16, 2018
Messages
153
Right, that's what I saw too. So it depends on what is providing "Flexible I/O". If that's a monolithic controller on the PCH, it'll be an all-or-nothing proposition. In some previous cases, Intel has also integrated an SCU into the PCH in addition to AHCI ("SATA") which would present as a separate PCIe device. Because this seems to be presenting as a conventional AHCI along with a multimode AHCI/NVMe interface, it isn't clear what's going on here. However, since it isn't offering SAS, the simple way to do a combined interface would be to have a bunch of AHCI interfaces and switching logic to make it able to switch to U.2. Therefore it's a very reasonable possibility it's a high portcount AHCI controller. This is the point where if I was doing this for a customer, I'd pull the Intel engineering docs and see if I could ferret it out, but that can be a lot of work.

I got my machine yesterday, and I also bought a HBA as precaution.

It seems promising to passthrough only some of the SATA controllers to the FreeNAS. I still need to do some testing, but I see two SATA controllers under ESXi. I toggle the passthrough for the second one, and connected harddisks via the miniSAS plug, and they were immediately imported by FreeNAS (it was a preexistant pool). I also plugged a SSD directly onboard, which is shown as storage by ESXi. The two HDD I connected to the miniSAS do not show as storage under ESXi, so I suspect that a "partial" passthrough was achieved, though I think I saw the same SSD showing up while booting FreeNAS, but I don't find this SSD under the FreeNAS gui. I will check more.
 

nojstevens

Cadet
Joined
Oct 2, 2019
Messages
5
I just ordered this exact same mobo to do ESXi and Freenas... looking forward to the build. Let me know if there are any things to be aware of. I hear cooling might be an issue so ordered a NH-U12S fan to go on the Xeon.
 

francisaugusto

Contributor
Joined
Nov 16, 2018
Messages
153
Top