Problem with Device Passthrough on ESXi 6.5

[ChrisNAS]

Hello,

Has anyone faced a device list in esxi 6.5 where all devices are "not capable" for passthrough? I have a fresh install of esxi and this is the case for me. The virtualization setting in the MB bios is set to enabled. I can see all the devices including drives and LSI 9211 adapter which was flashed to IT mode, but all devices are disabled for passthrough. I've been searching all day, but seems not many people having this problem because no solution yet.

Any ideas on how to fix this would be very much appreciated.

Thank you and Happy Holidays!

 

[Xelas]

Please list your FULL specs. The ones relevant here are CPU, motherboard, and all add-on cards, and which EXACT build of ESXi you are trying to use. ESXi 6.5 has had a handful of updates, and some of the earlier 6.5 releases were really buggy. Additionally, the Web GUI is a separate snap-in (what they call a "fling") that has a ton of bugs in it as well, and can be updated
independently of ESXi itself.

Some of the bugs can be bad enough to corrupt your VM's config files!! You have to be really careful with it.

Is this a fresh install, or an upgrade from a prior version of ESXi? What are you VM machine versions?

Also describe your drive layout, and how, exactly, you are booting into ESXi, how you have your datastores in ESXi, attached, and how you are booting into FreeNAS (from inside ESXi as a VM? Where is it's datastore?)

FWIW - I've been running FreeNAS as a VM in ESXi from 5.5 --> 6.0 --> 6.5, and have had an LSI controller in passthrough since day one with no real issues. I've had to fight with bugs in the Web GUI in ESXi 6.5, but I still think it's a HUGE advance of needing to run a fat client on a WIndows PC to manage anything.

EDIT: Here's proof of it working perfectly on my setup:

 

[RodyMcAmp]

also, virtulization usually has options in the bios in two different places, 1 enables virtualization vt-x the other is usually located in some strange out of the way place but it enables vt-d and you need both options for passthrough to work.

 

[ChrisNAS]

Thanks for the replies guys.

The system is not ideal... budget setup for a friend. I've been searching still all afternoon and starting to think the motherboard is just crap... or maybe just the bios is crap... or both. There's literally just "Secure Virtualization Mode" under cpu and I enabled that. I have gone through every bios option and see nothing else for V.

6.5.0 (Build 4887370)
M5A78L-M PLUS/USB3
FX8320E
LSI 9211-8i (flashed IT)
2 SSDs directly off MB
2 HDDs on LSI
Intel Dual Port

ESXi installed on one of the SSDs (did fresh re-install too). I literally just installed ESXi as new install, didn't create any VMs or mess with any datastores just went right to devices to make sure LSI card is there and found the list of items all unusable so I didn't go on to trying to install any VMs.

Thanks again for your replies.

 

[Xelas]

Ouch - FreeNAS can be incredibly unstable on AMD machines due to rather poor driver support. I'd strongly suggest ditching that and trying a supported solution. A lot of people have good luck on these forums running it on an HP Microserver or other low-end solutions. These can be had fairly cheaply and will work MUCH better - just max out the RAM on the thing. The power of the CPU isn't as critical as a well-supported platform (CPU/chopset) and no funky or low-rent IO adpaters or NICs. Again, that doesn't mean they have to be expensive!

Even a second-hand server pulled from EBay, if you know what to look for, can be a MUCH better fit than a new AMD build.

EDIT: If you're stuck with the hardware, you may also, possibly, have better luck running a disto such as Ubuntu on it (since Linux tends to be ahead of FreeBSD on drivers) and running it with ZFS, then deploying VMs on that with Docker. ZFS is fairly stable now-a-days on Linux.

EDIT2: VT-d is an Intel-only feature. AMD may have a similar feature, but it's probably called something different, and support for it may or may not be as well established.

 

[toadman]

I have found FreeNAS to be stable on my AMD systems. Been running it on AMD for over 5 years, both bare metal and as a VM. At least for my hardware.

The problem the OP has is that only the 900 series chipsets support IOMMU (I think), so obviously PCIe passthrough is not going to work on his 700 series board. I had to toss an 890 board and get a 970 board (I also have a 990, both ASUS) when I moved to ESXi. Having made that move, I can attest that passthrough worked with both ESXi 6.0 and 6.5.

 

[ChrisNAS]

Hope you all had a nice Christmas!

So I'm back in the hunt for a different motherboard+cpu+ram setup. Any suggestions that will work with esxi and be micro atx 9.6"?

EDIT:

Thinking this assuming friends budget affords:

Supermicro X11SSL-NF-O
Xeon E3-1220 V6
Samsung Memory - 16GB DDR4-2133 2Rx4 LP ECC

 

[Xelas]

Hope you all had a nice Christmas!

So I'm back in the hunt for a different motherboard+cpu+ram setup. Any suggestions that will work with esxi and be micro atx 9.6"?

EDIT:

Thinking this assuming friends budget affords:

Supermicro X11SSL-NF-O
Xeon E3-1220 V6
Samsung Memory - 16GB DDR4-2133 2Rx4 LP ECC

I strongly suggest looking at this board instead:
https://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSL-CF.cfm

It's a few bucks more, but it includes the LSI3008 SAS/SATA controller, which usually retails for $40-70 anyway and would take up one of the few slots the board has (limiting future expansion options). If you're planning to run ESXi, this will be a HUGE help - you'll be able to passthrough that controller straight to FreeNAS. Just grab some SAS/SATA breakout cables, and use the shortest cables that will comfortably work with your chassis. Longer cables are just more mess to organize.

My dream board, though, would be this one:
https://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSH-CTF.cfm

It's ~ $150 more, but it gets you 2x10Gb interfaces instead of 2x1Gb, and the C236 instead of the C232 chipset, which, if you get the E3-1225 CPU, will give you access to Intel Quicksync. If you're planning on running Plex with Plex Pass on this thing (within ESXi, not within a jail, so that you can passthrough the Intel video as well), you'll get the option of using hardware transcoding instead of software transcoding, which means you'll be able to transcode and stream high-quality video, or mulitple videos, without running out of CPU power.
Also, the 10GB interfaces really extend the life of this board. 1GB interfaces are the bottleneck for most NAS systems and are getting dated for decent new NAS builds with any substantial throughput. A stand-alone add-on Intel 2-port 10GB NIC with the same x550 chip costs $300+ (almost the price of the whole board!), so this board is actually a really good deal for the money. 10Gb switches are starting to come down in price a bit, FINALLY, and prices will continue to fall. I would plan to keep this board around at LEAST 5-6 years, so keep that in mind.

To illustrate the bottlenecking - my NAS, which is uses unspectacular and aging 3TB WD Red drives, has no SLOG or L2 ARC, has only 12GB of RAM allocated to FreeNAS (netting a ~9GB ARC), and no special tricks or tuning such as jumbo frames, can sustain reads and writes at 250-300 MB/s. I move 20-40GB files to/from the array all the time between the FreeNAS and other VMs through the software-only ESXi switch, which has no speed limitations. My array is about 65% full, by the way, too.

 

[ChrisNAS]

Xelas, thanks for the info. That's awesome.

It totally makes sense to at least get that CF motherboard as then can return the lsi 9211 card, which I don't like as it heats up and is quite large for a microatx.

Thanks for the tip on that. I also assume the onboard sas controller is a better quality than the old lsi 9211, yea? Do you know if I'd be able to pass through the drives without any modification? Like with the 9211 had to flash to IT mode.

Lastly, SM doesn't list a whole lot of ram options under the tested for the x11 board, do you know if this ram is a good/compatible set? Supermicro Certified MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 2Rx4 LP ECC REG RoHs

Thanks again for the tips and info.

 

[Xelas]

Xelas, thanks for the info. That's awesome.

It totally makes sense to at least get that CF motherboard as then can return the lsi 9211 card, which I don't like as it heats up and is quite large for a microatx.

Thanks for the tip on that. I also assume the onboard sas controller is a better quality than the old lsi 9211, yea? Do you know if I'd be able to pass through the drives without any modification? Like with the 9211 had to flash to IT mode.

Lastly, SM doesn't list a whole lot of ram options under the tested for the x11 board, do you know if this ram is a good/compatible set? Supermicro Certified MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 2Rx4 LP ECC REG RoHs

Thanks again for the tips and info.

You'll still need to reflash the controller to get it into IT mode. No getting around that. It's a minor PITA, but you only ever have to do this once for the life of the board. There are several decent walk throughs on the internet, and it's not too bad.

As for RAM - check the supermicro ram compatibility list for your board, and never buy Kingston Ram. If you'll be running VMs in addition to FreeNAS, you'll need 32GB ram. If you want to start off without VMs for now and want to save a few pennies, get 2x8 GB ram and leave room for expansion.
If you are new to this, suggest starting off without Esxi and just run FreeNAS natively, get familiar with it, then introduce the complexity of also managing Esxi.
Just keep the pool drives on the LSI controller.

If this is helpful, please use the Thanks button (applies to any good post in these forums).

 

[Ericloewe]

You'll still need to reflash the controller to get it into IT mode.

It's already in IT mode. It just needs to be updated.