ESX (U)EFI NVMe passthrough woahs

kfason

Cadet
Joined
Jul 25, 2018
Messages
4
Happy New years everyone!. I get that I am on the fringes with this but thought I'd see if anyone had advice.

With that said, I used to have FreeNAS on a Dell R510 12 bay with H200 HBA and S3500 L2ARC and S3700 SLOG. 12C24T L5640 and 96GB memory. For compute I was using a R710 with nvme via PCIe running ESX6.7U1 with 10GB SFP+ between them. VMs internal and on the FreeNAS along with Samba for media. Worked great and still would work great. I've since downsized a lot so when I got a R720xd (20C40T,256GB). I thought I would go from two hosts to one and virtualize FreeNAS and size it more for my use case via ESX 7U1. This works great with FreeNAS 11.3U3. The VM is 6C/32GB with H710 mini HBA. Installed and restored config process.

I in turn dropped the L2ARC and I thought I'd move the SLOG to NVMe so per all the great info here settled on an 800p as is reasonably priced and should give me some longevity for this hardware cycle. This is where the frustration started. After passing it through, FreeNAS would not boot. It would immediately shutdown.

2021-01-07_9-15-16.png


At this point I decided to leave this VM alone and fire up a new one to start fresh to keep impact low and allow reproducible change control. 4C/8GB with 2 VMDKs that I'd have in a mirror within FreeNAS. Same issue so tweaked the passthru.map and vmx file per this 900p FreeNAS bug with same results. If I remove the PCI passthrough the VM works fine. So then installed 12U1. This got me a little farther but during boot it would just have a timeout issue.

image.png


After trying many other things I re-installed the VM with BIOS instead of EFI as I run into EFI vs BIOS issues at work due to BIOS being more mature and more understood. It works fantastically now. Have it running with a mirror (via a couple VMDKs) and the 800p passed through as SLOG. So after all this time its something with EFI instead of BIOS via ESX. Before I sold it, I put an M10 16GB in the R510 and it worked fine on bare metal with 12U1 in EFI.

Obviously, I can reinstall the main VM in BIOS (And upgrade to 12U1) and import config and go about my day however with others having similar issues on the forum with nvme passthrough I want to contribute my experience and ask others for advice here. I'd rather move forward vs back.

I do intend to install TrueNAS 12U1 on this R720xd bare metal with the 800p to see how it behaves. I just need to plan a downtime since it provides alot.

What is different for FreeNAS between EFI and BIOS in relation to the boot command and switches? Are the kernels installed any different?

wacky thing is that the installer ISO for 12U1 sees the NVMe and will install to it, but it wont boot either. Whats different about the kernel on the ISO vs whats installed? Perhaps this is key and some module is not present in the installed kernel?

2021-01-07_8-59-17.png


I thought I might try the FreeBSD version TrueNAS is built on the see how it behaves as a comparison. Also, revisit VMWare EFI documentation to see if there are any .vmx tweaks to made there. Maybe the EFI framebuffer points to something there since that's the last thing outputted during boot.

Any other suggestions?
 
Top