bhyve nvme SSD passthrough

[Zofoor]

Hi all!

Recently I've found myself adding my first VM on a FreeNAS 12 bare metal.
It is a Windows 2019, where I would run the software "roon".

I am not happy with the disk performance, and trying to use the VirtIO drivers does not make the system boot anymore.
So, following the "roon" hardware guidelines, I want to dedicate an SSD to this VM. In this case my NVME.

Running pciconf -v -l allows me to identify the device:
"nvme0@pci0:4:0:0: class=0x010802 card=0xa801144d chip=0xa808144d rev=0x00 hdr=0x00"

So I added the following tunables (type loader):
vmm_load =true
pptdevs2 = 0/4/0
vmm_after = kldload if_re
if_re_load = NO

Then I rebooted and, if I understood correctly, the device should be not showed in dmesg.
So I run...
dmesg | grep nvme
nvme0: <Generic NVMe Device> mem 0xdf400000-0xdf403fff irq 16 at device 0.0 on pci4
nvme0: <Generic NVMe Device> mem 0xdf400000-0xdf403fff irq 16 at device 0.0 on pci4

The NVMe is also still visible and could be used to create a new pool, so it seems it has not been reserved for bhyve.

On the VM section of TrueNAS I don't see the nvme device
I tried then the following commands, curious that the same input generated different outputs...

root@freenas:~ # bhyve -S -s 7,passthru,0/4/0 WinServer
virtual machine cannot be booted
root@freenas:~ # bhyve -S -s 7,passthru,0/4/0 WinServer
bhyve: PCI device at 0/4/0 is not using the ppt(4) driver
device emulation initialization error: No such file or directory
root@freenas:~ # bhyve -S -s 7,passthru,0/4/0 WinServer
Unable to setup memory (17)
root@freenas:~ # bhyve -S -s 7,passthru,0/4/0 WinServer
Unable to setup memory (17)
root@freenas:~ # bhyve -S -s 7,passthru,0/4/0 WinServer
Unable to setup memory (17)

Any advice? I would really like to avoid implementing ESXi just for a single VM (an additional system to manage, keep updated, etc).

After rebooting the system, I run the following command
bhyve -S -s 7,passthru,0/4/0 WinServer
Unable to setup memory (17)

 

[zizzithefox]

Well, I have never tried windows 2019 on bhyve. I know I can setup Windows 2016 with virtio drivers and it works fine but you must manually install the disk and network driver from the Fedora virtio CD on a working AHCI setup. So, you have to install on AHCI, then add a virtio disk (a small file raw disk will do), install the driver, reboot, shutdown, change the type of the boot disk to virtio, boot, let it do its thing, and then you can remove the small raw disk. Pretty much what you can do in esxi to switch to paravirtual drivers.

From my experience, if you are trying to achieve the performance you can get from Windows in ESXi, either with paravirtual driver or PCI passthrough, I fear you are out of luck. It's never happening. And 2019 is beafier than 2016 in every respect. IO Performance in bhyve should double before I ever consider switching to a newer Windows than 2016 (or 2027 must come, with new NAS hardware I am afraid).

It's quite unlikely you can reliably boot Windows 2019 from a NVME device that you passthrough to the VM in TrueNAS without some serious tweaking. Is the EFI boot loader even capable of booting from NVME?
I don't know but it seems you are really asking for trouble here, as this functionality is recent and fairly rudimentary in bhyve. Recent benchmarks indicate that GPU passthrough performance is 50-70% the original, and PCI passthrough is really about GPUs and minor PCI-e cards. Storage cards are different beasts.

I am just saying: even if they seem to work, you should test it thoroughly in order to ensure there is no silent corruption of your data. I had that before with freenas on ESXi with a perfectly fine HBA. The only thing that saved me was ZFS and the fact they were mirrors. Repeat: this was ESXi 6.5, not bhyve.