USB pass through to VM created from FreeNAS GUI and iohyve can't see it

[Rand0mUser]

Hello,

I know this is something already discussed in many threads, but I searched a lot and didn't find any real working solution so I created a new thread.

I want to use a Z-Wave USB device with a domotic automation system running into a VM. The debian based VM was built using the UI, I'm running FreeNAS 11.2-U5.

I found this thread about virtualhere but it is not relevant since it won't work with USB device plugged in the mobo running FreeNAS. It needs to run the virtualhere server in other machine.

I tried following instructions here for bhyve PCI Passthrough. I started to have trouble at step 5 about the slot option, because I can't really pass it to bhyve since the VM was created from UI, and is started automatically by the FreeNAS system. I found the bhyve command used to launch the VM in the file /var/log/middlewared.log as described in this thread: How do you start/stop VMs from shell?. So in my case, command was:

Code:

bhyve -A -H -w -c 2 -m 1024 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm2A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,e1000,tap0,mac=00:a0:98:xx:xx:xx -s 29,fbuf,vncserver,tcp=192.168.1.xx:6416,w=1024,h=768,,wait -s 30,xhci,tablet -s 3:0,ahci,cd:/mnt/pool1raidz_9tb/9TB_dataset/Systemes-d-exploitation,hd:/dev/zvol/pool1raidz_9tb/Jeedom-thcz47 2_Jeedom_VM

My PCI parameters were as following (I'm trying to pass through the whole PCI USB controller to the VM):

Code:

pptdevs="0/20/0"

so I added it to the bhyve command:

Code:

bhyve -s 7,passthru,0/20/0

Using this whole command:

Code:

bhyve -A -H -w -c 2 -m 1024 -s 0:0,hostbridge -s 31,lpc -s 7,passthru,0/20/0 -l com1,/dev/nmdm2A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,e1000,tap0,mac=00:a0:98:xx:xx:xx -s 29,fbuf,vncserver,tcp=192.168.1.xx:6416,w=1024,h=768,,wait -s 30,xhci,tablet -s 3:0,ahci,cd:/mnt/pool1raidz_9tb/9TB_dataset/Systemes-d-exploitation,hd:/dev/zvol/pool1raidz_9tb/Jeedom-thcz47 2_Jeedom_VM

Then I get "passthru requires guest memory to be wired " error. I tried adding -S as explained here: Not working with pci passthru. VM could not start, and when I tried again same bhyve command the FreeNAS get freezed and I had to hard reset the server.

Here is where I am so far, I set up loader.conf options directly from the UI in Tunables like that:

So at boot my PCI USB controller is recognized from this:

xhci0@pci0:0:20:0: class=0x0c0330 card=0xa12f1849 chip=0xa12f8086 rev=0x31 hdr=0x00
vendor = 'Intel Corporation'
device = 'Sunrise Point-H USB 3.0 xHCI Controller'
class = serial bus
subclass = USB

to this:

ppt0@pci0:0:20:0: class=0x0c0330 card=0xa12f1849 chip=0xa12f8086 rev=0x31 hdr=0x00
vendor = 'Intel Corporation'
device = 'Sunrise Point-H USB 3.0 xHCI Controller'
class = serial bus
subclass = USB

I tried what's explained here: USB 3.0 PCI Controller Pass through (I didn't disable xhci driver in the kernel since I don't think this is mandatory), but I noticed this only seems to work if the VM was created using iohyve. Problem is since my VM was created from the UI, iohyve just don't see it:

Code:

root@freenas[~/scripts]# iohyve info -v -s -d
Guest  Size  CPU  RAM  Pool  OS  Loader  Tap  Con  VMM?  Running  rcboot?  Description

So obviously there is no way that I can pass the PCI controller to the VM using iohyve command such as

Code:

iohyve set vmname pcidev:1=passthru,0/20/0

So I guess this "could" work if I create a VM using iohyve but I didn't tried yet. Is there any way to pass the PCI device to the VM which was created from the FreeNAS GUI? I really want my system to stay as clean as possible, I don't want to wreck my install. Is there any way that iohyve can see the VM which was created from the GUI?

 

[dlavigne]

Did you find a solution for this?

 

[Rand0mUser]

Hello,

I made some progress, but it's still not working and I almost broke my server install.

So since I can't pass through to VM created using the GUI, I created a new debian VM with iohyve, following informations I found here: FreeNAS 9.10 : jouons avec bhyve et iohyve (in french) and here: iohyve manual. Here are steps:

Code:

iohyve create debianvm 10G
iohyve set debianvm loader=grub-bhyve
iohyve set debianvm os=debian
iohyve set debianvm ram=2G
iohyve set debianvm cpu=2
iohyve fetch https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.0.0-amd64-netinst.iso
iohyve install debianvm debian-9.9.0-amd64-netinst.iso
iohyve set debianvm boot=1

So far, the virtual machine was running.

Then, I had to pass through the PCIe USB controller to it. I made it using those commands:

Code:

iohyve set debianvm pcidev:7=passthru,0/20/0
iohyve set debianvm bargs="-S -A -H -P"

The -S parameter was required for wired memory.

And this is when I had critical issue. With those parameters, the VM was crashing the FreeNAS server just a few seconds after the VM boot. Since I had VM autoboot enabled with iohyve set debianvm boot=1 the server was stuck into a bootloop. It was starting, VM was booting, it was crashed, restart...

I just managed to SSH connect and launch command to remove VM autoboot:

Code:

iohyve set debianvm boot=0

and 2 seconds later server was crashed again. But I could boot it a last.

I tried launching VM manually, and it's still crashing the server, just at this point:

Here are VM infos:

Code:

root@freenas[~]# iohyve getall debianvm
Getting debianvm iohyve properties...
bargs            -S_-A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             0
con              nmdm0
cpu              2
description      Mon Jul  8 14:42:35 CEST 2019
install          yes
loader           grub-bhyve
name             debianvm
os               debian
pcidev:7         passthru,0/20/0
persist          1
ram              2G
size             10G
tap              tap0
template         NO
utc              YES

This VM setup will just crash my server at boot. The VM without those parameters has no PCIe USB controller pass through.

I don't know if I missed something about my settings, of if this is freeBSD / iohyve / bhyve issue of some kind.