[bhyve] Pci passthrough errors

[zhnu]

Hi, im trying to run bhyve using cli and passing a gpu to the VM but I get an error:

Code:

Assertion failed: (error == 0), function modify_bar_registration, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/pci_emul.c, line 491.
Abort (core dumped)

So far what I tried:

Detect the card in question:

Code:

pciconf -v -l

In my case there was 2 devices @pci0:7:0:1 @pci0:7:0:0
I added this 2 devices to the pptdevs list and the vmm_load (had to add this otherwise couldn't set the driver for the device it would complain "Device is busy") via web-ui freenas tunnables:

Code:

Variable: pptdevs
Value: 7/0/0 7/0/1
Type: Loader

Variable: vmm_load
Value: YES
Type: Loader

After reboot on the cli I run:

Code:

devctl set driver pci0:7:0:0 ppt
devctl set driver pci0:7:0:1 ppt
ifconfig tap1 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 addm tap1
ifconfig tap1 up
// (The S on bhyve command is necessary to hard wire memory I think)
bhyve \
  -APHS \
  -c 4 \
  -m 4096 \
  -s 0:0,hostbridge \
  -s 31,lpc \
  -l com1,/dev/nmdm3A \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  -s 3,e1000,tap1 \
  -s 29,fbuf,tcp=0.0.0.0:5903,w=1024,h=768,wait \
  -s 30,xhci,tablet \
  -s 7:0,passthru,7/0/0 \
  -s 7:1,passthru,7/0/1 \
  -s 4,ahci-hd,/dev/zvol/ssd/VMS2 \
  VMGUI

If I dont passthrough the device it works correctly, otherwise gives the error described on top

NOTES:
If you get "vm_reinit: Device busy" just "bhyvectl --destroy --vm=VMNAME"
In my case after installing ubuntu the vm couldn't find grubx64.efi just restarted the vm with the refind iso mounted "-s 5,ahci-cd,/pathTo/refind-cd-0.11.0.iso" and copied the contents from: /boot/efi/EFI/ubuntu to: /boot/efi/EFI/Boot

Any help or ideas on this matter would be appreciated.

 

[zhnu]

Update tried to boot with the kernel parameter "pci=nocrs" as suggested on "https://github.com/chyves/chyves/issues/3" (seems like a related issue) similiar error occurs:

Code:

Assertion failed: (pi->pi_bar[baridx].type == PCIBAR_IO), function passthru_write, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/pci_passthru.c, line 850.
Abort (core dumped)

Maybe I need to wait for a more recent version of bhyve on freenas?

 

[zhnu]

SPECS:
HPE Microserver Gen8:
BIOS: (VT-d and VT-x enabled)
CPU: Intel(R) Xeon(R) CPU E3-1225 V2
RAM: 16G DDR3 ECC
GPU: PNY Nvidia Quadro P400 (Also tried with gtx650 Ti and Amd R9 390)
OS: FreeNAS-11.0-U2 (e417d8aa5)

 

[zhnu]

Update: On Xenserver I manage to get PCI passthrough working (same system) it smells like a hypervisor error I will wait for newer versions to be released and post my update then.

 

[Stux]

Bhyve vga pass-thru is currently in the research stage of implementation...

https://wiki.freebsd.org/bhyve

And:

bhyve does not support VGA passthrough devices but work is under way to support them

https://wiki.freebsd.org/bhyve/pci_passthru

 

[zhnu]

My bad didn't notice there was a distinction between the 2 (pci and VGA passthrough)

 

[gremblin]

My bad didn't notice there was a distinction between the 2 (pci and VGA passthrough)

Any update on this? I'm trying to do the same thing. Pass an NVidia GPU to my Ubuntu VM. I don't need VGA support, I just want the GPU for hardware acceleration on video processing. Is this possible on bhyve? I can I use commands similar to yours for booting my VM that I have created in the FreeNAS GUI and pass a PCI port to it... even say for a NIC or something?

**EDIT**
Found his command reference here: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html