bhyve PCIe Passthrough not working | base or size not page aligned | failed to initialize BARs | device emulation initialization error: Inappropriate

[lukazy88]

Hello,

Im trying to passthrough my HBA (HP Smart Array H240 8 Port 12G PCIe x8 SAS HBA in IT-mode) to a Ubuntu 20.04 VM on TrueNAS-12.0-U4.1.
Im running a Intel Xeon E5-2680v2 in a Fujitsu D3128-B25 workstation mainboard both of which support VT-d.

I followed this guide for PCIe Passthrough: https://wiki.freebsd.org/bhyve/pci_passthru

here are my outputs:

Host VT-d support via acpidump -t | grep DMAR

Code:

root@freenas:~ # acpidump -t | grep DMAR
    DMAR: Length=196, Revision=1, Checksum=169,
    OEMID=A M I, OEM Table ID=OEMDMAR, OEM Revision=0x1,

Set values via the System tunables:

find out PCIe ID and PCI card MSI/MSI-x support:

Code:

root@freenas:~ # pciconf -v -l
ciss0@pci0:3:0:0:       class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00
    vendor     = 'Hewlett-Packard Company'
    device     = 'Smart Array Gen9 Controllers'
    class      = mass storage
    subclass   = SAS

root@freenas:~ # pciconf -lc
ciss0@pci0:3:0:0:       class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 32 messages, 64 bit
    cap 11[b0] = MSI-X supports 64 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
    cap 10[c0] = PCI-Express 2 endpoint max data 256(512) NS
                 link x8(x8) speed 8.0(8.0)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0019[300] = PCIe Sec 1 lane errors 0

verifying that the HBA is masked for PCIe passthrough after reboot:

Code:

root@freenas:~ # pciconf -lc
ppt0@pci0:3:0:0:        class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 32 messages, 64 bit
    cap 11[b0] = MSI-X supports 64 messages
                 Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
    cap 10[c0] = PCI-Express 2 endpoint max data 256(512) NS
                 link x8(x8) speed 8.0(8.0)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0019[300] = PCIe Sec 1 lane errors 0
   
root@freenas:~ # pciconf -lvb
ppt0@pci0:3:0:0:        class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00
    vendor     = 'Hewlett-Packard Company'
    device     = 'Smart Array Gen9 Controllers'
    class      = mass storage
    subclass   = SAS
    bar   [10] = type Memory, range 64, base 0xfb900000, size 1048576, enabled
    bar   [18] = type Memory, range 64, base 0xfba80000, size 1024, enabled
    bar   [20] = type I/O Port, range 32, base 0xe000, size 256, enabled

I can then add the HBA to my VM no problem.
But the VM will not boot (Autoboot at start of TrueNAS is turned off).
When I manually start the VM via the GUI is switches to off as soon as I reload the page or switch back to the VM section after looking at some other settings page.

I then dug into the log files at /var/log/libvirt/bhyve/Ubuntu.log and found this:

Code:

/usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc ->

bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned
bhyve: failed to initialize BARs for PCI 3/0/0
device emulation initialization error: Inappropriate ioctl for device

/usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc ->

bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned
bhyve: failed to initialize BARs for PCI 3/0/0
device emulation initialization error: Inappropriate ioctl for device

/usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc ->

bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned
bhyve: failed to initialize BARs for PCI 3/0/0
device emulation initialization error: Inappropriate ioctl for device

/usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc ->

bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned
bhyve: failed to initialize BARs for PCI 3/0/0
device emulation initialization error: Inappropriate ioctl for device

here I can see my 3 attempts to spin up the VM getting these errors every time:

Code:

bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned
bhyve: failed to initialize BARs for PCI 3/0/0
device emulation initialization error: Inappropriate ioctl for device

I could not find something usefull googeling these errors. Does anyone know why I'm getting these or how to resolv them?