Hello,
Im trying to passthrough my HBA (HP Smart Array H240 8 Port 12G PCIe x8 SAS HBA in IT-mode) to a Ubuntu 20.04 VM on TrueNAS-12.0-U4.1.
Im running a Intel Xeon E5-2680v2 in a Fujitsu D3128-B25 workstation mainboard both of which support VT-d.
I followed this guide for PCIe Passthrough: https://wiki.freebsd.org/bhyve/pci_passthru
here are my outputs:
Host VT-d support via
Set values via the System tunables:
find out PCIe ID and PCI card MSI/MSI-x support:
verifying that the HBA is masked for PCIe passthrough after reboot:
I can then add the HBA to my VM no problem.
But the VM will not boot (Autoboot at start of TrueNAS is turned off).
When I manually start the VM via the GUI is switches to off as soon as I reload the page or switch back to the VM section after looking at some other settings page.
I then dug into the log files at
here I can see my 3 attempts to spin up the VM getting these errors every time:
I could not find something usefull googeling these errors. Does anyone know why I'm getting these or how to resolv them?
Im trying to passthrough my HBA (HP Smart Array H240 8 Port 12G PCIe x8 SAS HBA in IT-mode) to a Ubuntu 20.04 VM on TrueNAS-12.0-U4.1.
Im running a Intel Xeon E5-2680v2 in a Fujitsu D3128-B25 workstation mainboard both of which support VT-d.
I followed this guide for PCIe Passthrough: https://wiki.freebsd.org/bhyve/pci_passthru
here are my outputs:
Host VT-d support via
acpidump -t | grep DMAR
Code:
root@freenas:~ # acpidump -t | grep DMAR DMAR: Length=196, Revision=1, Checksum=169, OEMID=A M I, OEM Table ID=OEMDMAR, OEM Revision=0x1,
Set values via the System tunables:
find out PCIe ID and PCI card MSI/MSI-x support:
Code:
root@freenas:~ # pciconf -v -l ciss0@pci0:3:0:0: class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Smart Array Gen9 Controllers' class = mass storage subclass = SAS root@freenas:~ # pciconf -lc ciss0@pci0:3:0:0: class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00 cap 01[80] = powerspec 3 supports D0 D3 current D0 cap 05[90] = MSI supports 32 messages, 64 bit cap 11[b0] = MSI-X supports 64 messages, enabled Table in map 0x10[0x2000], PBA in map 0x10[0x3000] cap 10[c0] = PCI-Express 2 endpoint max data 256(512) NS link x8(x8) speed 8.0(8.0) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 0019[300] = PCIe Sec 1 lane errors 0
verifying that the HBA is masked for PCIe passthrough after reboot:
Code:
root@freenas:~ # pciconf -lc ppt0@pci0:3:0:0: class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00 cap 01[80] = powerspec 3 supports D0 D3 current D0 cap 05[90] = MSI supports 32 messages, 64 bit cap 11[b0] = MSI-X supports 64 messages Table in map 0x10[0x2000], PBA in map 0x10[0x3000] cap 10[c0] = PCI-Express 2 endpoint max data 256(512) NS link x8(x8) speed 8.0(8.0) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 0019[300] = PCIe Sec 1 lane errors 0 root@freenas:~ # pciconf -lvb ppt0@pci0:3:0:0: class=0x010700 card=0x21c7103c chip=0x3239103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Smart Array Gen9 Controllers' class = mass storage subclass = SAS bar [10] = type Memory, range 64, base 0xfb900000, size 1048576, enabled bar [18] = type Memory, range 64, base 0xfba80000, size 1024, enabled bar [20] = type I/O Port, range 32, base 0xe000, size 256, enabled
I can then add the HBA to my VM no problem.
But the VM will not boot (Autoboot at start of TrueNAS is turned off).
When I manually start the VM via the GUI is switches to off as soon as I reload the page or switch back to the VM section after looking at some other settings page.
I then dug into the log files at
/var/log/libvirt/bhyve/Ubuntu.log
and found this:Code:
/usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc -> bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned bhyve: failed to initialize BARs for PCI 3/0/0 device emulation initialization error: Inappropriate ioctl for device /usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc -> bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned bhyve: failed to initialize BARs for PCI 3/0/0 device emulation initialization error: Inappropriate ioctl for device /usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 5:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc -> bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned bhyve: failed to initialize BARs for PCI 3/0/0 device emulation initialization error: Inappropriate ioctl for device /usr/sbin/bhyve -c cpus=16,sockets=1,cores=16,threads=1 -m 20480 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4:0,ahci,hd:/dev/zvol/Samsung 850 Evo 256 GB/Ubuntu-ztmv3q,hd:/dev/zvol/Toshiba-12TB/Toshiba-12Tb -s 30:0,xhci,tablet -s 31,lpc -> bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned bhyve: failed to initialize BARs for PCI 3/0/0 device emulation initialization error: Inappropriate ioctl for device
here I can see my 3 attempts to spin up the VM getting these errors every time:
Code:
bhyve: passthru device 3/0/0 BAR 2: base 0xfba80000 or size 0x400 not page aligned bhyve: failed to initialize BARs for PCI 3/0/0 device emulation initialization error: Inappropriate ioctl for device
I could not find something usefull googeling these errors. Does anyone know why I'm getting these or how to resolv them?