Why iohyve? Is the guidance outdated? PCI-Passthrough with bhyve

[seb101]

Hi folks,

My VM knowledge had to take a leap today as I wanted to dedicate some hardware NICs to a VM (with a view to using it to run a router/firewall VM).

The accepted knowledge on any forum post dedicated to this appears to be that you need to 'use iohyve' involving some tunables tweaks and then configuring the VM using iohyve in the command line. However other posts suggest that iohyve is 'outdated' and 'a stopgap'. Confusing. Now it seems that bhyve supports pci pass-through itself. https://wiki.freebsd.org/bhyve/pci_passthru

So my question... how do I add these passthrough config arguments that are detailed in the bhyve man page, namely: "-s 7,passthru,2/0/0" to my bhyve VMs created in the FreeNAS GUI? I was searching for a 'custom options' text field where maybe I could paste them in, but I can't find it. If it cannot be done through the GUI, where can I edit these in the config in a manner that will persist accross reboots/upgrades etc?

I realise the device masking from FreeNAS still needs to be done using a tunable at boot time (pptdevs, loader).

Thank you!

 

[seb101]

So... having done another hours worth of reading. Am I correct to assume this is not possible? Because the GUI is incapable of supporting custom arguments it would simply get overwritten every time I used the GUI, even if I was able to inject this into the config?

My only hope is to write a GUI extension to support custom arguments?!

I assume this is why people resort to iohyve... essentially its a way to bypass the gui for VM creation/config.

 

[sretalla]

The GUI is just using iohyve in the background anyway to run up the VMs as needed.

It's not really well supported until FreeBSD 12 for PCI passthrough anyway and with FreeNAS possibly being ported to Linux at some point, you will have much better options to use QEMU/KVM at that point.

I don't think the priority will be there to make any changes to the GUI at this time.

The generally accepted way to run a firewall on the same hardware as FreeNAS is to use a type 1 hypervisor like ESXi to run both FreeNAS and the firewall.

(I do that and am satisfied with the results)

 

[Chris Moore]

Just to add a little extra to this... The virtualization aspects of FreeNAS are not the primary development focus of the company. I don't think that feature is included in TrueNAS.

If you choose the route of a type 1 hypervisor and virtualizing FreeNAS, there are some very good guides in the forum from people that have done it. This is something where, if you do it wrong, you put all your data at risk. Please take the time to look through the useful links in my signature as I have collected several there.

 

[seb101]

I take your point about the 'purpose' of FreeNAS and the option of ESXi, but it seems... complex to say the least that I need to download/install/learn/configure an entirely new software stack (ESXi) in order to achieve what could be done with 20 characters of command line argument for bhyve in FreeBSD/FreeNAS today.

It could be so simple as to just add a single field on the FreeNas VM add/edit GUI for 'Custom Arguments' that were then passed directly through to the command line argument that the middleware runs. On the middleware side this is as simple as inserting this at line 228 of freenas/src/middlewared/middlewared/plugins/vm.py

Code:

args.append(str(self.vm['custom_args']))

I have no idea about Angular and the config DB setup but there can't be that much additional work to put that in the GUI surely?

I'd volunteer if there was a detailed contributer guide but, to be frank, the developer/contributor documentation on this system seems pretty sparse.

 

[seb101]

I the mean-time, can you see any pitfalls to just starting the VM using the bhyve command at post-init (added through Tasks->Init/Shutdown Scripts)?

bhyve -A -H -w -c 1 -m 512 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/pfserial -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd -s 4,e1000,tap2,mac=00:a0:98:09:de:0e -s 3:0,ahci,hd:/dev/zvol/msata/pfsense -s 5,passthru,2/0/0 -s 6,passthru,2/0/1 -s 7,passthru,2/0/2 -s 8,passthru,2/0/3 PFSense &

I suppose for completeness you'd also do a kill+destroy in the shutdown commands too.