KVM Support has landed

[Kris Moore]

It's been a quick 2 weeks for TrueNAS SCALE! Thanks to everybody who has been providing feedback and contributing. We were able to fix a whole slew of issues rather rapidly which was only possible with so many testers giving this a whirl.

Wanted to let everybody know that full KVM / VM support has now landed in SCALE and is ready for testing. It supports PCI/GPU passthrough, as well as nested VM support. I'll be periodically updating the developer-notes page when notable changes like this are merged. As usual, happy testing and please keep that feedback coming!

TrueNAS SCALE

TrueNAS SCALE is the latest member of the TrueNAS family and provides Open Source HyperConverged Infrastructure (HCI) including Linux containers and VMs. TrueNAS SCALE includes the ability to cluster systems and provide scale-out storage with capacities of up to hundreds of Petabytes. Just like...

www.truenas.com

 

[silverback]

Amazing work. I did not expect this for months.

 

[Kris Moore]

The work is going quicker than expected for sure. Even in two weeks its already become very usable and we've been fixing bugs / porting issues left and right :)

 

[KrisBee]

Dumb question (?) How does KVM machine share data with TrueNAS SCALE host? VM net implemented as macvatp so guest cannot communicate with its own host.

 

[waqarahmed]

@KrisBee so basically there are 2 modes. We default to using macvtap interfaces for the NIC selected. However if the NIC selected is a bridge which you have created in Network -> Interfaces, we don't use macvtap for those and if you select such a bridge for your VM's NIC device, you get bridged networking where guest is able to access the host bridge's network where you can have your physical iface.

Please let me know if you run into any issues. Thanks

 

[waqarahmed]

For anyone testing PCI devices in SCALE, following would be the steps which should be taken to configure a PCI device so that it shows up in PCI device options in the UI.

1) Identify the PCI device which you would like to use for passthrough - ( virsh nodedev-list pci )
2) Once you have identified the device and made sure you are not using it anywhere i.e pools etc, you should execute "virsh nodedev-detach pci_0000_26_00_0" where the last value you would change for the device you want to use.
3) Step (2) will detach your PCI device from the host and then you can finally use it with any of your VM guest by adding a PCI device to the VM.

There are some points to know, right now, user is responsible for ensuring that he/she uses the correct PCI device and is not being used by any other component as in that case it might crash and you would require a reboot to get out of it.
Secondly if there are no VM's created, libvirt would not be functional and the above steps would not work - so in that case you should start libvirt your self with "midclt call service.start libvirtd".

And finally a tip for someone who regularly wants to use this, you can create a short script which detaches the PCI device from host OS and have it run as a post init task so the device would automatically be removed on each boot from host OS and can be safely used with the guest.


Please feel free to let me know if you run into any issues. Thank you

 

[silverback]

For anyone testing PCI devices in SCALE, following would be the steps which should be taken to configure a PCI device so that it shows up in PCI device options in the UI.

1) Identify the PCI device which you would like to use for passthrough - ( virsh nodedev-list pci )
2) Once you have identified the device and made sure you are not using it anywhere i.e pools etc, you should execute "virsh nodedev-detach pci_0000_26_00_0" where the last value you would change for the device you want to use.
3) Step (2) will detach your PCI device from the host and then you can finally use it with any of your VM guest by adding a PCI device to the VM.

There are some points to know, right now, user is responsible for ensuring that he/she uses the correct PCI device and is not being used by any other component as in that case it might crash and you would require a reboot to get out of it.
Secondly if there are no VM's created, libvirt would not be functional and the above steps would not work - so in that case you should start libvirt your self with "midclt call service.start libvirtd".

And finally a tip for someone who regularly wants to use this, you can create a short script which detaches the PCI device from host OS and have it run as a post init task so the device would automatically be removed on each boot from host OS and can be safely used with the guest.


Please feel free to let me know if you run into any issues. Thank you

Will this work for VGA passthrough, thanks?

 

[waqarahmed]

@silverback it should

 

[KrisBee]

@KrisBee so basically there are 2 modes. We default to using macvtap interfaces for the NIC selected. However if the NIC selected is a bridge which you have created in Network -> Interfaces, we don't use macvtap for those and if you select such a bridge for your VM's NIC device, you get bridged networking where guest is able to access the host bridge's network where you can have your physical iface.

Please let me know if you run into any issues. Thanks

Thanks for the reply. I was stuck in FreeNAS mode, thinking a bridge would be created by the system.

@waqarahmed I appreciate it's early days, but networking via the TrueNAS SCALE webui seems to be WIP. For example, changing from a dhcp to static IP post install leaves a the NIC with two IPs. You can created a bridge via the WEbUI but again you end up with a secondary ip on the bridge and it's not displayed under network > interfaces, nor is the bridge selectable when creating a VM.

 

[waqarahmed]

@KrisBee can you please file share a debug of the system and create a ticket at http://jira.ixsystems.com with reproduction steps for the networking issue ?

About bridge not being selectable when creating a VM, can you please clarify ? Does it error out or does it not show the bridge at all ? I am using bridged networking with my guests and everything is running smoothly.

Looking forward to hearing from you. Thank you

 

[KrisBee]

So , is this just my user error? 1st boot post-install, I can re-configure the interface from dhcp to a static IP after the "test change" and then "save" step via the Webui, But I get this, as shown at the console:

The Webui is reachable via IP 192.168.0.45 and the relevant page show this:

Attempting to create a bridge, gives this output at the console:

So it looks as if a bridge is created but the secondary IP error is still present, but in my case the "test changes" pizza of doom just keeps whirling around and no "save changes" appears on the webui.

Using the console setup I can get to this stage:

But the webui is unreachable on IP 192.168.0.46 and the network config is lost on a re-boot.

I should say this instance of TrueNAS scale was installed and is running under virt-manager (libvirt) on my linux desktop. So ens3 in this case is a virtio NIC. Never had these kind of network problems when testing either FreeNAS or TrueNAS core in the same way.

 

[waqarahmed]

Thank you for sharing this, can you please create a ticket on https://jira.ixsystems.com following all the steps you took so that I can reproduce this ? ( the ones mentioned are good, if you want to add more information, that's good as well ).
About UI stuck in a spinner, that probably happened because the ip changed and the IP you were using for the web UI was not accessible. If you wait for 60 secs, it would automatically revert the changes and you would be back to using your old IP.

Though probably it would be easier if you had 2 ifaces in the SCALE vm and used the web UI with the one which is not going to be affected by the bridge change. Anyways, please create that ticket and post the ticket number here so any one following this would know as well the progress of the issue. Thanks

 

[KrisBee]

OK, after a bit of VM juggling, it's sorted. I was slow to recognise TrueNAS SCALE appears to need two NIC for this to work. One for a management network to access the WebUI and to allow the removal of IP from the second and the creation of a bridge with the second NIC as member. Did that before you just posted:

Code:

truenas# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UP group default qlen 1000
    link/ether 52:54:00:32:77:9e brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:7a:43:f5 brd ff:ff:ff:ff:ff:ff
    altname enp0s9
    inet 192.168.2.45/24 scope global ens9
       valid_lft forever preferred_lft forever
4: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7e:38:ae:c6:76:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.45/24 scope global br1
       valid_lft forever preferred_lft forever
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:41:46:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe41:4655/64 scope link
       valid_lft forever preferred_lft forever
truenas# brctl
zsh: command not found: brctl
truenas# apt install bridge-utils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 35.1 kB of archives.
After this operation, 111 kB of additional disk space will be used.
Get:1 http://apt.tn.ixsystems.com/apt-direct/unstable/debian bullseye/main amd64 bridge-utils amd64 1.6-3 [35.1 kB]
Fetched 35.1 kB in 1s (38.8 kB/s)       
Selecting previously unselected package bridge-utils.
(Reading database ... 106225 files and directories currently installed.)
Preparing to unpack .../bridge-utils_1.6-3_amd64.deb ...
Unpacking bridge-utils (1.6-3) ...
Setting up bridge-utils (1.6-3) ...
Processing triggers for man-db (2.9.1-1) ...
truenas# brctl show
bridge name     bridge id               STP enabled     interfaces
br1             8000.7e38aec6763d       no              ens3
                                                        vnet0
truenas#

Probably not meant to use "apt install" but wanted brctl just to confirm net config.

 

[waqarahmed]

So @KrisBee it's not like that, if we have a single interface and we want to change networking configuration, it's going to be tricky because we would lose network connectivity during the process.

I think for starters we should document this process as it's going to be tricky to manage and many users only have one NIC to begin with. Thank you for bringing this to our attention, we would add some documentation to make sure that this behaviour is explained and proper steps are mentioned on how we would advise you to configure your interface in such a case. ( I configured my machine with a single NIC and it now uses a bridge, so this is definitely possible, just tricky ; ) )

 

[KrisBee]

Over to the devs on that. I thought net config via "console set up" might work with one NIC, but failed. @waqarahmed What meaning, if anything, do the boot methods "UEFI-CSM" and "GRUB" have iin TrueNAS SCALE? Thanks again for your interest.

 

[waqarahmed]

UEF-CSM is Legacy BIOS support and GRUB mode is deprecated in SCALE ( we are fixing this and should have the fixes in by Monday which include clarifying UEFI-CSM ).

 

[KrisBee]

@waqarahmed I'm not submitting a ticket about creating a bridge when the system has a single NIC as it is perfectly doable using the webui alone, I just made a mess of it yesterday. Default post-install is dhcp on the single NIC, the procedure is simply:

1. connect to webui using DHCP IP of single NIC.
2. create bridge with a static IP and single NIC as member.
3. re-connect to webui using static IP of bridge.
4. remove DHCP from single NIC, test and save.
5. check network config survives a re-boot.

I get the numpty prize of the week.

 

[waqarahmed]

Sweet @KrisBee, that should work nicely as well. Depends where we start from. Glad you figured it out ; )

 

[silverback]

I have been successful passing through both nic and usb cards to a windows vm. I see them in the Device Manager of Windows and in the virsh conf file @ 'virsh edit "vmname"'. I successfully detach a VGA card and am able to add it to the vm pci devices in the GUI. It does not however appear in the virsh config file or make it to the windows vm.

 

[waqarahmed]

@silverback are you able to add the device via UI to the VM ?