Question about network config for iohyve

[Steasenburger]

Hey guys,
I am just trying to make a vm with the iohyve cli and the Ubuntu VM is finally running :D
However i have not managed to configure the network settings properly, so that in the Ubuntu VM I have access to the internet or at least my LAN.
So this is the configuration of my VM:

Code:

freenas# iohyve getall tvh
Getting tvh iohyve properties...
bargs            -S_-A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             0
con              nmdm1
cpu              2
description      Tue May 26 20:20:51 CEST 2020
install          no
loader           grub-bhyve
name             tvh
os               debian
persist          1
ram              1024mb
size             20G
tap              tap0
template         NO
utc              YES
vnc              NO
vnc_h            600
vnc_ip           127.0.0.1
vnc_tablet       NO
vnc_w            800
vnc_wait         NO

And this is what ifconfig returns to me:

Code:

freenas# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: lan
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 40:b0:76:09:07:4b
        hwaddr 40:b0:76:09:07:4b
        inet6 fe80::42b0:76ff:fe09:74b%re0 prefixlen 64 scopeid 0x1
        inet 192.168.0.97 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Attached to ubuntu
        options=80000<LINKSTATE>
        ether 00:bd:20:cc:f7:00
        hwaddr 00:bd:20:cc:f7:00
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 1553
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:99:f9:a1:40:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
freenas#

So what am i doing wrong here? Do i have to set the re0 adapter for the vm or the bridge? I have no idea what all these adapters are used for.
It seems to me that the re0 adapter is the "host" adapter, because it has the correct IP address.
So is the network going to be bridged over the host and the VM or is there a new adapter created (tap0?) and why doesn't it have internet?

Greetings and thanks

Edit: BTW: I've set iohyve_flags to "kmod=1 net=re0" in the tunables section. The reason why i want to use iohyve is the possibility of PCI-passthrough :)

 

[Samuel Tai]

Your VM uses the tap0 interface. The host uses the re0 interface. Both are bound together under bridge0.

Please also show the output of ifconfig inside the VM.

 

[Steasenburger]

Yeah i already tried that, but inside the Ubuntu Server VM there seems to be no ifconfig preinstalled.
Is there another way to get the needed infos?

 

[Steasenburger]

Okay ifconfig seems to be outdated and replaced by ip, well then here is what ip returns:

Code:

tvh@tvheadend:~$ ip -s link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast
    438624     6152     0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    438624     6152     0       0       0       0
2: enp0s3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:a0:98:a8:81:dd brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0

Yeah i know this doesn't look good, but I have no idea what to do

 

[Samuel Tai]

Try nmcli connection show to find the UUID for the enp0s3 interface. Then try nmcli connection up uuid <UUID from previous step>.

 

[Steasenburger]

Hmm unfortunatly the nmcli program is also not preinstalled and can only be installed by the network-manager package, which needs internet to download.
SSH is also not working because of the network problems and I really don't know how to copy a .deb file into the VM.
Will try again tomorrow

 

[Samuel Tai]

OK, then try ip link set enp0s3 up.

 

[Steasenburger]

Hmm I had a problem with the VM so that it wouldn't start anymore at all.
So i had to recreate the VM completly, this time i noted exactly how i configured it:

Code:

freenas# iohyve create ubuntu 10G
Creating ubuntu...
freenas# iohyve set ubuntu loader=grub-bhyve
Setting ubuntu loader=grub-bhyve...
freenas# iohyve set ubuntu os=debian
Setting ubuntu os=debian...
freenas# iohyve set ubuntu ram=2G
Setting ubuntu ram=2G...
freenas# iohyve set ubuntu cpu=2
Setting ubuntu cpu=2...
freenas# iohyve install ubuntu ubuntu-20.04-live-server-amd64.iso
Installing ubuntu...
GRUB Process does not run in background....

Code:

freenas# iohyve getall ubuntu
Getting ubuntu iohyve properties...
bargs            -A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             0
con              nmdm0
cpu              2
description      Wed May 27 10:53:14 CEST 2020
install          yes
loader           grub-bhyve
name             ubuntu
os               debian
persist          1
ram              2G
size             10G
tap              tap0
template         NO
utc              YES
vnc              NO
vnc_h            600
vnc_ip           127.0.0.1
vnc_tablet       NO
vnc_w            800
vnc_wait         NO

So right now i'm stuck in the installer menu where it asks about the network configuration.
I tried using DHCP but it couldn't establish a connection, so what am I supposed to type in?
Should i assign a new IP address for the VM?

Thanks for your help so far BTW ;)

 

[Samuel Tai]

Why are you using grub? Ubuntu will boot via UEFI.

 

[Steasenburger]

Ahh goiod to know that ^^
The problem is that most guides of installing ubuntu in a VM using iohyve are very old, so i just used what was described there.
I could imagine, that the switch from grub to uefi was just recently.

Anyway when i set the loader to uefi, it asks me about at firmware file. Do you know where i can get this?

 

[Samuel Tai]

Why are you using iohyve, and not the VM GUI, which does all this for you, including installing the UEFI firmware file? Anyway, the firmware file should be at /usr/local/share/uefi-firmware.

 

[Steasenburger]

Why are you using iohyve, and not the VM GUI, which does all this for you, including installing the UEFI firmware file? Anyway, the firmware file should be at /usr/local/share/uefi-firmware.

I know I know, I even got a Ubuntu VM running using the GUI for all my Docker containers.
The problem is that with the GUI i can't configure PCI passthru and this is a feature that i really need to get my DVB stick running.
I know that i ask dumb questions, but it's hard to find useful and up-to-date informations for iohyve :/

Anyway thank you for the tip with the firmware this worked well. Now i can boot using UEFI.
However I get this error when i try to install the Ubuntu image:

Code:

 freenas# open of tap device /dev/tap0 failed 

When I try to install ubuntu anyway after selecting "install ubuntu" i just get a blinking cursor.

Edit: I'm acutually thinking about switching to https://github.com/churchers/vm-bhyve, this still seems to be maintained in contrast to iohyve. Maybe this utility works better..

 

[Samuel Tai]

You may have to stop the Ubuntu VM, and then run ifconfig tap0 destroy. I've had tap interfaces hang up a couple of times, so subsequent VM starts then use tap1, tap2, etc.

 

[Samuel Tai]

For iohyve documentation, see the source at https://github.com/pr1ntf/iohyve.

 

[Steasenburger]

I've seen this thanks, but i've switched now to vm bhyve (active development and i think user friendlier) and finally got an Ubuntu 18.04 VM (no UEFI this time) to run :D
There were no big network problems this time, but the PCI passthru does not work yet.

I enabled IOMMU on my (AMD AM4 A320) mainboard and my CPU Ryzen 2200G should also support hardware virtualisation.
I also reserved the PCIe USB controller for the VM by adding pptdevs="6/0/0" to my tuneables.
This still seems to work, because when i run vm passthru it marks the USB controller as ready. Nice

Code:

DEVICE     BHYVE ID     READY        DESCRIPTION
...
re0        5/0/0        No           RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
ppt0       6/0/0        Yes          uPD720201 USB 3.0 Host Controller

After that i added "passthru0="6/0/0" to the configuration of the VM.
But when i now try to start the VM it always responds with this error:

Code:

May 27 20:46:58: initialising
May 27 20:46:58:  [loader: grub]
May 27 20:46:58:  [cpu: 1]
May 27 20:46:58:  [memory: 1024M]
May 27 20:46:58:  [hostbridge: standard]
May 27 20:46:58:  [com ports: com1]
May 27 20:46:58:  [uuid: ac3c36e7-a033-11ea-9ea3-40b07609074b]
May 27 20:46:58:  [utctime: yes]
May 27 20:46:58:  [debug mode: no]
May 27 20:46:58:  [primary disk: disk0.img]
May 27 20:46:58:  [primary disk dev: file]
May 27 20:46:58: fatal; pci passthrough not supported on this system (no VT-d or amdvi)

Okay, i've already read something about PCI passthru and this guide from FreeBSD hints that you have to enable something first

• bhyve support AMD-Vi/IOMMU by change r317600. By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.

But I can't enable the setting with sysctl, because it tells me that the setting doesn't exist ?

Code:

sysctl: unknown oid 'hw.vmm.amdvi.enable' 

Do you have an idea? Is this fix for AMD hardware already included in Freenas 11.3?

 

[Samuel Tai]

You may need to change a BIOS setting to enable AMD-Vi for that tunable to become visible.

 

[Steasenburger]

Hmm thats what i also thought, but i just checked it again, i've enabled everything related to virtualisation.
I read this here: https://forums.freebsd.org/threads/bhyve-passthrough-ryzen-x370.66446/#post-395577

Could it be that AMDVI support does not exist in FreeBSD 11.3?

 

[Samuel Tai]

You may have to compile a custom kernel to enable that. I think the FreeNAS devs don't enable PCI-passthrough in bhyve because it's not fully baked yet.

 

[Steasenburger]

Yup this could be the case.
Any hints how to compile the kernel? Is this a task that can be done for a more or less beginner?

 

[Steasenburger]

Any ideas why this isn't included in Freenas 11.3, if it will be available in Freenas 12 or how i can add it manually?