SOLVED KVM Guest Cannot Ping TrueNAS SCALE Host (or vice-versa)

[xioustic]

I created an Ubuntu VM with the default settings. This bridged the VM directly into my network (192.168.87.217).

The FreeNAS host is 192.168.87.216.

Neither can ping each other. I wanted to setup NFS on the FreeNAS host and have the Ubuntu VM mount the NFS share to access the files, but no luck. Seems to be a layer 2/3 issue after some troubleshooting.

As I recall from awhile back at work, this was a limitation of the network stack in KVM when directly bridged to the physical NIC. We needed to create a dedicated isolated virtual bridge network on the host (with DHCP) and attach it to both the host and the guest. Then they could ping each over using the isolated virtual bridge network.

Hopefully it's not that convoluted and I'm just missing something. Has anyone else encountered this or been able to reproduce it? Any solutions?

Thanks! I'm very excited about the transition to Linux-based OS, so please keep up the good work.

 

[qhao.cn]

Enabling host-guest networking with KVM, Macvlan and Macvtap

How to use macvlan on the KVM hypervisor with macvtap on KVM guests so that the host and the guests can ping each other and behave like they are all on the same subnet (LAN).

www.furorteutonicus.eu

 

[beagle]

I still can't get my VMs to work on SCALE. It still reboots the host when the VM is started but regarding network I created a bridge, attached it to one of the physical NICs and set it as DHCP. The same config I did on my CentOS host when I setup KVM.

 

[xioustic]

Enabling host-guest networking with KVM, Macvlan and Macvtap

How to use macvlan on the KVM hypervisor with macvtap on KVM guests so that the host and the guests can ping each other and behave like they are all on the same subnet (LAN).

www.furorteutonicus.eu

I haven't tried this yet but I'm pretty certain it's going to work. Thanks!

 

[KrisBee]

@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:

root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 

Code:

root@truenas:~# virsh list --all
 Id   Name        State
---------------------------
 2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
 Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 

 

[xioustic]

@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:

root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 

Code:

root@truenas:~# virsh list --all
Id   Name        State
---------------------------
2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 

Thanks Kris, I'll give that a try when I get the chance. If that is all that's needed to fix the issue that would be wonderful!

 

[Anda]

Hey guys,

@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:

root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 

Code:

root@truenas:~# virsh list --all
Id   Name        State
---------------------------
2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 

I had the same problem and did that solution too. But now If I start a VM I have a new vnet interface that is configured to "Media Subtype: 10Mb/s Twisted Pair", and I can't change that. Also, my transfer rate is super low now. I'm using SMB btw.

 

[KrisBee]

@TrueTFZ, Is ethtool meaningful on a vnet interface? Use iperf3 between vm and host.

 

[proligde]

Hi there! I came across this thread as I'm having the same problem: My VM that has been migrated from Core to Scale works perfectly fine, except I can't access the scale host itself anymore, while I can access all other hosts in my LAN.

I looked into this suggestion:

Enabling host-guest networking with KVM, Macvlan and Macvtap

How to use macvlan on the KVM hypervisor with macvtap on KVM guests so that the host and the guests can ping each other and behave like they are all on the same subnet (LAN).

www.furorteutonicus.eu

but I'm wondering if that's still the way to go, as it feels hacky to me, and I'm worried it interferes with whatever Scale might come up in their next releases in case this topic gets addressed, and the guests can access the host like in core by default.

I also tried to create a bridge with my primary NIC in it, but this just renders my TrueNAS unreachable from my LAN as soon as I test-apply the network settings.

I might add I'm not super familiar with virtual network interfaces, bridges and so on. It's well possible I just made a stupid mistake.

 

[chenseanxy]

Hi there! I came across this thread as I'm having the same problem: My VM that has been migrated from Core to Scale works perfectly fine, except I can't access the scale host itself anymore, while I can access all other hosts in my LAN.

I looked into this suggestion:



but I'm wondering if that's still the way to go, as it feels hacky to me, and I'm worried it interferes with whatever Scale might come up in their next releases in case this topic gets addressed, and the guests can access the host like in core by default.

I also tried to create a bridge with my primary NIC in it, but this just renders my TrueNAS unreachable from my LAN as soon as I test-apply the network settings.

I might add I'm not super familiar with virtual network interfaces, bridges and so on. It's well possible I just made a stupid mistake.

Hi there, I went into the exact same issue and tried troubleshooting it manually, and I got it working!

Here's the short answer:
Reboot TrueNAS after you created the bridge and specified an interface as the member. Then it's very likely TrueNAS would be inaccessible via network, so using console, delete the configs on your physical interface, and manually specify an IP on the bridge. Then TrueNAS should be accessible, and VMs can talk to the host. (You might also need to change your Application (k3s) interface, and update the interface for every app deployed)

How I got here:
(My setup: physical interface: eno1, bridge: br0)
After creating br0, binding it with eno1, and moving the IP address from eno1 to br0, the server became inaccessible. `ip addr` using console shows br0 is DOWN, and has the no-carrier flag, indicating it's upstream eno1 wasn't bound to br0 properly. Manually binding using `ip link set eno1 master br0` shows "Device or resource busy". Therefore, reboot, and it's solved.

 

[proligde]

Hi there, I went into the exact same issue and tried troubleshooting it manually, and I got it working!

Aah - Great! Thank you! I guess I'll give it a try as soon as I have some hours for potential accessibility-troubleshooting left ;-)

 

[proligde]

br0` shows "Device or resource busy". Therefore, reboot, and it's solved.

Okay, you really made my day! It actually did work without any issues, using only the GUI and without reboot!

Here is what I did:

1. Remove the IP from my primary NIC -> Apply
2. Create a bridge
   1. Assign the primary NIC as only member to that bridge
   2. Assign the IP formerly attached to the NIC to the bridge
   3. Apply
3. Test Network Changes

It took quite a while like 40 seconds before the GUI was able to reconnect, so I had only a few seconds to click the button to make changes permanent.

After that I assigned the NIC of my VM to the bridge and started it. Now it connects to the host without issue. Thanks again! Hope this helps others here searching to the same thing.

 

[kewitt]

Okay, you really made my day! It actually did work without any issues, using only the GUI and without reboot!

Here is what I did:

1. Remove the IP from my primary NIC -> Apply
2. Create a bridge
   1. Assign the primary NIC as only member to that bridge
   2. Assign the IP formerly attached to the NIC to the bridge
   3. Apply
3. Test Network Changes

It took quite a while like 40 seconds before the GUI was able to reconnect, so I had only a few seconds to click the button to make changes permanent.

After that I assigned the NIC of my VM to the bridge and started it. Now it connects to the host without issue. Thanks again! Hope this helps others here searching to the same thing.

This worked for me but too just to make it more clear.

1. Remove the IP from my primary NIC -> Apply
2. Create a bridge
   1. Assign the primary NIC as only member to that bridge
   2. Assign the IP formerly attached to the NIC to the bridge
   3. Apply
3. Test Network Changes
   1. I have 2 nic on 2 different networks so I didn't have to wait 40 seconds 88.0/24 internal uses 89.0/24 external use
4. Click on Virtualization
   1. Select VM
      1. Device
      2. Nic
      3. Bind to Br#

 

[Ziggy]

Hi. Having the same issue. In step 1 do you mean remove (via trashcan) icon the primary ip thru which I communicate via the webgui:

and then proceed to step 2? I know I can rollback if I mess up, but I'd rather be sure of what I'm doing in the beginning. Thanks

 

[guyp2k]

Hi. Having the same issue. In step 1 do you mean remove (via trashcan) icon the primary ip thru which I communicate via the webgui:
View attachment 48327
and then proceed to step 2? I know I can rollback if I mess up, but I'd rather be sure of what I'm doing in the beginning. Thanks

No, click on the line and the GUI for that interface will open up, scroll to the bottom and remove IP or uncheck DHCP if you have selected, then click apply and move to the next steps.

 

[Ziggy]

Thanks. Tried that by unticking dhcp. At network test phase it hung for ages and then web interface became unavailable. I physically rebooted the machine and gui came up again. Went to step 4 and no bridge was available to bind the nic. I rechecked the network interface and settings were as before.

 

[Kasazn]

Hi guys, do you report this bug to JIRA? Is it considered a bug? Previously on CORE, it worked without using this workarounds. As pointed out by others, it might be fixed as SCALE goes through updates.

 

[guyp2k]

Thanks. Tried that by unticking dhcp. At network test phase it hung for ages and then web interface became unavailable. I physically rebooted the machine and gui came up again. Went to step 4 and no bridge was available to bind the nic. I rechecked the network interface and settings were as before.

I have seen it drop the GUI and once the GUI is available you should have the opportunity to save your changes.

 

[Ziggy]

Hi guys, do you report this bug to JIRA? Is it considered a bug? Previously on CORE, it worked without using this workarounds. As pointed out by others, it might be fixed as SCALE goes through updates.

Hi Kasazn. I don't know if it's a bug or a feature - in that old phrase. As you and others have said, it's interesting that this workaround isn't required in Core. If I cannot get it to work with the advice here I will report it.

 

[proligde]

I assume that's not a bug - it's just the way KVM networking works according to the post linked at the beginning of this thread ( https://www.furorteutonicus.eu/2013...uest-networking-with-kvm-macvlan-and-macvtap/ ) It's an older post, but I guess it still checks out. Maybe Scale will ship with a bridge set up by default in future releases to make it work out of the box. On the other hand - some users might want that kind of NIC isolation, tho.