SOLVED KVM Guest Cannot Ping TrueNAS SCALE Host (or vice-versa)

xioustic

Dabbler
Joined
Sep 4, 2014
Messages
23
I created an Ubuntu VM with the default settings. This bridged the VM directly into my network (192.168.87.217).

The FreeNAS host is 192.168.87.216.

Neither can ping each other. I wanted to setup NFS on the FreeNAS host and have the Ubuntu VM mount the NFS share to access the files, but no luck. Seems to be a layer 2/3 issue after some troubleshooting.

As I recall from awhile back at work, this was a limitation of the network stack in KVM when directly bridged to the physical NIC. We needed to create a dedicated isolated virtual bridge network on the host (with DHCP) and attach it to both the host and the guest. Then they could ping each over using the isolated virtual bridge network.

Hopefully it's not that convoluted and I'm just missing something. Has anyone else encountered this or been able to reproduce it? Any solutions?

Thanks! I'm very excited about the transition to Linux-based OS, so please keep up the good work.
 

beagle

Explorer
Joined
Jun 15, 2020
Messages
91
I still can't get my VMs to work on SCALE. It still reboots the host when the VM is started but regarding network I created a bridge, attached it to one of the physical NICs and set it as DHCP. The same config I did on my CentOS host when I setup KVM.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:
root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 


Code:
root@truenas:~# virsh list --all
 Id   Name        State
---------------------------
 2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
 Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 
 

xioustic

Dabbler
Joined
Sep 4, 2014
Messages
23
@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:
root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 


Code:
root@truenas:~# virsh list --all
Id   Name        State
---------------------------
2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 

Thanks Kris, I'll give that a try when I get the chance. If that is all that's needed to fix the issue that would be wonderful!
 

Anda

Explorer
Joined
Aug 12, 2015
Messages
63
Hey guys,

@xioustic TrueNAS SCALE alpha or nightlies? Create a bridge interface on the SCALE host via the UI, make your NIC a member of that bridge and then create/attach your VM to that bridge. No problem to ping between host & guest or mounting a NFS share in the VM.

For example:

Code:
root@truenas:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 52:54:00:bd:40:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:68:7e:61:bb:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global br0
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:a0:98:26:12:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fca0:98ff:fe26:1283/64 scope link
       valid_lft forever preferred_lft forever
root@truenas:~# 


Code:
root@truenas:~# virsh list --all
Id   Name        State
---------------------------
2    1_deb10vm   running

root@truenas:~# ping -c 1 192.168.1.26
PING 192.168.1.26 (192.168.1.26) 56(84) bytes of data.
64 bytes from 192.168.1.26: icmp_seq=1 ttl=64 time=0.602 ms

--- 192.168.1.26 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
root@truenas:~# ssh chris@192.168.1.26
chris@192.168.1.26's password:
Linux deb10vm 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov  3 17:37:44 2020 from 192.168.1.25
chris@deb10vm:~$ sudo nfsstat -m
[sudo] password for chris:
/mnt from 192.168.1.25:/mnt/Spool/sdata
Flags:    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.25,mountvers=3,mountport=42318,mountproto=udp,local_lock=none,addr=192.168.1.25

chris@deb10vm:~$ 

I had the same problem and did that solution too. But now If I start a VM I have a new vnet interface that is configured to "Media Subtype: 10Mb/s Twisted Pair", and I can't change that. Also, my transfer rate is super low now. I'm using SMB btw.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@TrueTFZ, Is ethtool meaningful on a vnet interface? Use iperf3 between vm and host.
 

proligde

Dabbler
Joined
Jan 29, 2014
Messages
21
Hi there! I came across this thread as I'm having the same problem: My VM that has been migrated from Core to Scale works perfectly fine, except I can't access the scale host itself anymore, while I can access all other hosts in my LAN.

I looked into this suggestion:


but I'm wondering if that's still the way to go, as it feels hacky to me, and I'm worried it interferes with whatever Scale might come up in their next releases in case this topic gets addressed, and the guests can access the host like in core by default.

I also tried to create a bridge with my primary NIC in it, but this just renders my TrueNAS unreachable from my LAN as soon as I test-apply the network settings.

I might add I'm not super familiar with virtual network interfaces, bridges and so on. It's well possible I just made a stupid mistake.
 

chenseanxy

Cadet
Joined
Jul 10, 2021
Messages
1
Hi there! I came across this thread as I'm having the same problem: My VM that has been migrated from Core to Scale works perfectly fine, except I can't access the scale host itself anymore, while I can access all other hosts in my LAN.

I looked into this suggestion:



but I'm wondering if that's still the way to go, as it feels hacky to me, and I'm worried it interferes with whatever Scale might come up in their next releases in case this topic gets addressed, and the guests can access the host like in core by default.

I also tried to create a bridge with my primary NIC in it, but this just renders my TrueNAS unreachable from my LAN as soon as I test-apply the network settings.

I might add I'm not super familiar with virtual network interfaces, bridges and so on. It's well possible I just made a stupid mistake.

Hi there, I went into the exact same issue and tried troubleshooting it manually, and I got it working!

Here's the short answer:
Reboot TrueNAS after you created the bridge and specified an interface as the member. Then it's very likely TrueNAS would be inaccessible via network, so using console, delete the configs on your physical interface, and manually specify an IP on the bridge. Then TrueNAS should be accessible, and VMs can talk to the host. (You might also need to change your Application (k3s) interface, and update the interface for every app deployed)

How I got here:
(My setup: physical interface: eno1, bridge: br0)
After creating br0, binding it with eno1, and moving the IP address from eno1 to br0, the server became inaccessible. `ip addr` using console shows br0 is DOWN, and has the no-carrier flag, indicating it's upstream eno1 wasn't bound to br0 properly. Manually binding using `ip link set eno1 master br0` shows "Device or resource busy". Therefore, reboot, and it's solved.
 

proligde

Dabbler
Joined
Jan 29, 2014
Messages
21
Hi there, I went into the exact same issue and tried troubleshooting it manually, and I got it working!

Aah - Great! Thank you! I guess I'll give it a try as soon as I have some hours for potential accessibility-troubleshooting left ;-)
 

proligde

Dabbler
Joined
Jan 29, 2014
Messages
21
br0` shows "Device or resource busy". Therefore, reboot, and it's solved.

Okay, you really made my day! It actually did work without any issues, using only the GUI and without reboot!

Here is what I did:
  1. Remove the IP from my primary NIC -> Apply
  2. Create a bridge
    1. Assign the primary NIC as only member to that bridge
    2. Assign the IP formerly attached to the NIC to the bridge
    3. Apply
  3. Test Network Changes
It took quite a while like 40 seconds before the GUI was able to reconnect, so I had only a few seconds to click the button to make changes permanent.

After that I assigned the NIC of my VM to the bridge and started it. Now it connects to the host without issue. Thanks again! Hope this helps others here searching to the same thing.
 
Last edited:

kewitt

Cadet
Joined
May 2, 2021
Messages
3
Okay, you really made my day! It actually did work without any issues, using only the GUI and without reboot!

Here is what I did:
  1. Remove the IP from my primary NIC -> Apply
  2. Create a bridge
    1. Assign the primary NIC as only member to that bridge
    2. Assign the IP formerly attached to the NIC to the bridge
    3. Apply
  3. Test Network Changes
It took quite a while like 40 seconds before the GUI was able to reconnect, so I had only a few seconds to click the button to make changes permanent.

After that I assigned the NIC of my VM to the bridge and started it. Now it connects to the host without issue. Thanks again! Hope this helps others here searching to the same thing.


This worked for me but too just to make it more clear.
  1. Remove the IP from my primary NIC -> Apply
  2. Create a bridge
    1. Assign the primary NIC as only member to that bridge
    2. Assign the IP formerly attached to the NIC to the bridge
    3. Apply
  3. Test Network Changes
    1. I have 2 nic on 2 different networks so I didn't have to wait 40 seconds 88.0/24 internal uses 89.0/24 external use
  4. Click on Virtualization
    1. Select VM
      1. Device
      2. Nic
      3. Bind to Br#
 

Ziggy

Contributor
Joined
Oct 7, 2015
Messages
157
Hi. Having the same issue. In step 1 do you mean remove (via trashcan) icon the primary ip thru which I communicate via the webgui:
1626430866041.png

and then proceed to step 2? I know I can rollback if I mess up, but I'd rather be sure of what I'm doing in the beginning. Thanks
 

guyp2k

Dabbler
Joined
Nov 16, 2020
Messages
26
Hi. Having the same issue. In step 1 do you mean remove (via trashcan) icon the primary ip thru which I communicate via the webgui:
View attachment 48327
and then proceed to step 2? I know I can rollback if I mess up, but I'd rather be sure of what I'm doing in the beginning. Thanks

No, click on the line and the GUI for that interface will open up, scroll to the bottom and remove IP or uncheck DHCP if you have selected, then click apply and move to the next steps.
 

Ziggy

Contributor
Joined
Oct 7, 2015
Messages
157
Thanks. Tried that by unticking dhcp. At network test phase it hung for ages and then web interface became unavailable. I physically rebooted the machine and gui came up again. Went to step 4 and no bridge was available to bind the nic. I rechecked the network interface and settings were as before.
 

Kasazn

Explorer
Joined
Apr 17, 2021
Messages
60
Hi guys, do you report this bug to JIRA? Is it considered a bug? Previously on CORE, it worked without using this workarounds. As pointed out by others, it might be fixed as SCALE goes through updates.
 

guyp2k

Dabbler
Joined
Nov 16, 2020
Messages
26
Thanks. Tried that by unticking dhcp. At network test phase it hung for ages and then web interface became unavailable. I physically rebooted the machine and gui came up again. Went to step 4 and no bridge was available to bind the nic. I rechecked the network interface and settings were as before.

I have seen it drop the GUI and once the GUI is available you should have the opportunity to save your changes.
 

Ziggy

Contributor
Joined
Oct 7, 2015
Messages
157
Hi guys, do you report this bug to JIRA? Is it considered a bug? Previously on CORE, it worked without using this workarounds. As pointed out by others, it might be fixed as SCALE goes through updates.
Hi Kasazn. I don't know if it's a bug or a feature - in that old phrase. As you and others have said, it's interesting that this workaround isn't required in Core. If I cannot get it to work with the advice here I will report it.
 

proligde

Dabbler
Joined
Jan 29, 2014
Messages
21
I assume that's not a bug - it's just the way KVM networking works according to the post linked at the beginning of this thread ( https://www.furorteutonicus.eu/2013...uest-networking-with-kvm-macvlan-and-macvtap/ ) It's an older post, but I guess it still checks out. Maybe Scale will ship with a bridge set up by default in future releases to make it work out of the box. On the other hand - some users might want that kind of NIC isolation, tho.
 
Top