I installed TrueNAS in the Proxmox VE 8.0 system as a virtual machine.
Network card: Intel X520 (82599ES)
I have enabled the SR-IOV of the network card in Proxmox VE, and I can see that the network card used is VF in the command line of the TrueNAS system.But none of my Docker pods can connect to the management computer. Unless I turn off SR-IOV, the pods will never be able to communicate with the router or other devices.
Pods can ping each other, but cannot ping with TrueNAS.
192.168.4.100-192.168.4.110 is pods ip, 192.168.4.85 is TrueNAS ip.
A brief summary of the problems I encountered may be that SR-IOV cannot perform three-layer switching?
Network card: Intel X520 (82599ES)
root@truenas[~]# lspci -nn | grep Eth
01:00.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
root@truenas[~]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:6f:83:37:9a txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.4.85 netmask 255.255.255.0 broadcast 192.168.4.255
inet6 2408:8207:263b:b4d1:7884:9dff:fe96:5e61 prefixlen 64 scopeid 0x0<global>
inet6 fe80::7884:9dff:fe96:5e61 prefixlen 64 scopeid 0x20<link>
ether 7a:84:9d:96:5e:61 txqueuelen 1000 (Ethernet)
RX packets 389703 bytes 459705201 (438.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 130060 bytes 15352257 (14.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
kube-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.0.1 netmask 255.255.0.0 broadcast 172.16.255.255
inet6 fe80::348f:3aff:fefa:5d40 prefixlen 64 scopeid 0x20<link>
ether 6e:1c:85:f5:c4:1f txqueuelen 1000 (Ethernet)
RX packets 73075 bytes 8438713 (8.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 59881 bytes 11712820 (11.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
kube-dummy-if: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 172.17.0.1 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::74ba:eeff:fe84:f2f8 prefixlen 64 scopeid 0x20<link>
ether de:41:9d:48:2b:c7 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 560 (560.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 48852 bytes 39388684 (37.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 48852 bytes 39388684 (37.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth31dcc515: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::a87b:6aff:fe40:9e9a prefixlen 64 scopeid 0x20<link>
ether ba:01:ad:04:95:f2 txqueuelen 0 (Ethernet)
RX packets 12425 bytes 2599891 (2.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11631 bytes 2245952 (2.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth3fe6b465: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::787c:7fff:fe98:6809 prefixlen 64 scopeid 0x20<link>
ether 7a:7c:7f:98:68:09 txqueuelen 0 (Ethernet)
RX packets 3484 bytes 686976 (670.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4053 bytes 1492480 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth6b4cdf27: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::242a:7ff:fe7b:fb36 prefixlen 64 scopeid 0x20<link>
ether 26:2a:07:7b:fb:36 txqueuelen 0 (Ethernet)
RX packets 534 bytes 43230 (42.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 606 bytes 224072 (218.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth71a6ed38: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::544a:c0ff:fef8:f2f3 prefixlen 64 scopeid 0x20<link>
ether 56:4a:c0:f8:f2:f3 txqueuelen 0 (Ethernet)
RX packets 1 bytes 42 (42.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 93 bytes 11672 (11.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth8f81ad78: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::309c:eff:fe78:7f34 prefixlen 64 scopeid 0x20<link>
ether 72:54:ae:fa:2d:68 txqueuelen 0 (Ethernet)
RX packets 128 bytes 11446 (11.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 219 bytes 678086 (662.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethba2ef0da: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::b6:c9ff:fe2f:b547 prefixlen 64 scopeid 0x20<link>
ether 16:d5:00:80:9b:61 txqueuelen 0 (Ethernet)
RX packets 1102 bytes 154887 (151.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1162 bytes 499392 (487.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethc73e8958: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3c43:eff:fe97:aac2 prefixlen 64 scopeid 0x20<link>
ether 36:49:f2:e5:4d:a6 txqueuelen 0 (Ethernet)
RX packets 1953 bytes 186693 (182.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2067 bytes 203033 (198.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethf28b0b46: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::98af:e7ff:fe69:5e62 prefixlen 64 scopeid 0x20<link>
ether d2:e9:fa:41:89:f4 txqueuelen 0 (Ethernet)
RX packets 53453 bytes 5779320 (5.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40717 bytes 6441461 (6.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I have enabled the SR-IOV of the network card in Proxmox VE, and I can see that the network card used is VF in the command line of the TrueNAS system.But none of my Docker pods can connect to the management computer. Unless I turn off SR-IOV, the pods will never be able to communicate with the router or other devices.
Pods can ping each other, but cannot ping with TrueNAS.
192.168.4.100-192.168.4.110 is pods ip, 192.168.4.85 is TrueNAS ip.
A brief summary of the problems I encountered may be that SR-IOV cannot perform three-layer switching?
I've searched a lot, but I still can't solve this problem. Hope to get some help here, thanks.root@qbittorrent-ix-chart-7dc4895786-s4ghf:/# ping 192.168.4.85
PING 192.168.4.85 (192.168.4.85): 56 data bytes
^C
--- 192.168.4.85 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
root@qbittorrent-ix-chart-7dc4895786-s4ghf:/# ifconfig
eth0 Link encap:Ethernet HWaddr 4E:E9:BB:E7:78:C1
inet addr:172.16.4.223 Bcast:172.16.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:124424 errors:0 dropped:0 overruns:0 frame:0
TX packets:156930 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:18856471 (17.9 MiB) TX bytes:17659981 (16.8 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:8282 errors:0 dropped:0 overruns:0 frame:0
TX packets:8282 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:794064 (775.4 KiB) TX bytes:794064 (775.4 KiB)
net1 Link encap:Ethernet HWaddr E2:17:91:4E:06:94
inet addr:192.168.4.102 Bcast:192.168.4.255 Mask:255.255.255.0
inet6 addr: 2408:8207:263b:b4d1:e017:91ff:fe4e:694/64 Scope:Global
inet6 addr: 2408:8206:2633:2e30:1d7e:b413:8b51:1f0/64 Scope:Global
inet6 addr: fe80::e017:91ff:fe4e:694/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3134 errors:0 dropped:0 overruns:0 frame:0
TX packets:12412 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:460881 (450.0 KiB) TX bytes:544424 (531.6 KiB)
root@qbittorrent-ix-chart-7dc4895786-s4ghf:/# ping 192.168.4.107
PING 192.168.4.107 (192.168.4.107): 56 data bytes
64 bytes from 192.168.4.107: seq=0 ttl=64 time=0.069 ms
64 bytes from 192.168.4.107: seq=1 ttl=64 time=0.042 ms
64 bytes from 192.168.4.107: seq=2 ttl=64 time=0.043 ms
^C
--- 192.168.4.107 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.042/0.051/0.069 ms