Truenas will not communicate over bridged ports

A

Avanesov

Cadet
Joined
Aug 1, 2021
Messages
5
Ok, so my setup is like this:

I have 4 dell r710 servers with four 1Gb ports each. One server is running Truenas, the other 3 are running Proxmox. In order to isolate the traffic between each of the four servers from my household network I created a bridge (bridge192) on each server and added the last 3 physical ports as members. Each of the last three physical ports on each of the four servers connect directly to one of the remaining three servers so that each server has a direct 1Gb connection to all of the others.

The proxmox servers can all communicate (ping) each other with this configuration but not the Truenas server, likewise the Truenas server cannot ping any of the other servers over the bridged connection. This leads me to believe I have a configuration problem with my Truenas bridge. I set it up using the GUI interface.

My bridge IP addresses for the servers are (addresses are not assigned to bridge members):
Proxmox1: 192.113.1.1/24
Proxmox2: 192.113.1.2/24
Proxmox3: 192.113.1.3/24
Truenas : 192.113.1.10/24

Version: TrueNAS-12.0-U4.1
Output of ifconfig:
Code:
root@truenas[~]# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5a
        inet 10.113.2.10 netmask 0xffffff00 broadcast 10.113.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5c
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
bce2: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5e
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
bce3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:60
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge192: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:bf:01:42:0b:c0
        inet 192.113.1.10 netmask 0xffffff00 broadcast 192.113.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        member: bce3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>


If anyone can shed some light on why Truenas doesnt want to talk I would appreciate it. I suspect I am overlooking a BSD specific config option, or that maybe BSD doesnt like the direct peer-to-peer connection with the other servers, but im not sure and dont know how to check it.
 
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Did you disable hardware offloading for all physical member interfaces?
 
A

Avanesov

Cadet
Joined
Aug 1, 2021
Messages
5
Ok, just did that but im not sure the settings are sticking. I select each interface and and check the box for "Disable Hardware Offloading" then test and save the configuration and reboot. Once back up when I go to the interface settings again I immediately get a popup asking to confirm if I really want to disable offloading and I can see on the main screen behind the popup that the box isnt checked until I confirm. I can rinse and repeat this indefinitely.

ifconfig has some differences now although im not sure of the relevance:
Code:
root@truenas[~]# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5a
        inet 10.113.2.10 netmask 0xffffff00 broadcast 10.113.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:5c
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:5e
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:60
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge192: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Isolated server to server network
        ether 02:bf:01:42:0b:c0
        inet 192.113.1.10 netmask 0xffffff00 broadcast 192.113.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
 
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Avanesov said:
Once back up when I go to the interface settings again I immediately get a popup asking to confirm if I really want to disable offloading and I can see on the main screen behind the popup that the box isnt checked until I confirm.
Click to expand...
That's ok, it will stick.

Compare the output for bce1-3 with the one for bce0. Most HW offloading features like RXCSUM, TXCSUM, ... are missing.

The ifconfig for the bridge and the member interfaces looks good. Still no worky?
 
A

Avanesov

Cadet
Joined
Aug 1, 2021
Messages
5
Ya, still doesnt work. I ran ping to and from each of the Proxmox servers and Truenas. The puzzling thing is that all the servers are the same hardware for the MB, NICs etc and the ports work as a single non-bridged connection. I remember that on a previous install of Freenas on this same server I had problems getting link aggregation to work as well to the point I gave up and never revisited it before I reloaded anyway.
 
A

Avanesov

Cadet
Joined
Aug 1, 2021
Messages
5
Here is an "ip a" for one of my proxmox servers...is there by chance an incompatibility between the configurations?

Proxmox3 in my example from above:
Code:
root@r710sff-03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:bf brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c3 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c5 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:bf brd ff:ff:ff:ff:ff:ff
    inet 10.113.2.90/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::226:b9ff:fe5d:19bf/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.113.1.3/24 scope global vmbr192
       valid_lft forever preferred_lft forever
    inet6 fe80::226:b9ff:fe5d:19c1/64 scope link
       valid_lft forever preferred_lft forever
 
A

Avanesov

Cadet
Joined
Aug 1, 2021
Messages
5
FYI, I named the bridge "vmbr192" as a kneejerk to remain consistent with the naming scheme of the original virtual machine bridge that is created by default before I realized what I had done...
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Truenas will not communicate over bridged ports"

Similar threads

O
Jail cannot access network from datacenter
Replies
7
Views
2K
owenfi
O
roysen
My TrueNAS seems to use the wrong network card.
Replies
3
Views
3K
Patrick M. Hausen
Patrick M. Hausen
Excaliber
Slow internet download speed (local is working great)
Replies
2
Views
3K
Excaliber
Excaliber
R
  • Locked
Networking SNMP
Replies
5
Views
3K
tvsjr
T
L
Bridge0 Bridging Two VLAN's Together
Replies
1
Views
2K
lexcomman
L
Top