Truenas will not communicate over bridged ports

[Avanesov]

Ok, so my setup is like this:

I have 4 dell r710 servers with four 1Gb ports each. One server is running Truenas, the other 3 are running Proxmox. In order to isolate the traffic between each of the four servers from my household network I created a bridge (bridge192) on each server and added the last 3 physical ports as members. Each of the last three physical ports on each of the four servers connect directly to one of the remaining three servers so that each server has a direct 1Gb connection to all of the others.

The proxmox servers can all communicate (ping) each other with this configuration but not the Truenas server, likewise the Truenas server cannot ping any of the other servers over the bridged connection. This leads me to believe I have a configuration problem with my Truenas bridge. I set it up using the GUI interface.

My bridge IP addresses for the servers are (addresses are not assigned to bridge members):
Proxmox1: 192.113.1.1/24
Proxmox2: 192.113.1.2/24
Proxmox3: 192.113.1.3/24
Truenas : 192.113.1.10/24

Version: TrueNAS-12.0-U4.1
Output of ifconfig:

Code:

root@truenas[~]# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5a
        inet 10.113.2.10 netmask 0xffffff00 broadcast 10.113.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5c
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
bce2: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5e
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
bce3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:60
        media: Ethernet autoselect
        nd6 options=1<PERFORMNUD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge192: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:bf:01:42:0b:c0
        inet 192.113.1.10 netmask 0xffffff00 broadcast 192.113.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        member: bce3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

If anyone can shed some light on why Truenas doesnt want to talk I would appreciate it. I suspect I am overlooking a BSD specific config option, or that maybe BSD doesnt like the direct peer-to-peer connection with the other servers, but im not sure and dont know how to check it.

 

[Patrick M. Hausen]

Did you disable hardware offloading for all physical member interfaces?

 

[Avanesov]

Ok, just did that but im not sure the settings are sticking. I select each interface and and check the box for "Disable Hardware Offloading" then test and save the configuration and reboot. Once back up when I go to the interface settings again I immediately get a popup asking to confirm if I really want to disable offloading and I can see on the main screen behind the popup that the box isnt checked until I confirm. I can rinse and repeat this indefinitely.

ifconfig has some differences now although im not sure of the relevance:

Code:

root@truenas[~]# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:76:97:5a
        inet 10.113.2.10 netmask 0xffffff00 broadcast 10.113.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:5c
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:5e
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:76:97:60
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge192: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Isolated server to server network
        ether 02:bf:01:42:0b:c0
        inet 192.113.1.10 netmask 0xffffff00 broadcast 192.113.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

 

[Patrick M. Hausen]

Once back up when I go to the interface settings again I immediately get a popup asking to confirm if I really want to disable offloading and I can see on the main screen behind the popup that the box isnt checked until I confirm.

That's ok, it will stick.

Compare the output for bce1-3 with the one for bce0. Most HW offloading features like RXCSUM, TXCSUM, ... are missing.

The ifconfig for the bridge and the member interfaces looks good. Still no worky?

 

[Avanesov]

Ya, still doesnt work. I ran ping to and from each of the Proxmox servers and Truenas. The puzzling thing is that all the servers are the same hardware for the MB, NICs etc and the ports work as a single non-bridged connection. I remember that on a previous install of Freenas on this same server I had problems getting link aggregation to work as well to the point I gave up and never revisited it before I reloaded anyway.

 

[Avanesov]

Here is an "ip a" for one of my proxmox servers...is there by chance an incompatibility between the configurations?

Proxmox3 in my example from above:

Code:

root@r710sff-03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:bf brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c3 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr192 state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c5 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:bf brd ff:ff:ff:ff:ff:ff
    inet 10.113.2.90/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::226:b9ff:fe5d:19bf/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:26:b9:5d:19:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.113.1.3/24 scope global vmbr192
       valid_lft forever preferred_lft forever
    inet6 fe80::226:b9ff:fe5d:19c1/64 scope link
       valid_lft forever preferred_lft forever

 

[Avanesov]

FYI, I named the bridge "vmbr192" as a kneejerk to remain consistent with the naming scheme of the original virtual machine bridge that is created by default before I realized what I had done...