Four links, round robin and iscsi, only rx traffic RRing.

[Nalez]

Hello,
I am using freenas 8.2-beta (and also saw the same thing with 8.0.4) where my outbound traffic is only utilizing one network interface. I am presenting out iscsi storage to a 2 node ESXi cluster; and all of the LUNs are set to round robin within vmware. Looking at the traffic stats, only incoming traffic is being balanced out; all traffic leaving freenas is being bound to one interface.

Any ideas on what to look for?

Below is what I am seeing:
Interface Traffic Peak Total
vlan1 in 0.251 KB/s 1.740 KB/s 9.152 MB
out 0.442 KB/s 1.496 KB/s 46.263 MB

vlan10 in 106.699 KB/s 1.896 MB/s 15.865 GB
out 10.055 MB/s 35.145 MB/s 116.144 GB

lo0 in 0.000 KB/s 2.969 KB/s 6.747 MB
out 0.000 KB/s 2.969 KB/s 6.747 MB

em3 in 1.292 MB/s 1.795 MB/s 24.590 GB
out 0.000 KB/s 0.000 KB/s 5.701 KB

em2 in 348.936 KB/s 1.889 MB/s 11.548 GB
out 0.000 KB/s 0.000 KB/s 2.584 KB

em1 in 0.000 KB/s 2.627 MB/s 11.478 GB
out 0.000 KB/s 0.000 KB/s 3.076 KB



em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 10:bf:48:d4:b3:4c
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 10:bf:48:d4:b3:4d
inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 10:bf:48:d4:b3:4e
inet 192.168.10.12 netmask 0xffffff00 broadcast 192.168.10.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 10:bf:48:d4:b3:4f
inet 192.168.10.13 netmask 0xffffff00 broadcast 192.168.10.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=3<RXCSUM,TXCSUM>
ether 10:bf:48:d4:b3:4c
inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 10 parent interface: em0
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=3<RXCSUM,TXCSUM>
ether 10:bf:48:d4:b3:4c
inet 192.168.0.20 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 1 parent interface: em0

 

[jgreco]

Well, several immediately obvious issues:

1) if you have two hosts and four gigE's, you have a problem. Most link aggregation protocols do not just randomly blast packets out ports, but instead build affinity using one of several techniques, so that TCP packets remain correctly ordered and you're not melting the network processing stack on the far end with out-of-order packets.

2) in any case, you do not have link aggregation configured in the above setup

3) the networking configuration you do have is probably invalid, as you have several interfaces with IP addresses on the same network. I guess this is actually acceptable to modern FreeBSD, but you are probably expecting it to magically work the way you want it to, rather than the way it does.