Only One Interface Used in LACP

[stone961]

FreeNAS-8.0.2-RELEASE-amd64 (8288)

Motherboard: Intel S3000AH
http://download.intel.com/support/motherboards/server/s3000ah/sb/d72579003_s3000ah_tps_1_4.pdf

--From Manual linked above--
NIC 1 is an Intel® 82573E gigabit Ethernet controller resourced with an x1 PCI Express* interface from the Intel® ICH7R (PCI Segment C).
NIC 2 is an Intel® 82541PI Gigabit Ethernet Controller is resourced with a 32-bit/33 MHz PCI Segment from the Intel® ICH7R (PCI Segment A) on the S3000AHLX and S3000AH SKU only.

--from dmesg--
em0: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0x2000-0x201f mem 0x88180000-0x8819ffff,0x88100000-0x8817ffff irq 17 at device 0.0 on pci3
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0x1100-0x113f mem 0x88020000-0x8803ffff,0x88000000-0x8801ffff irq 17 at device 5.0 on pci4


#ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 00:15:17:7c:42:55
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:15:17:7c:42:55
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:15:17:7c:42:55
inet 192.168.1.216 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect
status: active
laggproto lacp
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

---------

Switch is an HP 2848 J4904A
http://h10010.www1.hp.com/wwpc/ca/e...6298-12136298-12136316-12136322-29584733.html

--Config--
interface 45
name "FreeNas EM0"
exit
interface 46
name "FreeNas EM1"
exit


switch# show trunk

Load Balancing

Port | Name Type | Group Type
---- + -------------------------------- --------- + ----- -----
45 | FreeNas EM0 100/1000T | Dyn1 LACP
46 | FreeNas EM1 100/1000T | Dyn1 LACP



switch# show lacp

LACP

PORT LACP TRUNK PORT LACP LACP
NUMB ENABLED GROUP STATUS PARTNER STATUS
---- ------- ------- ------- ------- -------
45 Passive Dyn1 Up Yes Success
46 Passive Dyn1 Up Yes Success



---------

When I setup FreeNas I chose LACP as the method for the link aggregation. On the switch the ports are set for LACP Passive.

Start transferring files from 1 computer on the same subnet and you see the spike about the middle of the graphs all on EM0, as I would expect it to be.

Then there is a lull in the graphs as I ready 3 computers to transfer data to the FreeNas box. My theory was PC 1 would start on EM0, PC 2 would start on EM1 and PC 3 back to EM0. Start transfer on PC1 then PC2 then PC3 and wait a while for the graphs to show decent amount of data. Watching the switch with link activity I see only one port is active during the transfers to FreeNas. Pull the graphs up after a period of time you can see that all traffic is on one interface and half of what a single PC transferring to the FreeNas could pull off.




My question are:
What am I doing wrong in this config that I am only getting traffic on one interface?
Or am I off base thinking traffic from multiple sources should be split across both interfaces?
Why do I get half the network speed on EM0 when I have more than one source transmitting?

 

[jgreco]

This is more of a question for HP Support; it's your HP device that's throwing all that traffic down the one connection. But I do notice you say it's set for "LACP Passive". I'm not sure that's what you want.

Many link aggregation protocols are rather limited. Typically, the default option is to pass traffic over a trunk based on a hash of the source and destination addresses. For a network where two nodes are sending traffic to a single node via two gigE's, the likelihood that both streams wind up on the same gigE is about 50%. Even on networks with careful distribution of addresses, it is pretty easy for hot spots to appear. That's why LACP with gigE is often avoided; most networks where this is a serious problem will replace the gear and use 10gE for the network from the switch to the server. At 10gE, you can also maybe do link agg, and if your clients are all gigE, you're much less likely to notice any hot spots, since the distribution of traffic has a much better chance to level out.

 

[stone961]

From the HP Documentation it defines LACP Active/Passive as:
Active: The port automatically sends LACP protocol packets.
Passive: The port does not automatically send LACP protocol packets, and responds only if it receives LACP protocol packets from the opposite device.

The switch default mode is passive and FreeNas send LACP packet otherwise I wouldn't see the FreeNas box at all.

There is no chance, although I would love it, of going to 10G at this point. Your point about only a 50/50 chance of the data splitting was why I tried 3 workstations all sending data at the same time, figured the odds of all three ending up on the same link were slim. Also from the HP documentation "Traffic Distribution: All of the switch trunk protocols use the SA/DA (Source Address/Destination Address) method of distributing traffic across the trunked links"

The other thing I still can't figure out is the spike of traffic on a single transmit but as soon as I added the second workstation to the transmitting of data, the throughput halved.

This weekend I am going to try different ways to trunk those line and also try the active setting for LACP. Worst case, I end up just using FreeNas round-robin on the lagg0 interface. There are two types of trunks that can be created on the switch and I used the recommended dynamic trunking when you configure two or more ports for LACP. I will also try the static trunk method. I will report back later what I find and hope that it helps someone else on down the line.

 

[jgreco]

The chances of all three ending up on the same ethernet are nontrivially high. This is easy to see in binary, where we consider the binary digit value (0 or 1) to represent choice-of-gigE and position within the number to indicate source node.

000
001
010
011
100
101
110
111

It becomes easy to see why the chances of this happening are still two-in-eight (000 and 111). Or one in four if you like to simplify. ;-) And trust me, that 25% works out to about 75%-in-practice, think Murphy's.

With a small number of hosts, you simply are never going to get ideal results.

 

[stone961]

Just was trying to do a proof of concept to show how it could easily scale to handle more connections thrown its way and still stay redundant in case of a single nic failure. Unfortunately can't simulate 100 workstations all shoving data to the FreeNas device at the same time.

 

[jgreco]

And why not?

 

[stone961]

Sorry wasn't clear.

I can simulate a loss of linkage easily enough and that does work great.

I don't have 100 workstation that I can setup to graphics files at the thing at the same time. Kind of wanted to see how it performed compared to our NetApp

 

[jgreco]

What sort of storage access do you require (CIFS, NFS, etc)? It might be (quite?) a bit of work, but one beefy FreeBSD server and a bunch of jails might be able to send quite the pounding. Most load balancing is done via address hashing, so what you really need is to be able to source traffic from different IP addresses. There is probably not a generalized solution to this problem, but I think it could be done. For NFS, for example, binding to a specific port might allow you to use ipfw to control the source IP address. But NFS is not too likely to be used to be serving 100 workstations with graphics files. So are we talking CIFS? Because I'm pretty sure there are smb tools that will work inside a jail.

I will also point out that you could experimentally try four workstations and make sure that two wind up on each ethernet trunk by experimentation with source/dest addresses. Four is probably sufficient to do fairly harsh testing.

 

[totobel]

damn, i got the same prob
1 ESX => 1 FREENAS (ISCSI) => no load balancing...
no way to change the load balancing method from src/dst ip to something else ?