Resource icon

LACP ... friend or foe?

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
LACP just makes it so that some traffic might get bound to a single NIC.

Yeah, but for the situation described, multiple clients with growth, LACP is an ideal option. The only other possibility would be to get a switch with a 40Gbps uplink, and run 40Gbps to it.

Something cheap like a used Dell N4032 (copper) or N4032F (SFP+) with the QSFP module (wanna say 5KFVW). Or just go way big with the N4064/N4064F. We've got some of the N4032F's here and they're nice but a bit noisy.

Unfortunately I have no use case for trying to stick 40GbE in a FreeNAS box. I believe I recall that the Chelsio T580-CR is supported, along with one of the Intel cards. It'd be hella-fun to try but I wouldn't want to be the one responsible for the mess if it didn't work.

Take that as a cautionary suggestion of a maybe-possibility.
 

VictorR

Contributor
Joined
Dec 9, 2015
Messages
143
20MB/sec? Something's wrong.

Yeah, that was really weird.
Just got back here tonight at 10pm.

Using Black Magic Disk Test for simulated video file streaming:
Direct-connected client is seeing ~700MB/sec writes and ~450-500/MB/sec reads
Via switch is ~600MB/sec and ~450/MB/sec reads

iPerf/Jperf shows more varied speeds on direct-connection from 470-800MB/sec from client to NAS, and 380-650MB/sec the other way. What's a little surprising is how much those numbers vary between different runs of the test, considering there nothing on the NAS and no other client accessing it.

For your Mac users you might want to make 2 targets on FreeNAS and manually put 1/2 your users on one IP and 1/2 on the other IP so you get both interfaces used. LACP just makes it so that some traffic might get bound to a single NIC.

When I woke up this morning with a clearer head, I figured my fall back could always be giving each NIC port it's own sub-domain and assigning 2-3 users to each. That would allow them 12-18 users.

Not that the NAS could deliver full res streams to anywhere near that many users simultaneously. But, they could down-res those files to much smaller proxies for 95% of editing work, then apply the edit marks(conform) back to the original raw at the end.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
iPerf/Jperf shows more varied speeds on direct-connection from 470-800MB/sec from client to NAS, and 380-650MB/sec the other way. What's a little surprising is how much those numbers vary between different runs of the test, considering there nothing on the NAS and no other client accessing it.

Was it singlethreaded? If so, not really that surprising at all. There is *so* *much* happening so quickly that the tiniest little bit of latency amplifies into a dip.
 

VictorR

Contributor
Joined
Dec 9, 2015
Messages
143
5 ports as a LAGG? Get rid of one. Or maybe even three. Betcha you have much better luck with a 20Gbps LAGG. The switch silicon probably doesn't even scale to handling 5 ports.

That's what the XS728T data sheet claims:

DIBQk6Y.png


This thing has to be online by Saturday. So, I will just go with assigning 3 clients to each of two NAS ports with separate IP addresses. Even that is serious overkill
 

VictorR

Contributor
Joined
Dec 9, 2015
Messages
143
Well, well, well, talk about strange. Today I walk into the office, run some iPerf tests - direct-connected 750-880MB/sec writes and 750MB/sec reads vs 5-port LACP ~725/725. Those speeds were consistent throughout the day. An occasional odd 1-2 second dip 100-300MB/sec, then back to the average.

We're going to run a 15-20TB transfer of raw video files up to 4K tomorrow and do md5 checksums to see if there is any corruption. Then, stream multiple simultaneous streams to a single editing station at different resolution/color bit depth. Adding one station at a time until we reach a maximum amount of concurrent streams at each major resolution before stuttering.

It will interesting to see if those occasional dips show in the streams
 

VictorR

Contributor
Joined
Dec 9, 2015
Messages
143
We did the 20TB transfer with MD5 checksums Sunday, giving us a chance to test out a nice little $8 app named Rapid Copy.
Transfer speeds were limited by source Redcode camera card(and checks), so that averaged ~300MB/secs
We added a second transfer from another card and client computer, then watched the interface report in FreeNAS GUI
I should have taken screenshots. Essentially, all TX traffic was routed over one interface, and RX was lightly scattered around a few others. I don't have the network stack knowledge to understand what was going on.

We will be doing another big ingest today or tomorrow and I will take screenshots so you guys can see what is going on.
Are there any network reports from the NAS or switch you would like to see?
 

Grogdor

Dabbler
Joined
Jan 26, 2016
Messages
12
2) LACP REQUIRES BOTH SIDES OF A CONNECTION TO SUPPORT LACP. This means you need a managed or smart switch. Typical consumer grade switches do not support LACP.

Can confirm that the Netgear GS108T ($100 8-port Gigabit managed switch that speaks LACP) worked great, used it in 2012/2013 for streaming HD media from FreeNAS to three Mac Pros for a networked two-way MIDI live VJ gig for 15k+ attendees :D
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Can confirm that the Netgear GS108T ($100 8-port Gigabit managed switch that speaks LACP) worked great, used it in 2012/2013 for streaming HD media from FreeNAS to three Mac Pros for a networked two-way MIDI live VJ gig for 15k+ attendees :D

While some small "web-managed" switches like the GS108T (which is among my favorite for remote deployment because it does vlans and can be powered by PoE) will do LACP, there's not much value to it since there's only a maximum of six clients, which isn't likely to be enough for LACP to be a winner, unless you've got an unusual application that's actually going to have six clients each of which constantly need 200Mbps or more.
 

mikesm

Dabbler
Joined
Mar 20, 2013
Messages
36
Folks, I am trying to figure out how to configure my FreeNAS server to use a 10 Gbps link as the primary link, but a 1 Gbps ethernet as a backup link to a common switch and VLAN. My VMWare hosts can do this without using LACP, but on FreeNAS should I try and do this using the LACP facility? How do I get it to prefer the 10G link and have both sides use it as long as it remains up, but if it goes down to failover to the 1G link?

Apologies if I am missing something obvious.

Thanks!
Mike
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
LACP requires all links to be identical. What ESXi does isn't actually LACP, and is more like something monitoring link status and reconfiguring the interfaces on the fly. Something like that could be written for FreeNAS but does not exist today. AFAIK..
 

mikesm

Dabbler
Joined
Mar 20, 2013
Messages
36
LACP requires all links to be identical. What ESXi does isn't actually LACP, and is more like something monitoring link status and reconfiguring the interfaces on the fly. Something like that could be written for FreeNAS but does not exist today. AFAIK..

So does this mean I should try and make this work with both interfaces on the same LAN?

thx
Mike
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
LACP requires all links to be identical. What ESXi does isn't actually LACP, and is more like something monitoring link status and reconfiguring the interfaces on the fly. Something like that could be written for FreeNAS but does not exist today. AFAIK..
My ESXi does LACP, but it's enterprise plus on vDS :D
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
My ESXi does LACP, but it's enterprise plus on vDS :D

I didn't say ESXi *couldn't* do LACP. I said that what's being described isn't LACP. Geeeez.

So does this mean I should try and make this work with both interfaces on the same LAN?

Sure. Go ahead. It'll break you in all new interesting and unpredictable ways, especially when one of the links goes down, but it's not like anyone wrote a sticky warning against it or anything.

What I meant when I said "Something like that could be written for FreeNAS but does not exist today" was that someone could author all of the complicated software functionality involved in monitoring the link states on multiple links and then migrating connectivity between the two different interfaces, which is either somewhat complex software that mangles up MAC addresses, or possibly something that does something akin to a virtual bridge interface, which would be the traditional UNIX answers. In reality FreeBSD is more cool than that and there's almost certainly some netgraph functionality that could be (ab)used to do this, such as one2many with an xmitAlg of 3, but the reality is that someone would have to write something to make this work correctly but that something does not exist today.

... goes to take the rest of his cranky grinchy pills and contemplates cursing at tapatalk.
 

mikesm

Dabbler
Joined
Mar 20, 2013
Messages
36
At the risk of pressing my luck, what would folks thing about adapting the example 30.3 shown here: https://www.freebsd.org/doc/handbook/network-aggregation.html, which is design for a similar use - overriding a wifi connection with a faster but not always connected ethernet interface. This would get exercised very infrequently in my case compared to the WiFi context, but would seem to be doing the same thing in terms of preferring the use of a specific interface over another and switching MAC addresses around to maintain state.

Thanks
Mike
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
LACP requires the use of the same port speed. This is typically enforced by switch firmware. I have no idea how they would have managed to get it to actually do that with a Cisco switch. If the lagg code in FreeBSD lets you actually do that, because it implements LACP, it would be reasonably considered to be broken behaviour, and I'd think the switch would still refuse to participate in a broken configuration.

I mean, you can try it, and if it works, great, but I really don't want to hear about it if it suddenly breaks after a switch firmware upgrade, or a FreeNAS firmware upgrade, or whatever. That is solidly in the "that should not work" boat. Of course lots of things that should not work do actually "work" for some random definition of that word.
 

Steve Brown

Explorer
Joined
May 7, 2015
Messages
80
Does anyone have trouble setting up LACP on Freenas 9.10 with a dual 10GB Chelsio card?
 

VictorR

Contributor
Joined
Dec 9, 2015
Messages
143
I've finally accepted that LACP does not work reliably for a small 10GbE LAN of ultra hi-def video streaming. Bandwidth to each client varies widely at times - from 700MB/sec down to periods of 10MB/sec, or none. It requires the restarting of client workstations or the LAGG interface, at times.

My guess is that the LAGG algorithm switching data flow between interfaces is not a problem with small, non-sequential/intermittent, requests like databases or web servers. But, during a continuous stream like video editing, it is a deal-breaker. Which is what several of the gurus around here forewarned months ago(not my guess at what is happening, but that it probably wouldn't work).

Oh well, I tried. It made configuration a lot easier, though.
 
Joined
Aug 25, 2014
Messages
89
Cyberjock,

I think the limitations are on the FreeBSD side of things. It indeed does work but the feedback from the Cisco 3560 switch is not enough to see the packet distribution. In my lab I am going to install 2 FreeBSD 10.x systems and see how LACP works between them with a Trunk and 4 VLANs for the 4 subnets. I hope to see some statistics on the Cisco end.

I truly hate the garbage tier 2 switches(Broadcom or Marvell) that say they can do X, Y and Z. When the rubber meets the road I found out they only kind of do X, Y and Z... A 7 year old 3750g-24tss on ebay for $250 is a much better way to work in a LAB than with this crap from Linksys, Belkin, Dell or HP. What a mess! I thought LACP would level the switching field so everyone could port-channel without issues! WRONG! These same wanna be switch vendors are still trying to figure out VLANs and Jumbo frames too! LOL

Thanks,
Joe
Thanks Cyberjock for all the good info. I was wondering when you mention HP switches does that include the HP ProCurve PoE+ switches also? I am looking into methods of improving performance and I have LACP on the list of things I can do now. My FreeNAS SAN is in production mode and is connected to my network via HP ProCurve PoE+ switche(s).
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I try not to break it down to specific models. Without firsthand info (and the fact that too many people see a guy with 18k posts and wrongly think I know everything about everything) I try not to get overly specific. Sorry. :/
 

Marcet

Contributor
Joined
May 31, 2013
Messages
193
Slightly off-topic question : Would you recommend using LACP between two Netgear GS724T switches, or is it also worthless ?
My understanding is, if I aggregate 4 ports, switches will have 4 Gb/s connection between them.
 
Top