Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

LACP ... friend or foe?

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

sandsfoot

Newbie
Joined
Dec 17, 2018
Messages
1
Hello

I have 4 ESXi hosts with 10GB cards and a FreeNAS with 2 x Dual 10GB cards. I do not have a 10GB switch available.

esx1: 192.168.1.2/24 nas1-1:192.168.1.1/24
esx2: 192.168.2.2/24 nas1-2:192.168.2.1/24
esx3: 192.168.3.2/24 nas1-3:192.168.3.1/24
esx4: 192.168.4.2/24 nas1-4:192.168.4.1/24

I'm setting up a 4 node DEV cluster and will have nested ESX clusters as well, so would like to use the same FreeNAS/iSCSI target. If I configure point-to-point on the physical ESX hosts, all is good.

But if I utilize the nested hosts, then their nested IP could migrate to another physical host and the point-to-point would fail.

For example, if esx-nested-1: 192.168.1.11/24 is running on esx1, all is good. But if this nested host is migrated to esx2, then the point-to-point fails.

Question - can LACP be used for where I have no switch - so set up a 4 member LACP/LB interface?

esx1: 192.168.1.2/24 nas1-1:192.168.1.1/24
esx2: 192.168.1.3/24 nas1-1:192.168.1.1/24
esx3: 192.168.1.4/24 nas1-1:192.168.1.1/24
esx4: 192.168.1.5/24 nas1-1:192.168.1.1/24

Thanks in advance.
 

guemi

Junior Member
Joined
Apr 16, 2020
Messages
16
Testing my luck because I'm out of ideas...

I have 2 10GBase-T NIC's in the system, Aquantia 107 to be exact using the driver provided here: https://www.ixsystems.com/community...tion-aqc107-xg-c100c.76856/page-2#post-580953

When putting DHCP on any of the two cards, it works just superb.

However, when trying to configure LACP - I just cannot get an IP-address.

On the other end of the cables is an Netgear XS716T configured like this:
1587412687885.png


When "LAG TYPE" is set to static, LAG State becomes "Link UP". However, when set to LACP Lag state goes to Link Down.


And obviously, my interface in FreeNAS does not give me an IP-address, configuration is as follows:

1587412811767.png



I should mention that there is a third NIC here, the onboard 1gbits card which is configured with a static 10.100.100.253/24 IP-address, which is the same subnet as I am trying to get DHCP on.

This is purely because I am working remotely so I need to have an IP-address to reach the webgui on and the onboard won't be plugged in anymore once the LAG starts working, but I doubt that's what's prevent the LAG from getting a DHCP given that the individual cards work fine.

I've ran both a static and DHCP working perfectly fine with both ways on the actual NIC's, so it does work - it's the just LAG that won't work.


Any ideas are helpful!
 

JoeAtWork

Member
Joined
Aug 20, 2018
Messages
65
I suggest make sure it works with a 1gig switch and then make sure it works in Linux with your Netgear switch. Cisco swithes you can debug lacp and see what is happening. Does Netgear say your config is fine? I am not sure LACP is going to help much if your using CIFS or iSCSI. NFS 4.1 yes but that does not require LACP.
 

the_jest

Member
Joined
Apr 16, 2017
Messages
40
I have a couple of basic questions that mainly, I guess, have to do with the official documentation about link aggregation.

My home network has a FreeNAS box that has two NICs; I've only been using one. I've started to use some of my enforced time at home to play around with, and thus learn more about, various bits of my infrastructure, and thought it might be worthwhile to use the second NIC. I do have managed switches that support LACP. (I want to stress that this really is just for the hell of it; I'm not doing this because I've observed that I'm unable to get the throughput I need and am seeking a solution.) The docs give the impression that link aggregation is generally a good thing, and also sort of give the impression that failover is some kind of second-tier practice.

I came here to ask a question about how to setup link aggregation, and saw this current discussion, which, despite originating five years ago, is still pinned.

So, my two questions are: If link aggregation is really as fraught as this thread seems to suggest, and has been for years, why doesn't the documentation give at least a hint of a warning about it?

And, is there any real downside to plugging my second NIC in and turning on failover mode? (Again, I've never had my primary NIC fail, but....)
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,067
I came here to ask a question about how to setup link aggregation, and saw this current discussion, which, despite originating five years ago, is still pinned.
Correct. Industry standards don't change rapidly, and this one isn't likely to change in any significant way, due to the science of it all.

So, my two questions are: If link aggregation is really as fraught as this thread seems to suggest, and has been for years, why doesn't the documentation give at least a hint of a warning about it?
Because the developers who write the official documentation play with nice, well-supported hardware and don't try janky setups or have unrealistic expectations such as "my 2Gbps LACP link should be fully saturated," therefore they are not focused on documenting the pitfalls.

Because the guy who wrote the sticky comes from the real world of actual professional Internet infrastructure engineering, is aware of the pitfalls, and got tired of forum users coming in with janky setups and totally unrealistic expectations, and the failure of it to work the way they wanted was getting blamed on FreeNAS.

It isn't the developers sitting here intercepting that. I'm the one sitting here on the front lines, so I often end up needing to be the one who introduces a healthy dose of reality into their expectations, which is frustrating when someone who's never done this stuff thinks they know more than someone who's been doing it for years.

LACP is great for the things that LACP is great for. That most emphatically is NOT so that you can LACP 4x1GbE ports together in an attempt to get a faster 4GbE link between your PC and NAS. If you have a classroom of clients, it is ABSOLUTELY a good option, not as good as upgrading to 10GbE, but definitely better than letting everything pound through a single 1GbE. You still have to make sure you're using well-supported hardware.

Basically if you think I'm wrong, you're free to prove me wrong on any point. I like being wrong. A day I'm wrong is a day I learn something new. That's actually pleasant, because a lot of my work involves being cynically correct. See, the thing is, I generally don't get proven wrong because when I talk about stuff, I talk about stuff I have direct experience with, and facts are on my side. If I don't know something, I just don't talk about it, and I am therefore still not saying wrong things about it. :smile:

And, is there any real downside to plugging my second NIC in and turning on failover mode? (Again, I've never had my primary NIC fail, but....)
Nope! Absolutely a good idea, if your situation can make use of it. In hypervisor environments where I have redundant switches and ethernet interfaces, I run all VLAN's on both switches, then configure storage to "prefer" one switch and other traffic to "prefer" the other, which means a switch getting its firmware upgraded doesn't knock stuff offline. Your switch does not need to support LACP in order for this to work.
 

JoeAtWork

Member
Joined
Aug 20, 2018
Messages
65
If you have LOTS of clients connecting to your FreeNAS machine LACP will help you. If you have ONE VMware server connecting to ONE FreeNAS machine you have to use iSCSI multi pathing or NFS 4.1 multichannel/trunking. Both work best with multiple VLAN's i.e. multiple subnets and have the subnet pinned on the interface you want to fill up, i.e. 4 port NIC you need 4 VLANS

Linux or other UNIXes that support NFS 4.1 will work as well as long as you tell your client to use those addresses that align with your subnets. I am assuming your UNIX client has a 2 or 4 port NIC in it. I am not sure if Windows can use multiple connections as a client.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,770
If you have LOTS of clients connecting to your FreeNAS machine LACP will help you. If you have ONE VMware server connecting to ONE FreeNAS machine you have to use iSCSI multi pathing or NFS 4.1 multichannel/trunking. Both work best with multiple VLAN's i.e. multiple subnets and have the subnet pinned on the interface you want to fill up, i.e. 4 port NIC you need 4 VLANS

Linux or other UNIXes that support NFS 4.1 will work as well as long as you tell your client to use those addresses that align with your subnets. I am assuming your UNIX client has a 2 or 4 port NIC in it. I am not sure if Windows can use multiple connections as a client.
SMB has support for similar voodoo, but I'm not sure what the state of that is in Samba. It used to come with data corruption warnings in the beginning. Windows even extends it to multiple NICs on a single subnet, but I forget the details and I think Samba only supported separate subnets.
 

the_jest

Member
Joined
Apr 16, 2017
Messages
40
LACP is great for the things that LACP is great for. That most emphatically is NOT so that you can LACP 4x1GbE ports together in an attempt to get a faster 4GbE link between your PC and NAS. If you have a classroom of clients, it is ABSOLUTELY a good option, not as good as upgrading to 10GbE, but definitely better than letting everything pound through a single 1GbE. You still have to make sure you're using well-supported hardware.

Basically if you think I'm wrong, you're free to prove me wrong on any point. I like being wrong. A day I'm wrong is a day I learn something new. That's actually pleasant, because a lot of my work involves being cynically correct. See, the thing is, I generally don't get proven wrong because when I talk about stuff, I talk about stuff I have direct experience with, and facts are on my side. If I don't know something, I just don't talk about it, and I am therefore still not saying wrong things about it. :)
Goodness, no, I didn't mean to suggest you were wrong! Quite the opposite, it seemed more likely that the docs were wrong, or at least vague. And I wasn't trying to get big throughput to my PC, I understand that's not the point. At best I would set up regular throughput to my desktop, and then do something on another computer so I could say "See, it works!"

the_jest said:
And, is there any real downside to plugging my second NIC in and turning on failover mode? (Again, I've never had my primary NIC fail, but....)
Nope! Absolutely a good idea, if your situation can make use of it. In hypervisor environments where I have redundant switches and ethernet interfaces, I run all VLAN's on both switches, then configure storage to "prefer" one switch and other traffic to "prefer" the other, which means a switch getting its firmware upgraded doesn't knock stuff offline. Your switch does not need to support LACP in order for this to work.
Great, thanks.

Forgive me if I should ask this in a separate thread, but: if I only have the two NICs, how do I set this up? As the docs warn, once I change the NIC that the web interface uses, I'll get kicked offline and be in trouble, but they don't say what to do about it. Right now I'm using eth0 on a static 192.168.1.10; if I want to add igb0 as the failover, is there a safe way to do this?

(Similarly, if I did want to create an LACP aggregation, how would that work? I imagine there might be an easier way to do the failover than the LACP.)
 

JoeAtWork

Member
Joined
Aug 20, 2018
Messages
65
SMB multi channel is VooDoo as I believe it is un-documented. NFS 4.1 is documented and for VMware is best as an NFS share that is offline does not cause your hypervisor to hang while trying to connect/timeout like VMware and iSCSI do.

Microsoft doing multiple nics on the same subnet is the same stuff that Dell/Equallogic does on the iSCSI filers they used to have, PS4100/6500. Not sure if they drink the same kool-aid today but back in the day it was bad... You had a supposed tier 1 vendor telling the storage guys to do stupid IP tricks.
 

Elliot Dierksen

Neophyte Sage
Joined
Dec 29, 2014
Messages
925
This is a stock rant of mine. Repeat after me: LAGG is NOT NOT NOT aggregation bonding. It is load balancing. When packets are transmitted of a LAGG (or port channel in Cisco switch speak), the transmitter uses a hashing algorithm to determine which link member it will use to transmit the packet. Any one connection can get no more throughput than that of a single physical link. With enough different connections and the most effective hashing method for your traffic patterns you can THEORETICALLY saturate all links, but I have never seen that happen in the real world. Choosing the right hashing method is crucial! I'll give you an example from my setup. All my VM hosts have an interface on the same separate network as my FreeNAS units. The FreeNAS units have a LAGG going to the switch. The default hashing method is by destination MAC. This is useless for the connection to the FreeNAS because all packets going to it will have the same destination MAC. The switch should use source MAC to load balance sending towards FreeNAS, but FreeNAS should use destination MAC (or IP) when sending towards the switch. Otherwise you get all the traffic using the same link. Then all you get from a LAGG is port redundancy. Sorry, that ended up being a longer rant than I had originally intended...

Edited to reflect @jgreco 's accurate comment.
 
Last edited:

Elliot Dierksen

Neophyte Sage
Joined
Dec 29, 2014
Messages
925
... quietly points out the "A" in both LACP and LAGG does actually stand for aggregation.
How dare you ruin a perfectly good rant with facts! :smile: I meant to say NOT bonding. Sigh....
 

pjdouillard

Newbie
Joined
Jun 5, 2020
Messages
1
Hello! I am a new user here and I came here looking for some answers. Found this thread and decided to post to it.

I am wondering why the "LACP behavior" of my FreeNAS changed without me doing anything in the course of some heavy file transfer this morning.

My setup is like this:
FreeNAS server running on a HP DL380 G7 and the 4 built-in NIC are in LACP with the option lagghash l3,l4 (default in FreeBSD for the bonding hash is l2,l3,l4) so that load is getting distributed properly among the interfaces by tcp connection based on ports used - and not the MAC.
The uplink switch (EdgeSwitch Lite 24) ports are also configured in LACP with load balancing mode (or hashing) set as "Source/Destination IP and TCP/UDP Port Fields" to reflect the same mode as FreeNAS.

So like I said, I was transferring some large amount of data from the FreeNAS server to two distinct physical servers - because I am re-purposing the DL380 for another project - and both servers were receiving their data at 1Gbps on their NICs. Things are working properly as they should.

Later this morning, I started some other large transfer again from FreeNAS to the same two servers, but now FreeNAS is only using one link instead of two like it did 90 min before. There is no error on any of the NICs of the DL380, nor on the switch ports in LACP. I changed nothing so I am really wondering what is happening on the FreeNAS side since it was working as it had always worked before.

LACP hashing mode using layer3 (IP address) and Layer 4 (ports) is how to setup any LACP to distribut the load by TCP connection - so you can get more output of your lagg. Single TCP session will still be capped at 1Gbps since each link are 1GbE, but multiple sessions are distributed. Standard LACP will use layer (MAC) and thus no balancing is done because the source MAC is always the same.

Any ideas as to why this suddenly changed?
 

vitaprimo

Junior Member
Joined
Jun 28, 2018
Messages
16
VMware and some storage manufacturers have butchered it into working because the average storage admin is too much of a doof to also have to learn proper IP networking, but it doesn't work on FreeBSD because it's fundamentally broken and LACP is designed to handle this sort of issue.

By "butchered" do you mean something like this?:


Screen Shot 2020-08-07 at 01.18.45.png

I'm trying out FreeNAS and went with iSCSI when I read about the special integration it has with FreeNAS, now I'm trying to get back to LACP and I've been researching whether if sticking with iSCSI and the performance hit of synced NFS.

I looked around for server appliances in the rack for something using LACP+iSCSI; a Synology unit, and it had the multiple NICs + non-overlapping subnets = ✔︎ MPIO repeated everywhere, however, in their KB there is some mention of LACP+iSCSI, it's very loose which is understandable since they're talking about two different protocols which don't even fall within the same category in addition of being accommodating enough to whatever the client/server on the other side if going to use--there are way too many variables. I can understand why they might want to be purposely vague but it doesn't make it any less frustrating when you have missing pieces.

I then took a smaller of the Synology units, replaced its disks with flash media and setup LACP+iSCSI on it and tested; I just needed to see throughput over 1Gb/s to confirm it's using more than one link for the data stream. It did, not by much because, I assumed because it's underpowered, because it's older, still constructing up the new array, IDK. It worked.

I went back to the docs from VMware and noticed that each point upgrade vSphere had some improvement; from 6.5 to 7 they all improved on LACP. But, LACP is a ratified standard, they can't just "improve" it, after going though Microsoft's Windows Server's and System Center [horrible] docu on LACP where they outright rename standards on the fly started seeing it differently because all vendor seem to have some new tech, feature or whatever but this side (the user/admin) it just seems like they all are at various stages of support for LACP.

I'm trying to be super open minded on this because I know barely the about it and I do want to learn but I keep thinking; if the team at VMware (and others) worked around the limitations of LACP and achieved truly making out of LACP the abstraction of the physical link while obviously keeping compatibility or else switches wouldn't cooperate, have they really done something that bad?

I've weeded myself from a few products when I thought vendor lock it was about to set, I still have PTSD episodes from iCloud and iTunes Match , I prefer standards for the same reason, but if it makes your life easier and plays nice with others…

I know FreeBSD are very by the book, it's actually the one thing I relate to FreeBSD, but could it be that this is the one time where it would maybe be OK the go around it and join the others?…maybe evolve the standard. :)


I would totally give 'em a pass if I was the StandardsMaster or whatever they call the wizard that machetes black cats into protocols.

All that aside; your posts, I've went through a couple and they are really good info. A good info dump beats endlessly-linked wikis any day.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
893
Here we go again... today I discovered that my failover LAGG had the interfaces swapped.

I could swear I had them right in the past as otherwise the system wouldn't have been able to transfer 250+MB/s but there we are. Once I unplugged IGB0, IX0 sprang to life and all was well in the land. Once my backup is done, I guess I'll rebuild the LAGG...

Kudos to Mikrotik for making it easy to see via their RouterOS Interfaces tab how the data was getting from A-B. Currently, about 99% of the switch traffic is just this backup. :smile:
 

jayecin

Member
Joined
Oct 12, 2020
Messages
47
I just finished getting LAG setup on my NAS. NAS has a 4 x 1GB Intel PCIE NIC connected to a Juniper EX 3300 series switch. Juniper EX series switch LAG load balancing is not configurable, however its already pretty advanced utilizing 802.3ad. The switch uses a hashing algorithm that utilizes Source IP, Destination IP, Source MAC, Destination MAC, Source Port and Destination Port. This greatly increases the ability to utilize all the ports as individual traffic flows between two hosts can be split among the aggregate interfaces. So IP1 to IP2 on FTP will use a different physical port than IP1 to IP2 on SFTP.

From Juniper regarding LAG Load balcning
In order to effectively increase the bandwidth of the LAG, traffic needs to be balanced across the member links. The balancing is done by the LAG hashing algorithm, which is designed to have the member links used increasingly equally as the traffic profile gets more diverse. The LAG hashing algorithm on EX Series switches determines the member link to be used for an incoming frame/packet depending on a subset of the following values in the frame/packet header:
  • Source MAC address
  • Destination MAC address
  • Source IP address
  • Destination IP address
  • Source port
  • Destination port
  • IPv6 flow label
  • MPLS label(s)
https://kb.juniper.net/InfoCenter/index?page=content&id=KB22943&actp=search

This seems very similar to the way FreeNAS does the Load Balance option

Load Balance: balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. This is a static setup and does not negotiate aggregation with the peer or exchange frames to monitor the link. The hash includes the Ethernet source and destination address, VLAN tag (if available), and IP source and destination address. Requires a switch which supports IEEE 802.3ad static link aggregation.

So if your switch supports 802.3ad LAG link aggregation, you will see a greater performance and link utilization of aggregate interfaces over a simpler LACP, Round Robin configuration without needing a heavily host saturated network to get proper distribution.

Edit: Take this back maybe, it was working but once I changed FreeNAS Lag to LoadBalance it brought down the LAG. Investigating.

Edit2: While the juniper side is configured with 802.3ad the FreeNas side LAG wont come up when configured with Load Balance, has to use LACP. So im not sure if just the juniper side is doing 802.3ad or if they are negotiating it.

Edit3: It appears to be working as expected. When performing a speed test from a single VM, I can see multiple physical interfaces being utilized through the LAG when performing multiple speed tests at the same time and even balances the traffic when performing only a single speed test, pretty evenly as well. I have 1gbps internet.

Edit4: While the LAG port works, it also seems to be creating a network loop. I am constantly seeing MAC addresses appear off the LAG interface on my switch, causing the devices owning that MAC to drop packets and drop from the network. It seems to happen pretty quickly, but its a problem. Once i disabled the LAG interface, the issue went away.

TLDR: Get an 10GBe adapter if you want more bandwidth.
 

Attachments

Last edited:

Elliot Dierksen

Neophyte Sage
Joined
Dec 29, 2014
Messages
925
Keep a couple things in mind. LACP or not has to do with the the LAG getting established. I am a strong proponent of LACP since it requires a successful protocol negotiation to add a member to a LAG. How the traffic is distributed is a an entirely separate question. Most switches I have worked with (Cisco, HP, Juniper) use the destination (MAC or IP) as the first criteria. That works well for most cases, but not for the traffic going TO your FreeNAS. FreeNAS (I think) by default uses destination as one of the first criteria which makes sense. You would want the switch to which FreeNAS is connected to use source address (MAC or IP) and the first criteria to distribute the traffic towards FreeNAS more evenly. Everything going to FreeNAS has the same destination. I am talking about storage there, not any VM/jail traffic.
 
Top