LAGG/LACP Network issues? Performance problem?

[rmccullough]

I did take a look at the LACP ... friend or foe?, and while I don't want to be naive I think I understand what LAGG/LACP is supposed to do.

I recently purchase an Ubiquiti UniFi Switch 24 (US-24) managed switch. It was highly rated, and you can purchase them new from Amazon for about what they go for used on eBay. It was replacing a TRENDnet 24-Port (TEG-S24Dg) unmanaged gigabit switch. My FreeNAS server has a Supermicro X9DRi-LN4F+ motherboard that provides 4 x Gigabit Ethernet LAN ports (Intel® i350 GbE controller). The idea was to connect my FreeNAS server to the US-24 utilizing all 4 network ports, and allow multiple clients to not have to share a connection with FreeNAS (this is where I get a bit fuzzy on LACP, but that isn't really my question in this thread). So I configured FreeNAS to use all 4 network interfaces in an LAGG (LACP), then configured the 4 ports on the US-24 to aggregate using LACP. I then used IPMI on my Supermicro to restart networking using /etc/rc.d/netif restart (for some reason this is needed whenever changing network interfaces). After doing this, the LAGG ( lagg0) interface was able to obtain an IP address from my DHCP server. Looking good so far.

Here is what this setup looked like:


Now, I only had the US-24 installed for about 24 hours, and LAGG/LACP configured for about 8 of those. What I noticed was a couple of things:
1) My network performance seemed to not be as good as it was. Accessing the internet was a bit sluggish, almost like it took a couple extra seconds to find the server. I ran some speed tests (fast.com and speedtest.net) and they were close to my numbers before the US-24, but slightly degraded.
2) My MacBook Pro (2017 with TB3/USB-C) had trouble connecting to my FreeNAS server and the services running in jails, but my Windows 10 Pro laptop did not.
3) My WiFi devices started having trouble. They said they were connected to WiFi but no internet was available. I think it was more likely that they could not obtain an IP address from my DHCP server.

I can't say for certain these issues were happening before enabling LACP on the switch, but I know they were happening after it was configured.

At this time I turned off LAGG/LACP on my FreeNAS server and reverted my switch to the TEG-S24Dg. My network is behaving as it was prior to the US-24 being installed.

My question is if this kind of networking issue is typical of enabling LACP? Is it a consequence of mixing consumer hardware (e.g. Linksys router configured as wireless bridge, TRENDnet switches, etc.) with business/enterprise hardware?

 

[rmccullough]

Oh, one thing I forgot to add, I put the WiFi Access Point in the office since it is on the 2nd floor of our house and provides better WiFi coverage than putting it in the crawlspace. I am not sure if it would have made a difference moving the AP downstairs and plugging directly into the US-24. I didn't think to try that.

 

[Elliot Dierksen]

I helped out a friend of mine who has a Ubiquiti UniFi switch, and I was severely underwhelmed. I suppose you can take that with something of a grain of salt as I am a snotty enterprise networking guy. That said, LACP is my preferred method of link aggregation. Unfortunately not every device does it very well. It does matter how each side of the link decides to spread the traffic across interfaces. ESXi can be very particular about that, and discard frames is they don't come to the interface where it expected them. I am a little puzzled as to why you would want FreeNAS to obtain a DHCP address in this configuration. You could have some delays if one side is doing fast LACP, and the other side is doing slow. That could make the DHCP request time out while the underlying layer 2 stuff is still getting established. Does any of this improve if you assign a static IP to FreeNAS?

 

[DrKK]

I used to be a Ubiquiti fanboy myself, until I discovered how thermonuclearly Mikro-Tik blew them right out of the water.

 

[Elliot Dierksen]

I used to be a Ubiquiti fanboy myself, until I discovered how thermonuclearly Mikro-Tik blew them right out of the water.

Is that before or after their big security vulnerability? https://nvd.nist.gov/vuln/detail/CVE-2018-14847
Sorry, I am being kind of a weanie there...

 

[DrKK]

Look, even by the most unforgiving rubric, Mikro-Tik can only be described as EXTREMELY responsive, and EXTREMELY competent, when problems are found.

 

[Elliot Dierksen]

Yes, they do all show as having been fixed. Sorry, I think I am excessively grumpy today after having to further band-aid a network put together with chewing gum and baling wire.

 

[rmccullough]

I helped out a friend of mine who has a Ubiquiti UniFi switch, and I was severely underwhelmed. I suppose you can take that with something of a grain of salt as I am a snotty enterprise networking guy. :) That said, LACP is my preferred method of link aggregation. Unfortunately not every device does it very well. It does matter how each side of the link decides to spread the traffic across interfaces. ESXi can be very particular about that, and discard frames is they don't come to the interface where it expected them. I am a little puzzled as to why you would want FreeNAS to obtain a DHCP address in this configuration. You could have some delays if one side is doing fast LACP, and the other side is doing slow. That could make the DHCP request time out while the underlying layer 2 stuff is still getting established. Does any of this improve if you assign a static IP to FreeNAS?

I did have a static mapping defined on my DHCP server (running on pfSense). However, I get what you are saying, and I could have assigned a static IP to my FreeNAS lagg0 interface.

That said, I am kind of done with this exercise. If I do anything with those additional LAN interfaces, I will add them as a failover (still trying to find good information about this). And if I want to get more speed/throughput, I will look at going to 10G rather than trying to have a half solution by using LAGG/LACP.

Frankly I am a little disappointed in the Ubiquiti hardware as well. Having to "adopt" Ubiquiti hardware to a UniFi controller, and that if the controller was lost it was a pain to get devices re-adopted, and if you did you lost all configuration. Seemed kind of lame to be honest.

What I was really interested in was why my WiFi devices started having trouble when I introduced the Ubiquiti switch.

 

[Elliot Dierksen]

If I do anything with those additional LAN interfaces, I will add them as a failover

That part is pretty simple, and doesn't require any additional config on the switch side. FreeNAS has a list of interfaces, and it sends and receives on the one that is up in order of preference.

And if I want to get more speed/throughput, I will look at going to 10G rather than trying to have a half solution by using LAGG/LACP.

Link aggregation is tricky because it isn't bonding, it is load balancing. Any single conversation can only get the bandwidth of a single link in the link aggregation group. You SHOULD get the effective bandwidth of all the links if the load balancing selection works properly, but that requires understanding the traffic. If the Ubiquiti is load balancing based on destination (IP or MAC address) facing FreeNAS, all the traffic will go down one link. If it load balances based on source or a hash of source/destination, then you might get something out of it. Do be careful with your selection of optics/cables. My recollection was that the Ubiquiti was quite picky about that. Your best bet is fiber when one side is picky about the optics/cables (I mean twinax cables that have a hard wired SFP+ connection on each side). The reason for that is you can make sure each side has optics it likes, and the fiber patch cable is a standard thing.