Issue with LACP?

[mloebl]

I'm new to FreeNAS but have been reading thru the forums for awhile, so please bear with me. Setup is as such -

FreeNAS 8.0.4
- Intel Gigabit ET Dual Port NIC
- lagg0 configured as LACP
- Onboard Marvel gigabit nic for admin

Server - Windows
- Intel Gigabit ET Dual Port NIC
- Teamed and configured for LACP
- Onboard Intel gigabit nic for Admin

Switch
- Dedicated TP Link TL-SG3210
- Two pairs currently configured as LACP ports
- One port vlaned for admin, rest vlaned for dedicated network for iSCSI

Admin NICs are on a separate switch.

I'm currently looking at using the FreeNAS for iSCSI on the Win7 box.

Essentially the switch I see is cycling in it's log and bouncing the LACP config for only the FreeNAS side, and as a result never truly bringing the network up. If I turn off LACP on the switch (for just the two FreeNAS ports), then switch to standard LAG, then configure FreeNAS to use one of the other methods such as load balance, it works fine. I thought it may have been the switch (it's brand new) maybe having a bug, but I see from searches others are using TP-Link, and the Windows side has been solid.

I'm also seeing another potentially related/unrelated issue that if I remove the LACP port from config on the FreeNAS ui, the whole box seems to ether loose networking or hang (I lose access thru the 3rd nic used for admin.) I haven't dug too deeply yet to see if it's actually locked or just lost it's mind with the NICs, I'll check tonight when I get home as it's currently in this state again now...

Thanks!

-Mike

 

[louisk]

I don't know anything about TP Link switches, but I've seen similar behavior on Cisco switches. It means the LACP connection isn't configured properly. I would go back and make sure there aren't any other options you can work with for LACP. On the Cisco side, I had to change the channel-group mode from active/passive to just 'on'.

 

[TheSmoker]

Any updates on this topic? I've tried the same dual port intel nic with TP-Link SG3210 but no luck ...

 

[mloebl]

Unfortunately no :( I think the issue is the LACP rate. In Windows, Linux, etc it's settable. However I don't see a setting in FreeBSD or the TP-Link switch to change this. I can't remember how I initially found this in Google searches, but I remember that the TP-LINK logs showed it flapping back and fourth every second (fast rate). My theory at the time was FreeBSD may be the slow rate (30s) and TP-LINK is fixed at fast (hence why I see the flapping in the log every second). I gave up for now unless someone knows of a way with the TP-Link to change the LACP Rate or had any other ideas.

-Mike

 

[mloebl]

BTW I should make it clear it's totally a theory about LACP rate, I'm not sure...

-Mike

 

[TheSmoker]

I am trying different things right now on the SG3210 ... if I add only 1 link it works and it's stable. If I add the second link it start the flip-flop thing.
Tried with both static LAG and LACP.

 

[TheSmoker]

If you add it as a static LAG it works, although the performance sux ... it's even worse than single link ...

 

[mloebl]

If you add it as a static LAG it works, although the performance sux ... it's even worse than single link ...

Yeah I noticed the same... In comparison I have 2 other LACP connections (one Linux, other Windows) and they work great.

-Mike

 

[TheSmoker]

Yeah I noticed the same... In comparison I have 2 other LACP connections (one Linux, other Windows) and they work great.

-Mike

For writing I get an average of 85MB/sec and for reading i get 116MB/sec ... like wtf?!?

 

[TheSmoker]

It seems the problem is withing the LACP coding in 8.3 and 9 which share the same code base. http://freebsd.1045724.n5.nabble.co...affic-distribution-performance-td5725545.html

Can one of the FreeNAS devs have a look at this?

 

[noogai93]

I believe i'm having the same issue with a sg3216 and a intel quad port, switch log shows:

1 2013-05-22 23:57:45 LAG level_6
Canceled Link Aggregation Group 1, members: Port 9-12.

2 2013-05-22 23:57:44 LAG level_6
Added new Link Aggregation Group 1, members: Port 9-12.

haven't found a solution, have you come across anything yet ?

 

[mloebl]

I believe i'm having the same issue with a sg3216 and a intel quad port, switch log shows:

1 2013-05-22 23:57:45 LAG level_6
Canceled Link Aggregation Group 1, members: Port 9-12.

2 2013-05-22 23:57:44 LAG level_6
Added new Link Aggregation Group 1, members: Port 9-12.

haven't found a solution, have you come across anything yet ?

I gave up and replaced the switch with a Cisco 10 port SG300. I hope to configure LACP again and test it out.

-Mike

 

[louisk]

I no longer have a cisco switch to post the configs for. I've switched to Juniper. I haven't noticed any performance problems with LACP with 8.x or 9.x. Tests with a single host regularly achieve up to 900Mbit (have to copy a suitably large file to get that, something several gigs in size. video frequently works well here).

The Juniper config bits look like:
set interfaces ge-0/0/40 description "FreeNAS Trunk port"
set interfaces ge-0/0/40 ether-options 802.3ad ae1
set interfaces ge-0/0/41 description "FreeNAS Trunk port"
set interfaces ge-0/0/41 ether-options 802.3ad ae1
set interfaces ae1 description "FreeNAS lagg"
set interfaces ae1 aggregated-ether-options lacp passive
set interfaces ae1 aggregated-ether-options lacp periodic slow
set interfaces ae1 unit 0 family ethernet-switching
set vlans prod interface ae1.0

louisk@switch0.cmhome> show interfaces ae1
Physical interface: ae1, Enabled, Physical link is Up
Interface index: 130, SNMP ifIndex: 616
Description: FreeNAS lagg
Link-level type: Ethernet, MTU: 1514, Speed: 2Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Current address: 08:81:f4:a2:4d:84, Hardware address: 08:81:f4:a2:4d:84
Last flapped : 2013-09-22 20:00:42 PDT (3w1d 16:15 ago)
Input rate : 7664 bps (9 pps)
Output rate : 80576 bps (10 pps)

Logical interface ae1.0 (Index 69) (SNMP ifIndex 619)
Flags: SNMP-Traps 0x0 Encapsulation: ENET2
Statistics Packets pps Bytes bps
Bundle:
Input : 5724 0 343440 0
Output: 1030416 0 110253212 0
Protocol eth-switch
Flags: None

{master:0}
louisk@switch0.cmhome> show lacp interfaces ae1
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/40 Actor No No Yes Yes Yes Yes Slow Passive
ge-0/0/40 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/41 Actor No No Yes Yes Yes Yes Slow Passive
ge-0/0/41 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/40 Current Slow periodic Collecting distributing
ge-0/0/41 Current Slow periodic Collecting distributing

{master:0}
louisk@switch0.cmhome>

 

[mloebl]

My Cisco SG300 has been running great and was configured thru GUI so no IOS settings here unfortunately. I will say it's been pretty stable *and* I figured out another issue where FreeNAS was not at fault (and TP-LINK did an awful job with info on...) Ubuntu 13.x (and maybe 12.x, not sure) has a Network Manager bug so can only configure round robin thru UI, not LACP or other modes. So that was part of my flapping issue I was seeing as well...

 

[KTrain]

How is that Cisco SG300 working out for you? I've been looking around for managed switches for my home lab and the SG200/SG300 series is on my list of possible suitors.

 

[mloebl]

So far so good. I've got two LACP pairs, and a Round Robin pair and haven't had any problems. Two iSCSI targets as well off the FreeNAS box. Seems pretty stable and been running since May of last year. One iSCSI target I had a common network storage, which also had a CCTV DVR constantly writing and no problems. Off the other iSCSI target had a dedicated mirror for my linux box for large file storage.

 

[KTrain]

So far so good. I've got two LACP pairs, and a Round Robin pair and haven't had any problems. Two iSCSI targets as well off the FreeNAS box. Seems pretty stable and been running since May of last year. One iSCSI target I had a common network storage, which also had a CCTV DVR constantly writing and no problems. Off the other iSCSI target had a dedicated mirror for my linux box for large file storage.

Cool! Thanks for the insight.