RegularJoe
Patron
- Joined
- Aug 19, 2013
- Messages
- 330
Hi All,
LACP should not be much of an issue, but it appears to be an issue on FreeNAS 9.3 / FreeBSD 9.3:
Does anyone know what IFCONFIG command I should use to get the cisco SRC-DST-IP on a FreeNAS/FreeBSD lagg interface? I have tried "ifconfig lagg1 laggproto lacp lagghash l2"
From the FreeNAS box I do the ifconfig for LAGG1
ifconfig lagg1
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:15:17:92:83:a9
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l2,l4
laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
=============================================
The Cisco 3560 I am testing with is currently set for "port-channel load-balance src-dst-ip"
This switch only supports these XOR settings:
port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
L2, L3 and L4 hashing is something I could not find much on with Google, I know some much larger switches support a lot more hashing options, i.e. here is the hash options on my 3850:
Switch(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
dst-mixed-ip-port Dst IP Addr and TCP/UDP Port
dst-port Dst TCP/UDP Port
extended Extended Load Balance Methods
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-dst-mixed-ip-port Src XOR Dst IP Addr and TCP/UDP Port
src-dst-port Src XOR Dst TCP/UDP Port
src-ip Src IP Addr
src-mac Src Mac Addr
src-mixed-ip-port Src IP Addr and TCP/UDP Port
src-port Src TCP/UDP Port
Switch(config)#port-channel load-balance extended ?
dst-ip Dest IP
dst-mac Dest MAC
dst-port Dest Port
ipv6-label IPV6 Flow Label
l3-proto L3 Protocol
src-ip Src IP
src-mac Src MAC
src-port Src Port
<cr>
=============================================
As you can see from here VMware is round robin load balancing the packets to VLAN991-994 very well, the physical interfaces are em0-3 and they are not looking very good from a load balancing standpoint.
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |||||
Interface Traffic Peak Total
vlan990 in 0.000 KB/s 1.877 KB/s 60.038 KB
out 0.000 KB/s 1.985 KB/s 62.167 KB
vlan994 in 73.537 KB/s 42.501 MB/s 25.217 GB
out 13.319 MB/s 20.975 MB/s 23.355 GB
vlan993 in 65.365 KB/s 38.893 MB/s 26.081 GB
out 11.708 MB/s 23.106 MB/s 23.548 GB
vlan992 in 71.980 KB/s 45.410 MB/s 25.576 GB
out 12.867 MB/s 19.849 MB/s 23.528 GB
vlan991 in 76.026 KB/s 41.777 MB/s 25.863 GB
out 13.323 MB/s 21.099 MB/s 23.466 GB
lagg1 in 286.911 KB/s 117.013 MB/s 102.736 GB
out 51.217 MB/s 79.533 MB/s 93.896 GB
lagg0 in 0.116 KB/s 2.244 KB/s 3.833 MB
out 1.809 KB/s 8.894 KB/s 12.613 MB
lo0 in 0.000 KB/s 11.223 KB/s 3.697 MB
out 0.000 KB/s 11.223 KB/s 3.697 MB
em3 in 0.000 KB/s 0.120 KB/s 50.018 KB
out 25.027 MB/s 44.006 MB/s 33.380 GB
em2 in 0.000 KB/s 0.120 KB/s 38.422 KB
out 26.190 MB/s 40.267 MB/s 36.277 GB
em1 in 0.000 KB/s 0.120 KB/s 15.738 GB
out 0.000 KB/s 0.496 KB/s 15.880 GB
em0 in 287.032 KB/s 117.013 MB/s 86.998 GB
out 0.000 KB/s 2.482 KB/s 8.360 GB
igb1 in 0.000 KB/s 0.973 KB/s 657.866 KB
out 0.000 KB/s 0.000 KB/s 0.000 KB
igb0 in 0.116 KB/s 2.211 KB/s 3.791 MB
out 1.809 KB/s 8.894 KB/s 12.613 MB
================================================
the cisco switch shows bits 0 and load 00, the load should be more than 00 and the bits as I am doing LACP Src-Dst-Ip and the least significant bit is different on FreeNAS and VMware. Hashing does not look like it is working right.
c3560SAN#sh etherchannel 3 det
Group state = L2
Ports: 4 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0
Ports in the group:
-------------------
Port: Gi0/5
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/5 SA bndl 32768 0x3 0x3 0x106 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/5 SA 32768 0015.1792.83a9 22s 0x0 0x1EB 0x9 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/6
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/6 SA bndl 32768 0x3 0x3 0x107 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/6 SA 32768 0015.1792.83a9 22s 0x0 0x1EB 0x8 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/7
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/7 SA bndl 32768 0x3 0x3 0x108 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/7 SA 32768 0015.1792.83a9 28s 0x0 0x1EB 0xA 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/8
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/8 SA bndl 32768 0x3 0x3 0x109 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/8 SA 32768 0015.1792.83a9 28s 0x0 0x1EB 0xB 0x3D
Age of the port in the current state: 0d:00h:21m:27s
Port-channels in the group:
---------------------------
Port-channel: Po3 (Primary Aggregator)
------------
Age of the Port-channel = 0d:09h:30m:46s
Logical slot/port = 2/3 Number of ports = 4
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi0/5 Active 0
0 00 Gi0/6 Active 0
0 00 Gi0/7 Active 0
0 00 Gi0/8 Active 0
Time since last port bundled: 0d:00h:21m:27s Gi0/7
Time since last port Un-bundled: 0d:00h:21m:29s Gi0/8
LACP should not be much of an issue, but it appears to be an issue on FreeNAS 9.3 / FreeBSD 9.3:
Does anyone know what IFCONFIG command I should use to get the cisco SRC-DST-IP on a FreeNAS/FreeBSD lagg interface? I have tried "ifconfig lagg1 laggproto lacp lagghash l2"
From the FreeNAS box I do the ifconfig for LAGG1
ifconfig lagg1
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:15:17:92:83:a9
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l2,l4
laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
=============================================
The Cisco 3560 I am testing with is currently set for "port-channel load-balance src-dst-ip"
This switch only supports these XOR settings:
port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
L2, L3 and L4 hashing is something I could not find much on with Google, I know some much larger switches support a lot more hashing options, i.e. here is the hash options on my 3850:
Switch(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
dst-mixed-ip-port Dst IP Addr and TCP/UDP Port
dst-port Dst TCP/UDP Port
extended Extended Load Balance Methods
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-dst-mixed-ip-port Src XOR Dst IP Addr and TCP/UDP Port
src-dst-port Src XOR Dst TCP/UDP Port
src-ip Src IP Addr
src-mac Src Mac Addr
src-mixed-ip-port Src IP Addr and TCP/UDP Port
src-port Src TCP/UDP Port
Switch(config)#port-channel load-balance extended ?
dst-ip Dest IP
dst-mac Dest MAC
dst-port Dest Port
ipv6-label IPV6 Flow Label
l3-proto L3 Protocol
src-ip Src IP
src-mac Src MAC
src-port Src Port
<cr>
=============================================
As you can see from here VMware is round robin load balancing the packets to VLAN991-994 very well, the physical interfaces are em0-3 and they are not looking very good from a load balancing standpoint.
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |||||
Interface Traffic Peak Total
vlan990 in 0.000 KB/s 1.877 KB/s 60.038 KB
out 0.000 KB/s 1.985 KB/s 62.167 KB
vlan994 in 73.537 KB/s 42.501 MB/s 25.217 GB
out 13.319 MB/s 20.975 MB/s 23.355 GB
vlan993 in 65.365 KB/s 38.893 MB/s 26.081 GB
out 11.708 MB/s 23.106 MB/s 23.548 GB
vlan992 in 71.980 KB/s 45.410 MB/s 25.576 GB
out 12.867 MB/s 19.849 MB/s 23.528 GB
vlan991 in 76.026 KB/s 41.777 MB/s 25.863 GB
out 13.323 MB/s 21.099 MB/s 23.466 GB
lagg1 in 286.911 KB/s 117.013 MB/s 102.736 GB
out 51.217 MB/s 79.533 MB/s 93.896 GB
lagg0 in 0.116 KB/s 2.244 KB/s 3.833 MB
out 1.809 KB/s 8.894 KB/s 12.613 MB
lo0 in 0.000 KB/s 11.223 KB/s 3.697 MB
out 0.000 KB/s 11.223 KB/s 3.697 MB
em3 in 0.000 KB/s 0.120 KB/s 50.018 KB
out 25.027 MB/s 44.006 MB/s 33.380 GB
em2 in 0.000 KB/s 0.120 KB/s 38.422 KB
out 26.190 MB/s 40.267 MB/s 36.277 GB
em1 in 0.000 KB/s 0.120 KB/s 15.738 GB
out 0.000 KB/s 0.496 KB/s 15.880 GB
em0 in 287.032 KB/s 117.013 MB/s 86.998 GB
out 0.000 KB/s 2.482 KB/s 8.360 GB
igb1 in 0.000 KB/s 0.973 KB/s 657.866 KB
out 0.000 KB/s 0.000 KB/s 0.000 KB
igb0 in 0.116 KB/s 2.211 KB/s 3.791 MB
out 1.809 KB/s 8.894 KB/s 12.613 MB
================================================
the cisco switch shows bits 0 and load 00, the load should be more than 00 and the bits as I am doing LACP Src-Dst-Ip and the least significant bit is different on FreeNAS and VMware. Hashing does not look like it is working right.
c3560SAN#sh etherchannel 3 det
Group state = L2
Ports: 4 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0
Ports in the group:
-------------------
Port: Gi0/5
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/5 SA bndl 32768 0x3 0x3 0x106 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/5 SA 32768 0015.1792.83a9 22s 0x0 0x1EB 0x9 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/6
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/6 SA bndl 32768 0x3 0x3 0x107 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/6 SA 32768 0015.1792.83a9 22s 0x0 0x1EB 0x8 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/7
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/7 SA bndl 32768 0x3 0x3 0x108 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/7 SA 32768 0015.1792.83a9 28s 0x0 0x1EB 0xA 0x3D
Age of the port in the current state: 0d:00h:21m:21s
Port: Gi0/8
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/8 SA bndl 32768 0x3 0x3 0x109 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/8 SA 32768 0015.1792.83a9 28s 0x0 0x1EB 0xB 0x3D
Age of the port in the current state: 0d:00h:21m:27s
Port-channels in the group:
---------------------------
Port-channel: Po3 (Primary Aggregator)
------------
Age of the Port-channel = 0d:09h:30m:46s
Logical slot/port = 2/3 Number of ports = 4
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi0/5 Active 0
0 00 Gi0/6 Active 0
0 00 Gi0/7 Active 0
0 00 Gi0/8 Active 0
Time since last port bundled: 0d:00h:21m:27s Gi0/7
Time since last port Un-bundled: 0d:00h:21m:29s Gi0/8