lagg on 2 x 10GB (ix0,ix1) not working.

Status
Not open for further replies.
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
I have a dual port 10GB with a Cisco SG500X. Without lagg0 everything is working.
when I configure lagg0 (using lacp) I can't access ip of lagg0. Cisco-Switch shows a working lacp and I can even sniff on that device and see broadcasts and multicast pakets. I can also see the arp-requests for that ip but it doesn't answer.

where do i start searching?
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
Code:
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f9
    inet 192.168.0.108 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:a3:c3:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active


this is without lagg
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
this is with configured lagg
Code:
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.108 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ab:65:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=18<COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=18<COLLECTING,DISTRIBUTING>
 
D

dlavigne

Guest
Is the lagg getting its IP by DHCP or is it staticly set?

The interfaces in the lagg should not have IP addresses and atm their IP addresses are in a different network.
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
I can remove the ip-adresses from ix0 and ix1 but it makes no difference.

Code:
ifconfig ix0 -alias 192.168.0.124
ifconfig ix1 -alias 192.168.0.108
 
[root@freenas] /var/log# ifconfig -a
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ab:65:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=18<COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=18<COLLECTING,DISTRIBUTING>


One funny thing: configured 192.168.123.1 on lagg0 and 192.168.123.2 on my computer. When I ping 192.168.123.1 I can see the ARP-Request on the lagg0-Interface

Code:
# tcpdump -nevfi lagg0 arp | grep 192.168.123
16:31:11.538425 ec:1a:59:3d:8b:4e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.123.1 tell 192.168.123.2, length 46
 
S

ser_rhaegar

Patron
Joined
Feb 2, 2014
Messages
358
It seems to me you have a switch configuration issue.

Are there multiple vlans configured on your switch?
Is the LACP interface of the switch ports setup to be a trunk instead of an access port?
Is it intentional that the lagg is on a different subnet?
Also just to reiterate dlavigne's question, is the lagg set for DHCP or a static IP?
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
I tested both: static ip and dhcp. I used another network so i don't trash access to bge0.

switch config is as easy as i can be:

Code:
switchce8705#show running-config
config-file-header
switchce8705
v1.3.7.18 / R750_NIK_1_35_647_358
CLI v1.0
set system queues-mode 4
!
bonjour interface range vlan 1                       
hostname switchce8705
logging host 192.168.222.253
username cisco password encrypted xxx privilege 15
ip ssh server
clock timezone CET 1
clock summer-time web recurring eu
sntp unicast client enable
sntp unicast client poll
sntp server 0.de.pool.ntp.org poll
sntp server de.poolntp.org poll
!
interface tengigabitethernet1/1/1
channel-group 1 mode on
!
interface tengigabitethernet1/1/2
channel-group 1 mode on
!
exit
 
c32767a

c32767a

Patron
Joined
Dec 13, 2012
Messages
371
I would check and make sure both ends are configured to use LACP.

The Cisco switch is set to channel-group mode on, which effectively disables LACP.
It looks like your lagg is configured to use LACP.

If that is the case, things will behave as you have observed. e.g. won't work.. :)

channel-group mode active might produce better results.
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
I tested FreeBSD-9.2 and FreeBSD-10 which doesn't even show a connect (status: inactive). As 10GB are enough we decided to use "Failover" which worked perfectly.
 
tmueko

tmueko

Explorer
Joined
Jun 5, 2012
Messages
82
We tested a little more and it turns out that the twinaxial-cable was the problem: with a normal 850nm multimode cable and transciever everything is working fine!

Code:
[root@freenas] ~# ifconfig -v ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
[root@freenas] ~# ifconfig -v ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
[root@freenas] ~# ifconfig -v lagg0
    lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.60 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "lagg on 2 x 10GB (ix0,ix1) not working."

Similar threads

H
  • Locked
LAGG with LACP, BSD issue?
Replies
3
Views
3K
cyberjock
C
freebsdsa
  • Locked
LAGG with LACP
2
Replies
35
Views
20K
Ericloewe
Ericloewe
D
  • Locked
LAG (LACP) Created and UP but not working
Replies
2
Views
3K
Samuel Catchpole-Radford
S
M
Freenas to VCenter 6.7 iSCSI VLAN/TAG with Nexus
Replies
10
Views
4K
marlonc
M
proto
  • Locked
SOLVED LAGG Members undefined, why?
Replies
3
Views
2K
proto
proto
Top