Lagg w. Failover and VLANS failing

[Jason Brooks]

Hello,

I have been playing with the ability to have redundant interfaces, but ran into a problem.

1. build link aggregation (failover mode) of em0 and em1 -> LAGG0
2. build vlans(100,999) using parent interface of LAGG0
3. assign ip addresses to vlan interfaces

When the first vlan is setup with an ip address, pinging and failover work. However when I add the second ip address to the second vlan, the first vlan stops working, but the second vlan does still work. This is regardless of which network connection is active. It's as if setting up a new vlan ip address clobbers the previous ones.

The output of ifconfig shows LAGG0 having em0 and em1 as members. when I add the second vlan (vlan999) and it's ip address, ifconfig's output for vlan100 changes: vlan: 100 goes to vlan:0 and Parent interface goes from lagg0 to <none>.

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:31
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 00:21:5a:af:77:00
        inet 192.168.1.28 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 00:15:17:cb:8a:31
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

vlan100: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:00:00:00:00:00
        inet 10.37.50.29 netmask 0xffffff00 broadcast 10.37.50.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        vlan: 0 parent interface: <none>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:31
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: bce1 flags=0<>
        laggport: em1 flags=5<MASTER,ACTIVE>
vlan999: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:31
        inet 192.168.3.29 netmask 0xffffff00 broadcast 192.168.3.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 999 parent interface: lagg0

This also happens if I use interfaces em1 and bce1.

So I ran some tests to see if I was configuring link aggregation and vlans correctly:

First I tried just setting up two ethernet interfaces with link aggregation (failover mode) with success.

• setup lagg0 with em0 and em1 as members
• tested failover worked...

Then tore down the link aggregation to test vlans:
Setup em0

• added vlan100 (interface ip: 10.37.50.29/24)
• added vlan999 (interface ip: 192.168.3.29/24)
• pinging ip addresses on the lans works.

System specific information:
FreeBSD 9.3-RELEASE-p28 amd64
running on an HP Proliant DL360G5.

Is this a bug, or am I doing it wrong?

 

[jgreco]

You're doing it wrong; do not mix interface types. Either use the em's OR the bce's, don't mix.

There looks like another issue here but I'm not interested in trying to troubleshoot with a broken configuration. I'm one of only a few people here who's worked with layered network configs like this, and I'll tell you that getting the basics right helps.

 

[Jason Brooks]

You're doing it wrong; do not mix interface types. Either use the em's OR the bce's, don't mix.

There looks like another issue here but I'm not interested in trying to troubleshoot with a broken configuration. I'm one of only a few people here who's worked with layered network configs like this, and I'll tell you that getting the basics right helps.

Hello,

Actually, that's the first way it failed: I used em0 and em1 and got exactly the same error. So I will start again with em0 and em1 and show you the output of ifconfig for each step. For entries that don't change such as em0 and em1, I will leave them out of subsequent entries.

First step: nothing setup: em0 and em1 are plugged into the switch. bce0 is my network connection on default vlan0

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:31
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 00:21:5a:af:77:00
        inet 192.168.1.28 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 00:21:5a:af:67:f0
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (none)
        status: no carrier

Second Step: lagg0 configured with em0 and em1

• em1 difference: mac address altered to match em0 and

Code:

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: em1 flags=0<>
        laggport: em0 flags=5<MASTER,ACTIVE>

Third Step: addition of vlan100 and vlan999

Code:

vlan100: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:00:00:00:00:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        vlan: 0 parent interface: <none>
vlan999: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 999 parent interface: lagg0

Note how vlan 100 no longer appears with parent interface of lagg0. (It did start configured as it was the first entry I made.) BUT here is a screen shot of the configured vlans:

And now, I go and configure vlan100 and the relevent lines become:

Code:

vlan999: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:00:00:00:00:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        vlan: 0 parent interface: <none>
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:30
        inet 10.37.50.28 netmask 0xffffff00 broadcast 10.37.50.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 100 parent interface: lagg0

Note how vlan999 and vlan100 switched places. The vlan screen did not change but here's a screen shot of the configured interfaces:


So if my configuration is wrong, I would be happy to fix it but it's not at all clear in the documentation how to do this, and it appears something is wrong with how the gui is configuring freebsd.

--jason

 

[cyberjock]

Can you attach a debug file? System -> Advanced -> Save debug?

 

[jgreco]

Yeah, I'd say the middleware is hamfisting the vlan configuration. To be fair, configuring vlans is kind of finicky so I'm guessing that whoever twiddled those lines of code probably never actually tried it

So what happens if you do "ifconfig vlan999 vlan 999 vlandev lagg0"?

 

[Jason Brooks]

Can you attach a debug file? System -> Advanced -> Save debug?

 

[Jason Brooks]

Yeah, I'd say the middleware is hamfisting the vlan configuration. To be fair, configuring vlans is kind of finicky so I'm guessing that whoever twiddled those lines of code probably never actually tried it :)

So what happens if you do "ifconfig vlan999 vlan 999 vlandev lagg0"?

This change:

Code:

vlan999: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 999 parent interface: lagg0

So I went ahead and configured vlan999 in the gui and the vlan100 configuration was stepped on. I modified your idea: I entered and got the following: (both vlans and lagg0 shown):

Code:

ifconfig vlan100 vlan 100 vlandev lagg0


vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:30
        inet 10.37.50.28 netmask 0xffffff00 broadcast 10.37.50.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 100 parent interface: lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:cb:8a:30
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: em1 flags=0<>
        laggport: em0 flags=5<MASTER,ACTIVE>
vlan999: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:15:17:cb:8a:30
        inet 192.168.3.28 netmask 0xffffff00 broadcast 192.168.3.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 999 parent interface: lagg0

And both vlans work!

--jason

 

[Jason Brooks]

And both vlans work!

--jason

So, is there a good place to put a post-boot hack? to send that command?

Thanks!

 

[jgreco]

You can do that, but what's going to happen is that any change to networking which causes the system to "reconfigure" is likely to stomp it. There's no post-network-reconf script hook.

 

[Jason Brooks]

Hello,

Do you need me to file a bug report on this?

--jason

 

[jgreco]

I'd say that's probably a good idea. Include a link to the thread, and post the issue number here, please.

 

[Jason Brooks]

Hello,

I found a different thread and bug report and added to them.

https://forums.freenas.org/index.ph...hen-adding-other-new-network-interfaces.39574
https://bugs.freenas.org/issues/12571

 

[Jason Brooks]

I should add though that rebooting the system after configuration brought the lagg0 and vlan interfaces up and running correctly.

 

[jgreco]

Well, that's better, but I think the network reconfiguration hook has a tendency to deconfigure-and-reconfigure things (possibly unnecessarily) when changes are made to networking, and of course it shouldn't go dead or deaf.

 

[Jason Brooks]

I agree. Thanks for your patience AND your time!

--jason

 

[jgreco]

Not a problem, with enough patience we usually get people sorted out.