SOLVED Jails and lagg

[belisarius]

I've seen occasional reference in the forums to the fact that lagg and jails might not work together. Based on what I've been experiencing, that's entirely possible, but I was hoping to get a more definitive "That won't work" or even better, "Here's how to make it work".

I've got two Intel 82576 integrated NIC's on the motherboard hooked up to my switch as LACP. The switch supports it, and it's configured, and works perfectly fine so long as I don't have any jails. As soon as I add jails, things start to get odd.

I've setup the jails and FreeNAS to operate on DHCP. Both the host and the jails are a 50/50 proposition to get an address - I'm running dnsmasq on an independent machine and I see the offers going out 100%, but they're just never received by FreeNAS or the jails. Now, instead of not having an address, I just don't have network connectivity on either FreeNAS or the jails. I bounce the netif service, and everything works on FreeNAS. Ditto for the jails.

So things sorta work. But every time I reboot FreeNAS (which, admittedly, won't be very often), I need to restart netif, which is a bit annoying, and probably indicates I'm not doing something right.

Here's my ifconfig output:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:25:90:2a:fe:aa
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:25:90:2a:fe:aa
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8800<SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:25:90:2a:fe:aa
        inet 192.168.0.22 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:77:1b:fd:3d:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 16 priority 128 path cost 2000
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 2000
        member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000
        member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 20000
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:f7:3a:00:12:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:12:ba:00:0f:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:84:a9:00:10:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:82:01:00:11:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

 

[dlavigne]

Did you make any progress with this?

 

[belisarius]

Nope - at this point I'm considering re-configuring to create a storage network and ditching LACP, but I might just live with it and restart the networking with each reboot, as that hopefully won't happen terribly often.

 

[b0bje]

I got a 4 port NC364T connected to a 1810-24g configured as a lagg (previously it was a simple em0 and re0 lacp)

My jails are configured with a static ip, VMIMAGE is on for all jails and I don't experience any of your described problems.

Drop a note if you want me to provide any config files or screen output..

 

[belisarius]

I've got the Intel version of that card in my VM server - maybe I'll try dropping that in the FreeNAS box and see what happens. It might be that it's some weird incompatibility between FreeNAS and those integrated Intel NICs . . .

 

[belisarius]

I eventually installed the same NIC as b0bje, the NC364T, but still no joy. I have to manually restart netif after boot on both the host FreeNAS and the individual jails. C'est la vie - I shouldn't need to reboot very often.

 

[zuzuzzzip]

Seems like I am having the same issue as OP ...
Any news/workarounds for this issue? Is there a bug report I can follow?

 

[dlavigne]

No open bugs for 9.x as no further work is being done on 9.x jails. It's too early to test to see if the rewritten 10 jail system fixes this.

 

[belisarius]

Yep, I'm still having the issue myself. Right now I'm resigned to restarting networking/jails on a reboot (which doesn't happen very often).

 

[zuzuzzzip]

For me, this workaround is still working.

Basicly disable VIMAGE on the jail and select a NIC.
For me it is the VLAN interface on a lagg (see screenshot).

 

[belisarius]

Ok, I've made two changes recently, and now LAGG works. Not sure which did it. I upgraded to FreeNAS 9.10, and I swapped out my switch. The old switch was a no-name Zyxel that claimed LACP support. I replaced it with an HP Procurve I snagged off eBay.

Now, all my networking just works when it comes back up after a reboot, jails and all.