Unable to get past bridge when using vlan in iocage jail

[WilbertNL]

Hello all,

I'm sorry to post yet another thread with regards to VLAN's. I thought I had it all worked out when I got it working in my 11.3 BETA1 environment, but I can't get it to work in my 11.2U6 setup.

So I have a iocage jail called test2, using vnet. It's configured for VLAN ID 178. I am able to ping devices connected to the bridge, like the VLAN178 interface IP, but I can't seem to get beyond the bridge. Although.. test2 does get an IP address for my gateway using DHCP, eventhough I can't ping the router after getting an IP. From the Freenas self, I'm able to ping both the test2 and the gateway on vlan 178.

The test2 jail is fresh out of the box, so no firewall configured. 'allow_raw_sockets' is checked.

To clarify (hopefully) what does and does not work:

Ping from freenas:
gateway 192.168.178.1 <<< works <<< freenas 192.168.178.42 >>> works >>> test2 192.168.178.101 (DHCP)

Ping from test2:
gateway 192.168.178.1 <<< fails <<< test2 192.168.178.101 (DHCP) >>> works >>> freenas 192.168.178.42

Set-up connectivity:
test2 (epair0b/vnet0.10) >>> bridge178 >>> vlan178 (bridge member) >>> re0 (parent for vlan178)

ifconfig freenas:

Code:

...
vlan178: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80001<RXCSUM,LINKSTATE>
        ether 70:85:c2:01:ea:c4
        inet 192.168.178.42 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 178 vlanpcp: 0 parent interface: re0
        groups: vlan
...
bridge178: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:3e:03:85:ac:b2
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 2000
        member: vlan178 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 20000
...
vnet0:10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: test2 as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:4b:48:3c
        hwaddr 02:22:d0:00:0f:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
...

ifconfig test2:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:60:4b:48:3d
    hwaddr 02:22:d0:00:10:0b
    inet 192.168.178.101 netmask 0xffffff00 broadcast 192.168.178.255
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

Since, to me at least, this seems weird, I tried rebooting the freenas on various occasions, but to no avail.

As far as I can see, the vlan itself is kinda working but as mentioned: from the jail itself, I can't seem to get past the bridge with neither PING, nor HOST, etc.

Thanks for reading my issue, hope you can help!

 

[dlavigne]

Bridging is a new feature in 11.3 which probably explains the difference.

 

[WilbertNL]

Alright, thanks. Does that mean that what I'm trying to achieve isn't possible? And if it is, what do I need to do?

Still, I do have some sort of vlan connectivity, I just can't break out of the host/bridge.

 

[WilbertNL]

Could someone please help me? I'm kinda stuck and I would really like to continue.

Basically what I'm trying to achieve here is to get my iocage jail test2 to connect to VLAN 178. If this is accomplished, hence the test2, I need to setup an VPN gateway, so I'll be needing VNET to allow TUN by my understanding. I only have one nic on my freenas, re0.

Either I'm missing some very basic understanding with regards to configuring VLAN's on freenas or it's just not possible... I have to assume the former. My skills in networking and VLAN's however are quite advanced and I know all the tagging-stuff in my infrastructure is configured properly, as can be concluded from my initial information where the freenas itself can ping the gateway and other devices on that vlan. If only my jail would be so compliant :)

Any help would be greatly appreciated! If you need any additional info, I'll be happy to provide it.

 

[alexr]

Sounds a lot like what I've been stuck on for almost a year: https://www.ixsystems.com/community/threads/iocage-jails-in-vlans.72287 https://jira.ixsystems.com/browse/NAS-100773

 

[WilbertNL]

Hi Alexr,
Thanks for pitching in. In my case the iocage actually does get an IP address from the DHCP server and the host is also able to ping the jail... and vice versa.

The host is also able to ping the router and another iocage jail running on a 11.3Beta host, both on the same vlan 178.

I tried setting the default_interface to vlan178, but to no avail. I do still get an IP address from DHCP, but the jail is unable to ping (or otherwise connect to) anything outside of the host.

 

[WilbertNL]

Just did another test: if the VLAN 178 IP address from the host (192.168.178.42) is removed, the jail doesn't get an IP address. Configuring the jail to use a static IP still leads to no connectivity over the VLAN. Obviously I can no longer ping the host from the jail, since the host no longer has an IP on that subnet.

For full disclosure: the host is also no longer able to ping anything on the VLAN 178 subnet, so we can rule out that some unexpected routing is/was going on via other subnets.

 

[WilbertNL]

I'm sorry, I'm having a hard time understanding why it's so hard to either get a definitive answer or some help with this issue. I've searched through a lot of posts on the internet from people struggling with this issue. I appreciate that it works in 11.3 BETA, but since I can't find a roadmap with regards to a release date of the stable, 11.2U7 it is for now.

... FreeNAS is used as a development platform where new things are created and tested before they are moved up to the commercial product. They develop everything in FreeNAS to be compatible with TrueNAS.
Finally, FreeNAS is a gift, please don't criticize it too much.

(Chris Moore link)

I love FreeNAS, have been using it for quite a few years, and I mean no disrespect to any of the developers or contributors, but I can't imagine this functionality not being a must-have in the commercial product.

So my guess is that I, and quite a few others, must be missing something basic here. Can anyone please tell me what that is?

Thanks!

 

[WilbertNL]

Some posts mention that the main interface (re0) needs to be a member of the bridge the VLAN is connected to. Also that the vnet_default_interface should be the VLAN interface. So I added the VLAN178 interface to bridge0 (where re0 resides) and set vnet_default_interface for the iocage jail to vlan178. But alas, the jail now gets an IP address from the untagged vlan.

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:3e:03:85:ac:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 2000
        member: vlan178 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: epair6a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000
        member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000

 

[alexr]

Just did another test: if the VLAN 178 IP address from the host (192.168.178.42) is removed, the jail doesn't get an IP address. Configuring the jail to use a static IP still leads to no connectivity over the VLAN. Obviously I can no longer ping the host from the jail, since the host no longer has an IP on that subnet.

Sounds like you were able to replicate my problem. I wish I had a clue as to why it's not working.

 

[WilbertNL]

Haha, yeah, same here.... I wish an expert would comment on this post, or someone else who got it working.

 

[sretalla]

What do you see from route -4 show 8.8.8.8 in the jail?

Have you defined a gateway in the GUI? (only for one of the network interfaces?)

When you set the jail to use DHCP, I think you're saying you don't get an address... do you get a gateway? or do you specify one when you assign a static IP?

 

[WilbertNL]

Hello sretalla, thanks for taking an interest!

Code:

root@test2:~ # route -4 show 8.8.8.8
   route to: 8.8.8.8
destination: default
       mask: default
    gateway: 192.168.178.1
        fib: 0
  interface: epair0b
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 

I have not defined a gateway in the GUI explicitly, but this is wat netstat shows:

Code:

root@test2:~ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.178.1      UGS     epair0b
127.0.0.1          link#1             UH          lo0
192.168.178.0/24   link#2             U       epair0b
192.168.178.101    link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

When you set the jail to use DHCP, I think you're saying you don't get an address... do you get a gateway? or do you specify one when you assign a static IP?

Not quite, I actually do get an IP Address from the DHCP server. But that seems to be the only form of connectivity I have beyond the host. I'm not quite sure if this is a result from a "trick" that VNET does, as in some sort of VNET DHCP relay to the jail or something.

Please see my first post on what connectivity I manage to get.

Let me know what else I can check for you!

 

[Newfoundland.Republic]

I think that we all have the same issue: VLANs are broke in 11.2x... My thread (under jails) is https://www.ixsystems.com/community/threads/help-with-jails-and-vlans.80657/.

Seems like VLANs and jails are broken until 11.3 :(

 

[WilbertNL]

Seems like VLANs and jails are broken until 11.3 :(

Thanks for your input! It would be awesome if someone/a developer could actually confirm VLAN doesn't work for jails in 11.2. Either that or some confirmation that someone got it working :) I myself can confirm that it works in 11.3 BETA.