SOLVED Jail networking broken with VLAN

[Juan Manuel Palacios]

Hi @Patrick M. Hausen,

Just as many others here, I'm trying to solve the problem of putting some of my jails on separate VLANs, but unfortunately without much success, perhaps due to TrueNAS' practice of keeping IP addresses on bridge members, rather than on the bridges themselves.

My NAS only has a single physical interface, igb0, with its own static IP and already bridged to a few jails whose networking stacks have been working "fine" for a while, all of them using that interface's untagged native LAN:

Code:

-> ifconfig igb0
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: igb0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether ac:1f:6b:17:37:ba
    inet 10.0.0.5 netmask 0xffffff00 broadcast 10.0.0.255
(...)

-> ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
(...)
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000

But now I want to move some of those jails to separate VLANs, for which I created an initial testing VLAN interface and a bridge in the TrueNAS UI, and followed the recommendations you've already given multiple times elsewhere, e.g. https://www.truenas.com/community/threads/jail-using-incorrect-vlan.105422/#post-725792, to the letter, but unfortunately to no avail, as VLAN-based networking for a test jail is still not working.

But before I get any deeper into debugging and all the things I've tried (plenty!), not to mention attempting to move igb0's IP to bridge0 while making sure I can still access the TrueNAS web GUI and my existing jails don't break, I first wanted to ask you something I may be misunderstanding about that post of yours.

If I'm understanding correctly, you explain that, for bridges to work appropriately, their member interfaces MUST NOT have IP addresses, any of them, and rather the address should be on the bridge itself, according to FreeBSD documentation (Advanced Networking); and in your screenshots, vlan2 (DMZ), member of the "bridge2" bridge, can be seen without an IP, accordingly. Further, you explain that jails should be bound to these properly-configured-bridges, e.g. "vnet0:bridge2" in your "cloud" jail's network properties.

However, and again if I'm reading correctly, bridge2 does not seem to have an IP in the configuration screenshot that you posted for it. Did you happen to add it later? Remove it for privacy purposes when posting? Or is there something I'm misunderstanding?

Thank you,

Cc @DobleJs @RueGorE This might be of interest to you, as I believe we're trying to solve very similar problems (if you haven't solved it yourselves already, that is, of course).

 

[Patrick M. Hausen]

1. You MUST move the IP address from igb0 to the bridge.
2. If there is a bridge on igb0 you cannot use VLANs on igb0.

Corollary: if you want to use VLANs, you cannot use igb0 untagged, you MUST use all networks tagged. And by 1. you must create a bridge for each VLAN and put the IP address (if the NAS needs one and this is not just for jails) on the bridge interface and not on the VLAN.

However, and again if I'm reading correctly, bridge2 does not seem to have an IP in the configuration screenshot that you posted for it. Did you happen to add it later? Remove it for privacy purposes when posting? Or is there something I'm misunderstanding?

In my network bridge2 connects only jails. So the NAS does not need an IP address in that network.

The rule is: IF there is an IP address for the NAS it MUST be on the bridge and not on any bridge member. That does not mean that there must be an IP address at all.

 

[victort]

Hi @Patrick M. Hausen,

Just as many others here, I'm trying to solve the problem of putting some of my jails on separate VLANs, but unfortunately without much success, perhaps due to TrueNAS' practice of keeping IP addresses on bridge members, rather than on the bridges themselves.

My NAS only has a single physical interface, igb0, with its own static IP and already bridged to a few jails whose networking stacks have been working "fine" for a while, all of them using that interface's untagged native LAN:

Code:

-> ifconfig igb0
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: igb0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether ac:1f:6b:17:37:ba
    inet 10.0.0.5 netmask 0xffffff00 broadcast 10.0.0.255
(...)

-> ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
(...)
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000

But now I want to move some of those jails to separate VLANs, for which I created an initial testing VLAN interface and a bridge in the TrueNAS UI, and followed the recommendations you've already given multiple times elsewhere, e.g. https://www.truenas.com/community/threads/jail-using-incorrect-vlan.105422/#post-725792, to the letter, but unfortunately to no avail, as VLAN-based networking for a test jail is still not working.

But before I get any deeper into debugging and all the things I've tried (plenty!), not to mention attempting to move igb0's IP to bridge0 while making sure I can still access the TrueNAS web GUI and my existing jails don't break, I first wanted to ask you something I may be misunderstanding about that post of yours.

If I'm understanding correctly, you explain that, for bridges to work appropriately, their member interfaces MUST NOT have IP addresses, any of them, and rather the address should be on the bridge itself, according to FreeBSD documentation (Advanced Networking); and in your screenshots, vlan2 (DMZ), member of the "bridge2" bridge, can be seen without an IP, accordingly. Further, you explain that jails should be bound to these properly-configured-bridges, e.g. "vnet0:bridge2" in your "cloud" jail's network properties.

However, and again if I'm reading correctly, bridge2 does not seem to have an IP in the configuration screenshot that you posted for it. Did you happen to add it later? Remove it for privacy purposes when posting? Or is there something I'm misunderstanding?

Thank you,

Cc @DobleJs @RueGorE This might be of interest to you, as I believe we're trying to solve very similar problems (if you haven't solved it yourselves already, that is, of course).

I’ve done this. Basically you need to set your switch to tag all VLANS (this will include LAN as VLAN1) as @Patrick M. Hausen said, then your setup will look like this

1. physical > vlan1 > bridge0 (this is your LAN)
2. physical > vlan10 > bridge10 (this is your VLAN)

IP goes on the bridge

To have a jail on the VLAN, simply set the vnet0:bridge10

 

[Juan Manuel Palacios]

1. You MUST move the IP address from igb0 to the bridge.
2. If there is a bridge on igb0 you cannot use VLANs on igb0.

Corollary: if you want to use VLANs, you cannot use igb0 untagged, you MUST use all networks tagged. And by 1. you must create a bridge for each VLAN and put the IP address (if the NAS needs one and this is not just for jails) on the bridge interface and not on the VLAN.


In my network bridge2 connects only jails. So the NAS does not need an IP address in that network.

The rule is: IF there is an IP address for the NAS it MUST be on the bridge and not on any bridge member. That does not mean that there must be an IP address at all.

OK, understood.

However, if I may ask, why is it that, if there's already a bridge on igb0, using an untagged network, there cannot be any VLANs on igb0? I'm not arguing it should work, it clearly isn't in my so many attempts (perhaps because that setup violates some basic FreeBSD networking principle that I'm unaware of), but I'm just wondering why, out of confusion, after the sheer number of things I've tried so far.

For example, without ever interrupting the igb0 untagged network, nor stopping even a single of the so many jails on top of it, nor disrupting network traffic to them, when I create a VLAN on top of igb0 and set it to DHCP, it (the VLAN interface) acquires an IP address instantly, so at least some traffic does flow to it. However, if I remove DHCP from that VLAN, add it to a new test bridge, and set the latter to DHCP, then that bridge never ever acquires an IP (and, of course, networking for test jails on top of that test bridge is completely broken).

Thank you,

 

[Patrick M. Hausen]

According to FreeBSD docs and conversation with developers the bridge does not pass tagged frames so you cannot put a VLAN on top of a bridge. Putting an IP address or a VLAN on a bridge member produces a scope violation and breaks multicast.

And last you cannot have more than one interface with DHCP enabled, anyway.

 

[Juan Manuel Palacios]

According to FreeBSD docs and conversation with developers the bridge does not pass tagged frames so you cannot put a VLAN on top of a bridge. Putting an IP address or a VLAN on a bridge member produces a scope violation and breaks multicast.

And last you cannot have more than one interface with DHCP enabled, anyway.

The VLANs I've tried were not on top of the bridge, strictly speaking, but actually directly on top of igb0:

Code:

-> ifconfig vlan100
vlan100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: VLAN2_100
    ether ac:1f:6b:17:37:ba
    groups: vlan
    vlan: 100 vlanproto: 802.1q vlanpcp: 0 parent interface: igb0
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>

Still, however, tagged traffic from the VLAN onto the test bridge, and from there onto the test jail on top of said bridge, was clearly not being passed, no matter what I did (static IP, DHCP, dynamic routing, static routing, etc.).

On the other hand, I just read about having multiple interfaces with DHCP enabled for them, e.g. https://forums.freebsd.org/threads/dhcp-client-on-several-interfaces.15341/, and it's definitely just not worth it, so much easier to assign static IPs and setup reservations on my router to prevent conflicts.

 

[Juan Manuel Palacios]

I’ve done this. Basically you need to set your switch to tag all VLANS (this will include LAN as VLAN1) as @Patrick M. Hausen said, then your setup will look like this

1. physical > vlan1 > bridge0 (this is your LAN)
2. physical > vlan10 > bridge10 (this is your VLAN)

IP goes on the bridge

To have a jail on the VLAN, simply set the vnet0:bridge10

Simple, clean, well organized, just how I like it!

This serves me not just to chart a migration to such a setup, but also, finally, to sleep well at night not continuing to wonder why on earth my attempted setup was not working! ;) (it's not like I'm a complete networking neophyte, so the lack of success was severely disrupting my inner peace! ;)

 

[Juan Manuel Palacios]

Hi everyone!

I was about to come here yesterday with a desperate cry for help, 'cause nothing I was doing on my TrueNAS rig was getting my jails to pick up my VLAN tags, after spending a considerable amount of time reconfiguring my pfSense router to route and filter for them, until I decided to reread this thread and recheck everything, and fortunately my eye caught this one tiny bit of advice from @Patrick M. Hausen:

2. If there is a bridge on igb0 you cannot use VLANs on igb0.

And, indeed, I had all my VLAN interfaces created, attached to corresponding bridges, and jails assigned to those bridges as desired… but I still had igb0 added to the TrueNAS-created bridge0 interface, with my UniFi Controller jail running on top of it, as I needed to have it running to configure VLANs on my UniFi switch.

As soon as I realized this, and stopped the UniFi Controller jail, and restarted my test jail, traffic immediately started flowing to it through its bridge, and it picked up its assigned VLAN tag! And then I started a second test jail, added to a different VLAN:bridge pair, configured with a different VLAN tag, and it picked up the different and correct VLAN in turn!

So, thank you so much @Patrick M. Hausen and @victort for all the help, your advices were lifesavers!

For anyone reading this who may want to do something similar, and is struggling to get it working, I guess I could summarize the key takeaways as follows:

1. FreeBSD bridges function as virtual switches, so any kind of interface you may want to add to them, whether hardware or virtual, can only be added to a single bridge at a time, which makes perfect sense (see below).
2. You can get VLAN interfaces working on top of hardware interfaces that are also up-and-running themselves, but if the VLAN interface needs to go on a bridge, then the underlying hardware interface cannot be on a bridge itself, as a direct result of the point above; you'd be trying to add the hardware interface to two bridges/switches at once, even if indirectly/transitively, so traffic will not flow through the second one. Think of it as bridges sort of claiming exclusive locks on the interfaces added to them, and/or how much sense it'd make to try to connect a physical device to more than one physical switch at a time (without pulling weird, black magic tricks, of course).
3. Putting those two together, if you have a collection of jails and you want to distribute them across different VLANs, then the layout you have to follow is rather simple, mnemonic, and organized:
   1. One VLAN interface per each VLAN tag, on top of whatever hardware interface(s) you may have available (multiple VLAN interfaces can go on top of a single hardware interface --e.g. right now I have 5 VLANs on top of a single igb NIC--, obviously being limited by available bandwidth and desired throughput). For example, "vlan10" for VLAN tag Id 10.
   2. One bridge interface per each VLAN interface, e.g. "bridge10" for "vlan10", with the VLAN interface manually added to its corresponding bridge (with RSTP protocol enabled, preferably).
   3. Jails that you want to put on a specific VLAN, e.g. VLAN 10, should have their "interfaces" attribute in the jail's "Network Properties" set to "vnet0:bridge10", so that they get automatically added to the correct bridge upon starting up. You also have to decide whether you want to enable vnet, static IP assignment, DHCP, how to configure the jail's resolver(s), etc., but that goes beyond the scope of assigning a jail to a specific VLAN.
   4. Make sure no jail is left assigned to the "vnet:bridge0" interface, if bridge0 is not a bridge that you created manually and configured per this recipe, as that'll trick TrueNAS into creating the broken setup that @Patrick M. Hausen keeps warning everyone about, which potentially includes the attempt to add an interface that's already on a bridge to a second one (thus breaking the exclusivity contract I've been describing).
   5. For any given VLAN, if you also want to enable networking for your TrueNAS host on it, and not just for the collection of jails attached to the corresponding bridge, then the TrueNAS' IP for that VLAN has to go on the bridge, not the VLAN interface (needless to say, being careful to not lose connectivity to your NAS if you happen to need to move an IP from one interface to another one, which will always be highly dependent on your network topology --e.g. how many NICs do you have?-- and other tools that might help --e.g. IPMI, serial console, etc.--). Conversely, if for a given VLAN you do not require networking for your TrueNAS host on it, then no need to even assign an IP to the corresponding bridge (nor, needless to say, to the counterpart VLAN interface, of course).

Following that mnemonic recipe, coupled with appropriate switch & router VLAN configuration, DNS resolution, routing and filtering, etc., you should be able to easily distribute any number of jails on a single host across any number of VLANs, plus networking for your TrueNAS host.

Enjoy!