VLANs Bridges and LAG Interface best practice questions

NinthWave

Contributor
Joined
Jan 9, 2021
Messages
129
[...]

The reason not to use the "native VLAN" is that first pf can get confused (rules on the untagged interface "catch" tagged traffic) and second, as soon as you configure a bridge on the physical interface, you cannot use VLANs, anymore.

physical --> LAGG --> VLAN --> bridge

[...]

What do you mean you mean you cannot use VLANs anymore ?

Is it that let's say today I create VLAN10, VLAN20 and VLAN40
Then BRIDGE10, BRIDGE20 and BRIDGE40 that I attach to bce0

If a later time I realise I need one more VLAN on my network, I an toast because since bridges are already connected to bce0, I won't be able to add them ?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
What do you mean you mean you cannot use VLANs anymore ?

Is it that let's say today I create VLAN10, VLAN20 and VLAN40
Then BRIDGE10, BRIDGE20 and BRIDGE40 that I attach to bce0

If a later time I realise I need one more VLAN on my network, I an toast because since bridges are already connected to bce0, I won't be able to add them ?
You cannot use a bridge that contains the untagged bce0 as a member. That's why I advise to use only tagged VLANs on a trunk port/interface.
 

NinthWave

Contributor
Joined
Jan 9, 2021
Messages
129
You cannot use a bridge that contains the untagged bce0 as a member. That's why I advise to use only tagged VLANs on a trunk port/interface.
Ok, so that's why if VLANs are ever to be used in TrueNAS, one should have 2 physical interfaces:
  • one for untagged traffic, lets say root system
  • one for tagged trafic, lets say all jails and VMs
I am sorry I make you repeat yourself but english is not my primary tongue and I'd like not to make the same mistakes twice.

I do have 2 NICs but when I first installed TrueNAS I only had one LAN network so used solely bce0. But I am trying to evolve my network and many things evolve around my T610 server (TrueNAS)

Yesterday, I created a "devjail" in a VLAN and then ally my jails would not start giving me "vnet error".
As a consequence of that, I deleted all my BRIDGEs and VLANS and devjail and now it seems ok.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
You don't need two interfaces. If you have only one just run all VLANs tagged - what's the problem with not having any untagged interface? I have all my management interfaces in VLAN 3 across my home network ...

I have:
  • lagg0 - link aggregation built from physical igb0 and igb1
  • vlan1 on top of lagg0 - tagged
  • vlan2 on top of lagg0 - tagged
  • vlan3 on top of lagg0 - tagged
  • vlan 9 on top of lagg0 - tagged
  • bridge2 on top of vlan2 - because that's where I connect jails and VMs
No untagged interface anywhere - why would you think that is necessary?

My OPNsense runs the same architecture but without the bridge - just lagg and vlans. And both devices plug into a Cisco 2960-L, two ports each, with LACP.

On the Cisco I have:
Code:
LAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/10, Gi0/11, Gi0/12, Gi0/13, Gi0/14, Gi0/18
2    Server                           active
3    Management                       active    Gi0/9
9    VPN                              active

interface Port-channel2
 description TrueNAS CORE
 switchport trunk native vlan 1001
 switchport mode trunk
!
interface GigabitEthernet0/7
 description TrueNAS CORE
 switchport trunk native vlan 1001
 switchport mode trunk
 channel-group 2 mode active
!
interface GigabitEthernet0/8
 description TrueNAS CORE
 switchport trunk native vlan 1001
 switchport mode trunk
 channel-group 2 mode active


And as I said the only limitation is with a bridge containing the untagged interface. If you insist on mixing tagged and untagged you can only use tagged (vlanX) interfaces for jails. And you must pre-create the bridge interfaces statically for everything to work. And you cannot put a bridge on the untagged and hence you cannot connect jails to the untagged. That's all there is to it. (I'm repeating myself). But why don't you run all networks tagged and be done with it?

Possibly you should re-read this post of mine explaining everything in detail:
 
Last edited:

NinthWave

Contributor
Joined
Jan 9, 2021
Messages
129
No untagged interface anywhere - why would you think that is necessary?
But why don't you run all networks tagged and be done with it?
That's because I started with a single LAN network. Everything was there. When I started to segment/harden my network, I added new tagged vlans but for lack of network understanding, I just left the "safe" stuff (call it management) in the untagged vlan1; by default in any cheap smart switch (tplink).

While I read in some tutorials featuring netgear products in wich the author created a management vlan and "deleted" the default vlan1, it says in tp-link switch doc that vlan1 can not be removed. It does not mean that one cannot tag everything but from that; that's what I did. I tagged the new subnets but left the safe hosts in vlan1, untagged. I'll correct that.
Possibly you should re-read this post of mine explaining everything in detail:
Yes, I'll read again carefully the entire thread.

Thanks for your support.
 

NinthWave

Contributor
Joined
Jan 9, 2021
Messages
129
And so due to my experience and the mentioned limitation that you cannot run VLANs and untagged on a physical IF if that IF will simultaneously be a bridge member, I recommend to people here to run either all untagged (an "access port") or all tagged (a "trunk port") in TrueNAS, too. Most server mainboards come with 2-4 interfaces, anyway, so you can get your untagged access to the TrueNAS control plane and have a separate interface for all your VM VLANs to your heart's content.
While configuring my switch on which the TrueNAS server is connected, I forgot to tag the trunk port in one of the vlans. About 5 seconds after I connected the cable in that port, all the LEDs started flashing at once and as you said somewhere, my whole network was down, even upstream on the main switch.

Could you explain why such a massive effect is possible ? I am curious to understand that.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Whenever there is a loop in a layer 2 broadcast topology like Ethernet, packets are multiplied exponentially. That's because a packet sent to the broadcast address like an ARP request entering one port is supposed to be broadcast (ha!) out all other ports of the switching infrastructure.

So one packet in - N packets out. But due to the loop all these packets will come back one more time. Only this time it's N packets in - N^2 packets out. And so on and so forth. Loops bring down any layer 2 infrastructure that uses broadcasts.

That's why there is the STP protocol, which is supposed to discover loops and disable individual links until there are no more loops. Only that for the FreeBSD bridge the default setting for the STP protocol is "off".

That's why you can connect most standard switches in loops all you like - they will figure it out. But be careful when connecting hosts doing the bridging in software.
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
A precision about STP : Indeed, it will detect and disable the loop as explained by @Patrick M. Hausen, but it does not guarantee you that the end result will be the most optimal path. Just setting STP On will avoid catastrophe but may create bottlenecks.

As such, you still need to be careful. If you create loops on purpose for redundancy, then you must configure STP to optimize it and not just leave it On by default. If you do not intend to create any loop, always check to see if STP ended up disabling some path. If it does, you will now know that you have a mistake somewhere that need to be fixed.
 

NinthWave

Contributor
Joined
Jan 9, 2021
Messages
129
Just setting STP On will avoid catastrophe but may create bottlenecks.
Thanks to you and @Patrick M. Hausen.

When I decided to go the vlan way on TrueNAS, I bought a second hand smart switch. It worked for my cameras and WAP but as soon as I added TrueNAS in the equation, it crashed the whole network as Patrick has stated. Of course, I suspected the second hand switch to be at fault and reverted to the dumb switch until I figured what was happening.

This is a very tricky thing. Unless I did not look thoroughly enough, I did not find warnings until I found this thread.

I guess FreeBSD wizards won't fall in to this pitfall but TrueNAS Core has a more neophyte fan base and as such, I believe that more warnings from IXSystems would be welcome. Unless of course I did not look thoroughly enough.
 
Last edited:

beaster

Dabbler
Joined
May 17, 2021
Messages
27
per @Patrick M. Hausen
It's general good practice not to use data traffic on the un-tagged VLAN unless you are only using a single VLAN/broadcast domain on your switch.

The moment you start using VLAN's you should carve off one or more VLAN for use with STP (Native VLANs). These untagged VLANs are used to communicate the STP paths.

The Best practice here is to not carry data traffic on those VLANs
You may be forced also to use separate native VLANs for different connections.

Most network equipment will default to using VLAN 1 as the Native / Untagged VLAN.
Some cheap network equipment will not let you use anything other than VLAN 1.

About the only issue I've had with Truenas to date in regards to IGMP support under this setup.
I've had partial issues with multicast over VLANs/Bridges, however I've not had time to determine the root cause.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
I've had partial issues with multicast over VLANs/Bridges, however I've not had time to determine the root cause.
Having an IP address on a bridge member interface instead of the bridge interface breaks multicast as I keep repeating.
 

beaster

Dabbler
Joined
May 17, 2021
Messages
27
Maybe worth noting to anyone who is curious, I scrubbed my TRUENAS CORE setup and installed a fresh installation of TRUENAS SCALE. The VLAN trunking setup works just as it does with CORE. The Debian based OS operates just as the BSD based one did. The Web UI is not currently as mature as the one on the CORE platform at the time that I am writing this. The current setup is cumbersome, but you can get this working with some persistence. If you are setting up 5+ VLANs, then using the True Command CLI makes this far quicker and I would recommended it.

I actually had some benefits with moving to SCALE, my new Intel 2x10Gbps NIC was having issues with 13.0, I switched to SCALE and the new drivers seem to have resolved the issue so far.
 
Last edited:
Top