Can someone tell me what I am doing wrong regarding Bridges

[PackElend]

. Also the IP address (if present) goes on the bridge interface, not the physical or the VLAN.

can you tell why as I understand:

1. Bridge: L2, switching frames between interfaces connected to bridge, if it has to leave the bridge it goes to VLAN interface
2. VLAN interface:
   1. L2 + adding/removing 802.1Q-Tag
   2. transition to L3 (if required, e.g. calling GUI on IP within this VLAN), so reachable by IP.
      At least, that is how is set on my MikroTik Router (VLAN Interface has IP and NETWORK configured)

 

[Patrick M. Hausen]

can you tell why as I understand:

1. Bridge: L2, switching frames between interfaces connected to bridge, if it has to leave the bridge it goes to VLAN interface
2. VLAN interface:
   1. L2 + adding/removing 802.1Q-Tag
   2. transition to L3 (if required, e.g. calling GUI on IP within this VLAN), so reachable by IP.
      At least, that is how is set on my MikroTik Router (VLAN Interface has IP and NETWORK configured)

If you bridge a VLAN interface it becomes a strictly layer 2 interface. The layer 3 function moves up to the parent, because all bridge members share the same layer 3 ... If there is no bridge in place, you are correct. The VLAN interface is a layer 3 interface in that case.

This is due to the architecture of the FreeBSD network implementation, you cannot directly compare it to an embedded router or switch.

 

[PackElend]

The layer 3 function moves up to the parent, because all bridge members share the same layer 3

can you tell, me what you mean by the parent? The bond or logical interface?
If so, it could mean that clients in such a setup could see each other:

Code:

et0 (tagged port at switch)
 -vlan8-br8-VM8
 -vlan9-br9-VM9

 

[Patrick M. Hausen]

In a switch you have VLAN X and that is assigned to multiple ports. So the interface named VLAN X is a logical/virtual switch in that context. All the ports assigned to that VLAN are bridged/switched.

In FreeBSD as with many routers (not switches!) a VLAN interface is a routed subinterface. You can create e.g. VLAN 5 on igb0 and VLAN 5 on igb1 but that does not imply that these two interfaces are in any way connected. They are not. If you want to turn FreeBSD into a software switch you need to create a bridge interface with these two VLAN interfaces as members. Now systems connected to these can communicate with each other. If the TrueNAS system itself needs to communicate in that bridge network, the IP address goes on the bridge interface and not the individual members. Like with your switch you put the IP address on the VLAN (virtual) interface and not the individual switched ports.

What in your switch is "VLAN x" in FreeBSD is bridgeX ...

 

[PackElend]

thx for not giving up on me

This is due to the architecture of the FreeBSD network implementation

also, apply to SCALE, as it is Debian based?
I read Routing and Firewalling VLANS with FreeBSD | Klara Inc and FreeBSD jails with VNET and NAT · boucek.me, as VNET is any FreeBSD thing I'm not sure if applies to SCALE as well. May they keep things similar in the foreground between CORE and SCALE.
I know Linux Virtualization, Chroot Jail but I cannot find an equivalent to VNET

In FreeBSD as with many routers (not switches!) a VLAN interface is a routed subinterface. You can create e.g. VLAN 5 on igb0 and VLAN 5 on igb1 but that does not imply that these two interfaces are in any way connected.

that is clear, the connection between VLAN 5 and 1 is routing --> Router

If you want to turn FreeBSD into a software switch you need to create a bridge interface with these two VLAN interfaces as members.

so the bridge does routing? Seems possible if I read Chapter 33. Advanced Networking | FreeBSD Documentation Portal
I would expect that only if, in our case, VLAN 1 and 5 are in the broadcast domain, the communication would work.
So adding a bridge upstream VLAN interfaces enable or include the routing capabilities of FreeBSD?
That is how I understand your configuration

Code:

------------------------------------------------------
| eth0                   eth1         TrueNAS SCALE  |
| ||                      ||                         |
| ----------bond0-----------                         |
|            ||                                      |
|          bridge1                                   |
|     |------||------------------|                   | 
|     |       |-----|            |                   |
|   VLAN1----------VLAN2---------VLAN3               |
|     |             |             |                  |
|   bridge1        bridge2        bridge3            |
|                                                    |
------------------------------------------------------

If the TrueNAS system itself needs to communicate in that bridge network, the IP address goes on the bridge interface and not the individual members. Like with your switch you put the IP address on the VLAN (virtual) interface and not the individual switched ports.

there is where it drives you nuts as a port is an Ethernet Interface as well :) but a port of the Bridge . The VLAN interface is your gate to L3 (on my MikroTik Router but only if I allow the VLAN to leave the bridge by allowing it to flow through the mysterious Bridge CPU Port).
And there is still

If you bridge a VLAN interface it becomes a strictly layer 2 interface. The layer 3 function moves up to the parent, because all bridge members share the same layer 3 ... If there is no bridge in place, you are correct. The VLAN interface is a layer 3 interface in that case.

Reading posts in the TrueCharts Discord, it is the bridge that gives access to the host, TrueNAS System or does it only apply to the setup above?
Otherwise, NIC->VLAN interface (without IP)->VM (single Docker container) would be real isolation as the bridge would still allow L2 access, regardless if it has an IP or not.
(on my router it is a bit different, with VLAN, even L2 communication is only possible if a VLAN interface is added, without IP config. That the Access Port is only a Member Port of the VLAN aware Bridge is not enough. I have tested that this morning).


all I want to achieve in the end is to extend my router on a stick topology to the application on my TrueNAS
I hope my understanding is correct, that VLAN binding is possible with docker as it allows interface assignment unlike Kubernetes (TN Apps).

Code:

     
     www
     |
     |
------------------------------
|  ROUTER                    |
|  FIREWALL, VLAN-GATEWAYS   |
------------------------------
     ||
     ||VLAN-TRUNK
     ||
------------------------------
|  SWITCH, VLAN CAPABLE      |
|  only L2 operation         |=== OTHER DEVICES such as APs
------------------------------
  ||                     ||   
  || TAGGED TRAFFIC ONLY ||  
  ||                     ||  
------------------------------------------------------
| eth0                   eth1         TrueNAS SCALE  |
| ||                      ||                         |
| ----------bond0-----------                         |
|            ||                                      |
|   VLAN1----------VLAN2---------VLAN3               |
|     |             |             |                  |
|   bridge1        bridge2        bridge3            |
|   |-VM |-Docker  |-VM |-Docker  |-VM  |-Docker     |
|                                                    |
------------------------------------------------------

(I'm aware of TrueChart's MetalLB and Docker-Compose but it is not said, if interface assignment in Docker-Compose allows real isolation)

 

[Patrick M. Hausen]

If you are running SCALE forget all I wrote and find someone who knows Linux and Docker networking. I am strictly referring to FreeBSD. Sorry about joining in in an apparently wrong thread. This SCALE/CORE thing is confusing.

But in case you are at least interested out of curiosity ...

In FreeBSD the bridge does not do routing but bridging, hence the name. It's a virtual switch.

In case of a Mikrotik switch or a router with switch silicon builtin a VLAN with n ports looks like this:

Code:

                ┌────────────────────────┐                  
 VLAN 5         │                        │                  
┌───────────────┴───────────────┐    ┌───┼──────────────────┐
│ ┌───┐ ┌───┐ ┌───┐ ┌───┐       │    │ ┌─┴─┐┌───┐┌───┐      │
│ │   │ │   │ │   │ │   │  ...  │    │ │ 5 ││ 6 ││ 7 │ ...  │
│ └───┘ └───┘ └───┘ └───┘       │    │ └───┘└───┘└───┘      │
│ Access ports (untagged)       │    │                      │
└───────────────────────────────┘    └──────────────────────┘
                                       Trunk port (tagged)  

In case of FreeBSD the role of the virtual VLAN 5 interface of your router is taken by a bridge interface. You can bridge a number of physical interfaces like this to turn your TrueNAS into a "switch":

Code:

 bridge0                        
┌───────────────────────────────┐
│ ┌───┐ ┌───┐ ┌───┐ ┌───┐       │
│ │   │ │   │ │   │ │   │  ...  │
│ └───┘ └───┘ └───┘ └───┘       │
│ igb0  igb1   VM 1  VM 2       │
└───────────────────────────────┘

You can also connect virtual interfaces like VMs and jails to that same bridge. These are virtual ports and they work just the same. They connect the VM to the physical LAN as if it was attached to a switch. It also allows the VM or jail to communicate with the host IF the host has got an IP address on that bridge interface. That's why the IP address must go on bridge0. You don't assign an IP address to all your ports of your Mikrotik, either. You assign it to VLAN 5.

Finally if you want a tagged VLAN interface in FreeBSD that's a subinterface of a physical port and never of a bridge. So instead of e.g. igb0 you end up with multiple interfaces:

igb0.5
igb0.6
igb0.7
...

and these send and receive tagged frames. If you want to connect VMs to that you need to create a bridge interface again, because FreeBSD by default does not switch. So if in my drawing you replace igb0 with igb0.5 and connect igb0 to a trunk port to a real switch, then the VMs will be connected to VLAN 5.

Linux does this roughly similarly but unfortunately I do not know the details. Your second drawing does make sense from a network engineers point of view, but I do not know how much control SCALE gives you about the assignment of containers to bridges/VLANs. In CORE you can build something like that easily. Build a trunk over lagg, N VLANs, N bridge interfaces and X, Y and Z jails connected to each bridge.

The main point is that in your Mikrotik the VLAN 5 is a (virtual) switch already. Where in FreeBSD and Linux a VLAN is just a subinterface. To connect multiple interfaces virtual or real, tagged or not, you need to create a bridge interface with all of these as members.

 

[PackElend]

But in case you are at least interested out of curiosity ...

for sure I am :)

so basically we are talking about traffic within different broadcast domains on TN. As soon as routing is involved it goes off TN towards the router (besides running a software router on TN).

find someone who knows Linux and Docker networking.

I cross my fingers that someone will join this discussion

 

[Patrick M. Hausen]

so basically we are talking about traffic within different broadcast domains on TN.

You know "broadcast domain" - why didn't you say so?

Exactly. If you want a broadcast domain that contains more than a single interface from TN's point of view - and this includes physical, VLAN, lagg and VMs, jails, ... - then you need a bridge interface that lumps all of these together.

 

[PackElend]

You know "broadcast domain" - why didn't you say so?

I still have difficulties sticking to the official wording, network speech, which makes it clear that I know at least some of the basics