SOLVED VLANs and bridge - Something is broken

[Itay1778]

Hi, everyone
So I remake my network (new switches, VLANs, etc), and the time to add VLANs to TrueNAS arrive.
So I add the 2 VLANs that I need one is for management and one is for file share (SMB, NFS) and jails.
It took me a few tries but I managed to get everything that I want to a working state. But I have one problem the Management VLAN (VLAN2) adds itself to the bridge that is used in the jails, and it shouldn't be like that.
My networking config:
physical interface: em0 - tag VLANs - 2,3 untag - none
I add in the WebUI
VLAN2 - Management (TrueNAS WebUI IP, SSH)
VLAN3 - SMB IP, Jails - From the bridge0
bridge0 (Bridge Members - VLAN3)
So far everything looks fine but for some reason, I still don't understand her VLAN2 added itself to bridge0 and it is not good!
I try to add to system > Tunables

Variable	Value	Type
ifconfig_bridge0	deletem vlan2	RC

don't do much...

what to do? for now, if I want my jails to work properly I need manually to run this cmd after a reboot (or at least I hope is only needed after a reboot) ifconfig bridge0 deletem vlan2

Hope that I explained everything clearly.

 

[Patrick M. Hausen]

You need to manually create all bridge interfaces you do want to use and then in the jails' network config set vnet_default_interface to "none" and explicitly set the e.g. "vnet0|bridge0" interface for the jail.

Otherwise TN will try to perform all sorts of magic finding the physical interface with the default route and creating a bridge on the fly etc.

To be able to do that, set the auto boot flag of all your jails and possibly VMs to off and reboot your TN. This way it will come up without bridge interfaces. Then you can create bridge0, bridge1, ... with appropriate VLANs as members. I like to number the bridges just like the VLAN in question.

And then - important - if you have IP address configuration on VLAN X you MUST remove that from the VLAN and put it on the bridge interface. Documented in the FreeBSD handbook but unfortunately not in the TrueNAS docs.

 

[Itay1778]

You need to manually create all bridge interfaces you do want to use and the in the jails' network config set vnet_default_interface to "none" and explicitly set the e.g. "vnet0|bridge0" interface for the jail.

Otherwise TN will try to perform all sorts of magic finding the physical interface with the default route and creating a bridge on the fly etc.

To be able to do that, set the auto boot flag of all your jails and possibly VMs to off and reboot your TN. This way it will come up without bridge interfaces. Then you can create bridge0, bridge1, ... with appropriate VLANs as members. I like to number the bridges just like the VLAN in question.

And then - important - if you have IP address configuration on VLAN X you MUST remove that from the VLAN and put it on the bridge interface. Documented in the FreeBSD handbook but unfortunately not in the TrueNAS docs.

Thank you!! like I said already I create a new bridge in the UI, but what I forgot is to move the vnet_default_interface to "none" and now it's working as it should be. And I manage to do that without rebooting, I stop the jails change to none, remove manually the member the I don't want in the bridge, and started the jails.

Why do I need to move the IP from the VLAN to the bridge?

 

[Patrick M. Hausen]

Because the FreeBSD documentation says so The real reason is that having IP configuration on a bridge member interface breaks multicast. So some IPv4 applications won't work and IPv6 won't work at all. A bridge interface is a virtual switch. A switch port (member interface) is layer 2 and in general does not have an IP address. So the address needs to be on the bridge.

Kind regards,
Patrick

 

[Itay1778]

Because the FreeBSD documentation says so The real reason is that having IP configuration on a bridge member interface breaks multicast. So some IPv4 applications won't work and IPv6 won't work at all. A bridge interface is a virtual switch. A switch port (member interface) is layer 2 and in general does not have an IP address. So the address needs to be on the bridge.

Kind regards,
Patrick

Okay, got it. Thanks for the explanation.
Moving the IP to the bridge.

 

[victort]

Good morning,

Im having an issue trying to get VLANs to work also.
I currently have one physical NIC only, that is attached to my LAN with an IP.

1. I have created a VLAN.
2. Create a bridge50 with the VLAN as the member and give it an ip of my VLAN network.
3. Assign the jail to use bridge50 as its vnet, and set the vnet_default_interface to none.

But the jails still cant get an IP from my VLAN DHCP.

Interfaces are configured like this.

em1 physical with a LAN ip
vlan50 parent is em1 with no ip
bridge50 member is vlan50 with VLAN IP

 

[victort]

Good morning,

Im having an issue trying to get VLANs to work also.
I currently have one physical NIC only, that is attached to my LAN with an IP.

1. I have created a VLAN.
2. Create a bridge50 with the VLAN as the member and give it an ip of my VLAN network.
3. Assign the jail to use bridge50 as its vnet, and set the vnet_default_interface to none.

But the jails still cant get an IP from my VLAN DHCP.

Interfaces are configured like this.

em1 physical with a LAN ip
vlan50 parent is em1 with no ip
bridge50 member is vlan50 with VLAN IP

 

[Patrick M. Hausen]

Configure your jails with static IP addresses. Or search a bit - DHCP needs special jail permissions and iX removed some of the tuneables from the UI. You would need to set at least iocage set dhcp=on <jailname> and possibly that will tell you more options that are necessary.

 

[victort]

I’m not sure if my interfaces are set up properly. I’ve searched but can’t seem to find a good on how to do it. Especially if I have only one NIC.

 

[Patrick M. Hausen]

VLAN on NIC, bridge with VLAN as member, jail assigned to bridge. You cannot use the same NIC untagged - it must be all VLANs ...

 

[victort]

VLAN on NIC, bridge with VLAN as member, jail assigned to bridge. You cannot use the same NIC untagged - it must be all VLANs ...

So I can’t have my default LAN network plugged in?

How do you assign a VLAN to a NIC?

Could I ask you for a step by step guide from static routes and everything? If I need those?

Assuming I’m starting from TN connected to my LAN on one physical NIC as the only interface.

 

[Patrick M. Hausen]

When you created the vlan50 you set a parent interface, didn't you? If that is em0 you cannot put an IP address on em0 in parallel. You can for example create another VLAN, e.g. 10, put that IP address there, then tell your switch to put all LAN ports in VLAN 10.

You do have a VLAN capable switch and want to connect your TrueNAS over 1 NIC to carry multiple networks, right?

Something like this?

Code:

┌─────────────┐                      ┌──────────────────────────┐   
│             │      Trunk Port      │                          │   
│  TrueNAS    │──────────────────────│     Switch with VLANs    │   
│             │  VLANs 1, 2, 3, ...  │                          │   
└─────────────┘                      └───┬─────────┬─────────┬──┘   
                                         │VLAN 1   │VLAN 2   │ VLAN 3
                                         │         │         │       
                                         │         │         │       
                                         │         │         │       
                                         │         │         │       
                                     ┌──────┐  ┌──────┐  ┌──────┐   
                                     │      │  │      │  │      │   
                                     │  PC  │  │  PC  │  │  PC  │   
                                     │      │  │      │  │      │   
                                     └──────┘  └──────┘  └──────┘    

 

[victort]

When you created the vlan50 you set a parent interface, didn't you? If that is em0 you cannot put an IP address on em0 in parallel. You can for example create another VLAN, e.g. 10, put that IP address there, then tell your switch to put all LAN ports in VLAN 10.

You do have a VLAN capable switch and want to connect your TrueNAS over 1 NIC to carry multiple networks, right?

Something like this?

Code:

┌─────────────┐                      ┌──────────────────────────┐  
│             │      Trunk Port      │                          │  
│  TrueNAS    │──────────────────────│     Switch with VLANs    │  
│             │  VLANs 1, 2, 3, ...  │                          │  
└─────────────┘                      └───┬─────────┬─────────┬──┘  
                                         │VLAN 1   │VLAN 2   │ VLAN 3
                                         │         │         │      
                                         │         │         │      
                                         │         │         │      
                                         │         │         │      
                                     ┌──────┐  ┌──────┐  ┌──────┐  
                                     │      │  │      │  │      │  
                                     │  PC  │  │  PC  │  │  PC  │  
                                     │      │  │      │  │      │  
                                     └──────┘  └──────┘  └──────┘    

Yes to all.

But the em0 has the LAN network, which is untagged.

I want to run multiple VLANs on TN but have TN be on the LAN.

Switch is Unifi switch which has all ports trunked by default. I know this because on Proxmox when I create a VM with a VLAN tag, it gets an IP from the VLAN DHCP.

 

[Volts]

Configure the Unifi switchport to perform tagging for the default/management VLAN 1 on the port connected to TrueNAS.

(In general I would encourage you not to think of VLAN 1 as the LAN in any special way. It's just another one of the VLANs like all the rest.)

 

[victort]

I followed this guide, but even VMs cant connect to the vlan50 network.

How to setup VLANs within FreeNAS 11.3

This post is a follow-up solution for solving how to setup VLANS with 11.2U7 (https://forums.lawrencesystems.com/t/freenas-jails-with-vlan/3046/11). Both 11.2 and 11.3 are VLAN able, however the documentation for getting a working setup is lacking. Pre-requisites: 1. This guide assumes your...

www.truenas.com

I was able to get to the TrueNAS using the VLAN1 interface, but thats about all.

 

[Volts]

That guide isn’t half bad. It’s hard to guess what might still be necessary on your system.

Share a screenshot of the switch port profile, the TrueNAS VLAN and bridge interface settings, a jail’s network settings, and the output of ifconfig on the host.

 

[victort]

After much headache I finally got it to work. My router wasn’t resolving the domain name to the correct IP.

Switch port profile is tagging LAN as VLAN 1 and VLAN 50 as VLAN 50 going into my TrueNAS.
TrueNAS has four interfaces:
em0 physical with no IP

vlan1 with main IP that I use linked to em0

vlan50 also linked to em0 with no IP

bridge50 with vlan50 as member (IP doesn’t matter) or does it?

I also had to make sure to set vnet_default_interface to none in jail settings, and vnet0|bridge50

VMs can just be set to use the bridge50 in the NIC device settings.

 

[victort]

That guide isn’t half bad. It’s hard to guess what might still be necessary on your system.

Share a screenshot of the switch port profile, the TrueNAS VLAN and bridge interface settings, a jail’s network settings, and the output of ifconfig on the host.

That guide worked perfectly for me as I had physical access to my TrueNAS.

 

[Patrick M. Hausen]

If the NAS host does not need to communicate in VLAN 50 and it is just for hosting jails, the NAS does not need an IP address on that bridge interface at all. The entire VNET/bridge/etc. architecture is layer 2. Glad to see you got it working.

 

[victort]

[EFAULT] Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/ix-plugin-hub/iocage-plugin-index.git /mnt/RSCPOOL2/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git stderr: 'Cloning into '/mnt/RSCPOOL2/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git'... fatal: unable to access 'https://github.com/ix-plugin-hub/iocage-plugin-index.git/': Couldn't connect to server '

Having this issue now when trying to get to community plugins. Internet connection seems up.