Jail using incorrect VLAN

[jordanparry]

Info of setup
TrueNAS Version: TrueNAS-13.0-U3.1
PFSense firewall
VLAN11 - Management Net - 10.11.0.254
VLAN99 - Plex Net - 10.99.0.254



igb2 - Jails NIC - Network not configured directly on nic
igb3 - Management NIC - 10.11.0.2/24
vlan99 - parent interface igb2
bridge99 - bridge members - vlan99

I have been researching this for some time on both the forums and youtube and believe I have set everything up correctly, however when starting a plex jail, it gets an IP on the wrong subnet

I would like the plex Jail to get an address from DHCP of 10.99.0.0/24 but it seems to get an address of 10.11.0.0/24.

See screenshots below, I believe I have done this correctly after watching a video from Lawrence systems. The interface on my switch (US-16-150W) is set to all networks so that I can specify the VLAN interface in the jail using the correct bridge.

After poking around I noted that the ifconfig seems backwards where it has added igb3 instead of igb2 to the vlan

After realising that bridge0 somehow had an address of 10.99.0.1 (my desired IP for the plex jail) i destroyed that bridge using "ifconfig bridge0 destroy". Which has now seemingly corrected the bridge99 interface of having a member of igb3 however now it is trying to obtain an ip address from VLAN1 which does not have a DHCP server.

Have I missed something in telling the Jail to use VLAN99?

Any help would be massively appreciated

 

[NASbox]

I think you might have found a bug (Add is this a bug to your subject to attract dev's attention). Thought I would try to help you out, so I spun up a jail and then put it down and changed the VLAN.

I have a lag with several VLANs, so when I changed the VLAN to 192, under the net_default_interface setting I don't get a 192.168 address, instead I get the address from the first boot which is in another VLAN.

I ran the Jail update (I was using an old jail I had set up several months back that was empty), and now the IP address is stuck on 192.168.x.x and I can't get it to change. I'm wondering if something is getting set within the Jail that overrides the GUI settings.
I repeatedly stopped the jail, selected different VLANs from the selection drop down, saved, restarted the jail. I tried 2 different VLANs and AUTO a couple of times, and I always got the same IP Address.

For what it's worth I'm still on TrueNAS-12.0-U8.1

 

[Patrick M. Hausen]

You need to set the vnet_default_interface to "none" and further down in the interfaces section do the proper assignment. Also it helps to statically create all bridge interfaces in advance with the VLANs as members and assign the jails to the bridge and not the VLAN interfaces, All IP address configuration MUST be on the bridge, not the VLAN. A bridge member interface MUST NOT have any layer 3 address configured.

 

[jordanparry]

Thank you @NASbox I could swear I was going mad!

Can't see an option to edit the title, do I need to re-submit the post with a new title?

Glad to hear its not just me having this problem though it seems its been around for some time if present in 12.0-U8.1.



I have also noted the post from patrick however it seems I have set it up as described by them

 

[NASbox]

You need to set the vnet_default_interface to "none" and further down in the interfaces section do the proper assignment. Also it helps to statically create all bridge interfaces in advance with the VLANs as members and assign the jails to the bridge and not the VLAN interfaces, All IP address configuration MUST be on the bridge, not the VLAN. A bridge member interface MUST NOT have any layer 3 address configured.

Thanks for the reply @Patrick M. Hausen -- I need a bit of clarification:

ifconfig from a root shell shows I have a number of interfaces vlanXXX:

Code:

vlanXXX: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: VLANXXX
    options=400<LRO>
    ether 00:xx:xx:xx:xx:7a
    inet XXX.XXX.XXX.10 netmask 0xffffff00 broadcast XXX.XXX.XXX.255
    groups: vlan
    vlan: XXX vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
XXX.XXX.XXX.10 is the IP address of the box.
I also have a bridge with the 3 VLANS (XXX,YYY,ZZZ)  + a VNET entry
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:xx:xx:xx:39:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlanXXX flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 10000
    member: vlanYYY flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 10000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000
    member: vlanZZZ flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 10000
    groups: bridge
    nd6 options=1<PERFORMNUD>
The VNET entry is:
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: syncthing as nic: epair0b
    options=8<VLAN_MTU>
    ether 00:xx:xx:xx:xx:f8
    hwaddr 02:xx:xx:xx:xx:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

The main network interfaces tab shows an entry for exch VLAN:

Code:

Name      Type    Link State DHCP  IPv6 Auto Configure  IP Addresses
vlanXXX   VLAN    UP         no    no                   xxx.xxx.XXX.10/24
vlanYYY   VLAN    UP         no    no                   xxx.xxx.YYY.10/24
vlanZZZ   VLAN    UP         no    no                   xxx.xxx.ZZZ.10/24

Do I have all the required interfaces? What do I need to add if any and where?
I set the vnet_default_interface to None, and then under the network properties / interfaces it says vnet0:bridge0
I'm not sure what to change.

 

[Patrick M. Hausen]

You need to reboot with all jails disabled, then create the bridge interfaces in the UI, make the VLANs members of the bridge interfaces, then move the IP address configuration from the VLAN interfaces to the bridge interfaces if there is an IP address present. If a VLAN is used only for jails the NAS itself does not need to have an IP address on that. That all depends.

The main point is: the way the FreeBSD network stack works - and this is documented - a bridge member interface MUST NOT have an IP address configured. The IP address MUST be on the bridge interface. Otherwise things will simply break in unexpected ways.

HTH,
Patrick

 

[NASbox]

You need to reboot with all jails disabled, then create the bridge interfaces in the UI, make the VLANs members of the bridge interfaces, then move the IP address configuration from the VLAN interfaces to the bridge interfaces if there is an IP address present. If a VLAN is used only for jails the NAS itself does not need to have an IP address on that. That all depends.

The main point is: the way the FreeBSD network stack works - and this is documented - a bridge member interface MUST NOT have an IP address configured. The IP address MUST be on the bridge interface. Otherwise things will simply break in unexpected ways.

HTH,
Patrick

@Patrick M. Hausen Thanks for your reply.... even though the board is calling me a "Guru" it is grossly exaggerating... I'm functional, but not very experienced with BSD and networking.

My first question is, if I take the IP Addresses off the VLANs, will I loose connectivity to the box?
I have 3 VLANs assigned, and they all include static addresses xxx.xxx.xxx.10 which are the addresses that can be used to access the box on these 3 VLANs. The interfaces seem to have created a /24 out of each of these addresses in their respective VLAN interfaces (xxx.xxx.xxx.10/24). If I remove the address and move it to a bridge what happens?

I assume that I need to create these interfaces in the Network/Interfaces tab?

Should I create 1 bridge/VLAN?

If I have multiple jails using the same VLAN, do I need to create multiple bridges?

Any idea where I might find an example?

Thanks again.

 

[Patrick M. Hausen]

One bridge for each VLAN that connects jails.
If you move the IP address from the VLAN to the bridge you will not lose connectivity. You might want to increase the timeout for the "Test" phase of the network UI from 60 seconds to e.g. 300. Connectivity will drop for a short time then come back.
Yes, you create the bridge interfaces in the network UI.

 

[jordanparry]

I've followed this but I just get an error saying that it will not get a DHCP address when trying to use bridge99 (Vlan99). I have plugged something else into this switch port and given it a vlan of 99 and it gets an address without issue. It seems something is broken but as to what is a mystery.

The only interface with a statically assigned address is the management interface.

 

[Patrick M. Hausen]

This does not look like you enabled DHCP for the bridge99 interface? Does your NAS need an IP address in VLAN99? Why not set statically?
My VLAN 2 serves only jails and VMs - so no IP address necessary. The rest is configured statically.



Also you might want to set these two tunables. This gives the bridge interfaces a permanent predictable MAC address.

 

[NASbox]

@jordanparry just wondering if you got it figured out?

@Patrick M. Hausen thanks for the post....
Do I understand correctly that bridge1 is used to access the GUI?
Are there any jails connected to it?
If so any chance you could show me how the bridge/vlan(s) are defined.
I'm still struggling with this stuff. I have 1/2 a clue about linux netwoking and almost no clue when it comes to FreeBSD.

I see that you have some serious systems and have both SCALE and CORE running. How would you rate the maturity/stability of SCALE? Is it ready for "production" use, or are there still a lot of rough edges? What types of workload do you have running in the Jails/VMs

I was thinking that maybe I should be considering a switch to SCALE... I would really like to be running a bunch of docker containers to provide services... NextCloud, SearxNG, just for starters. These types of containers are pretty much ready to go for Linux, it seems like I'd have a bit of a learning curve to set up jails from scratch.

My box is a Gen2 i7-3770 with 32GB of DDR3 that sits idle most of the time. I use it for backups and a library for training materials, videos, photos, music, business records, and any other stuff that I want to store long term. Since I'm almost the only user the workload is minimal, but I really rely on it and stability is more important than extra features. I probably should ask this question in another thread, but since we have this discussion going I would really value your input as I know you are very experienced.

Your thoughts (and anyone else with something to add)?

 

[Patrick M. Hausen]

Do I understand correctly that bridge1 is used to access the GUI?

Yes.

Are there any jails connected to it?

Also yes. Otherwise I would not need a bridge. Actually just right now - no. I'm in the process of moving things to get an IDS in front of my public services.

If so any chance you could show me how the bridge/vlan(s) are defined.

I'll show you both the bridge1 that does not currently have jails but could as well as the bridge2 that does have jails. Again note that you can run the bridge interfaces with an IP address (bridge1) or without (bridge2) depending on whether the NAS itself needs to communicate in that particular VLAN.

So here's the basic networking:



And here's the jail:



The important part is to set vnet_default_interface to "none". Then in the network settings I assigned the interfaces as "vnet0:bridge2".

I see that you have some serious systems and have both SCALE and CORE running. How would you rate the maturity/stability of SCALE? Is it ready for "production" use, or are there still a lot of rough edges? What types of workload do you have running in the Jails/VMs

I run SCALE with one productive application - Onlyoffice. Everything else is on CORE. That is not going to change (much) because I run most of my applications in jails. Jails are the most stable and versatile container technology in existence. We provide hosting services based on jails and ZFS and are shortly reaching 1000 productive customer jails. Networking for jails is straightforward, simple, easy to debug - if you know your FreeBSD basics. The problem is that iX messed up the bridge setup.

There's an open issue in JIRA. I have been complaining for years. I'm tired. You can get everything working correctly with manual configuration as shown. Our own hosting environment of course does it right from the start. We use Ansible for that.

[NAS-114755] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

The problem is that in the default configuration - one network port, user spins up a jail, plugin, VM - they try to be user friendly and automatically create a bridge interface. Then the system ends up with a physical interface with IP address configuration and a bridge that contains that as a member. This is explicitly forbidden by the FreeBSD documentation. A bridge member interface MUST NOT have an IP address. Simple, really. The IP address MUST be on the bridge, not the member interface(s).

Your situation with all VLANs in one bridge is caused by that "magic" of iX not being prepared for multiple networks connected to jails. People with more complex setups have encountered all sorts of surprises including e.g. bridge loops and broadcast storms.

I was thinking that maybe I should be considering a switch to SCALE... I would really like to be running a bunch of docker containers to provide services... NextCloud, SearxNG, just for starters. These types of containers are pretty much ready to go for Linux, it seems like I'd have a bit of a learning curve to set up jails from scratch.

True. If they are ready to use. I prefer to roll my own installations to make sure they are ready to use. Your choice.

Kind regards,
Patrick

 

[NASbox]

@Patrick M. Hausen .... Thank you, Thank you, Thank you.... I'll have a crack at this over the next couple of days and see if I can figure it out. I may have more questions later.

I'm wondering what type of workload your jails are hosting? Is it some specialty application, website hosting, file serving?

I don't know if the applications I need will run under FreeBSD. I assume NextCloud does, I think SearxNG is just a PHP app, so it should be OK. I don't know about Guacamole and some of the other things that I want to run.

Can you comment on the maturity of SCALE? If I do need Linux would I be better off to move to SCALE or set up a Linux VM on CORE and run DOCKER in the VM? My impression is that FreeBSD is more secure/more robust than Linux, and certainly a bit of diversity might help if some sort of worm got into the network. Is this impression well founded?

.... Jails are the most stable and versatile container technology in existence. .... Networking for jails is straightforward, simple, easy to debug - if you know your FreeBSD basics.
....
True. If they are ready to use. I prefer to roll my own installations to make sure they are ready to use. Your choice.

How big a learning curve am I in to to know enough FreeBSD Basics? How would you suggest I get up to speed quickly?

I agree that I too prefer to make my own installation, but I'm not sure that I'm up to the task without spending more time than I can afford.

 

[Patrick M. Hausen]

Code:

root@freenas[~]# iocage list
+-----+-----------+-------+--------------+---------------------------+
| JID |   NAME    | STATE |   RELEASE    |            IP4            |
+=====+===========+=======+==============+===========================+
| 2   | cloud     | up    | 13.1-RELEASE | 192.168.2.53              |
+-----+-----------+-------+--------------+---------------------------+
| 3   | gitea     | up    | 13.1-RELEASE | 192.168.2.51              |
+-----+-----------+-------+--------------+---------------------------+
| 6   | grafana   | up    | 13.1-RELEASE | 192.168.2.55              |
+-----+-----------+-------+--------------+---------------------------+
| 5   | mineos    | up    | 13.1-RELEASE | 192.168.2.56              |
+-----+-----------+-------+--------------+---------------------------+
| 7   | observium | up    | 13.1-RELEASE | 192.168.2.54,217.29.46.43 |
+-----+-----------+-------+--------------+---------------------------+
| 1   | proxy     | up    | 13.1-RELEASE | 192.168.2.50              |
+-----+-----------+-------+--------------+---------------------------+
| 4   | rdp       | up    | 13.1-RELEASE | 192.168.2.52              |
+-----+-----------+-------+--------------+---------------------------+

"cloud" is Nextcloud
"proxy" is a reverse SSL proxy for all public facing ingress
"rdp" is Guacamole

In addition I have two Ubuntu VMs running Atlassian Confluence. The proxy jail does ingress for these, too.

Maturity of SCALE - no idea. I just toy around with it. Tested the cluster features, wasn't too impressed. Ubuntu VMs run rock solid on CORE for me. Other forum members seem to have problems. We have not yet figured out what the common factor of the problematic ones might be.

To learn FreeBSD, Michael W. Lucas' books are generally considered the best source:

About Me - Michael W Lucas

Welcome!

mwl.io