Attach VLAN to Jail: How ???

[Andy McClements]

FreeNAS-11.3-U1

I want to run an instance of Plex server in a Jail, and need this to be attached (only) to a VLAN separate to anything else the box is connected to. I already successfuly use Vlans on this box for other services running on the main OS instance'. I have the new VLAN already configured and tested OK by running up a new main OS interface to check connectivity.

I created a bridge interface instance 'bridge0', with the new VLAN as the only member.

I created and saved a new Jail, using VNET, with ipv4 interface 'vnet0:bridge0', and the desired IP/Mask/Gateway. So far all good.

However, when I attempt to start the Jail, the GUI reports:

Starting Jail
Error: [EFAULT] Stopped plex due to VNET failure

The log says only:

[2020/03/09 14:45:14] (ERROR) iocage.callback():62 - + Configuring VNET FAILED

The FreeNAS documentation does not seem to address this sort of configuration. Is there a HOWTO or something else I can refer to for working with VNET, and Jails with VLAN's ?

Google suggests lots of folk have in the past have had trouble with my sort of configuration, but of course a lot has changed in FreeNAS so possibly much of what I've found is irrelevant.

It'd be good to hear from anyone if this is even a supported configuration, and if so how to achieve it !

 

[ornias]

I created a bridge interface instance 'bridge0', with the new VLAN as the only member.

This scentence needs a place in a book: "FreeNAS and how to break things", you shouldn't mess with bridge0 manually.

I created and saved a new Jail, using VNET, with ipv4 interface 'vnet0:bridge0', and the desired IP/Mask/Gateway. So far all good.

Nope: so far multiple things are already broken byond jail compatibility.


One rule:
The order of Interfaces:
1. Physical
2. Link aggragation (optional)
3. VLAN
4. Bridge

If you want to have a VLAN connected to a Jail or VM, only the HIGHEST (highest number in above order) interface is allowed to have a IP connected to it and the highest interface NEEDS to be a bridge.

So: If you have a VLAN10, you need to create a BRIDGE10 and ONLY set the IP on the bridge, and NOT have an IP on either the: VLAN, Link-agg or Physical interace.


A good how-to:

How to Configure Jail VLANs with TrueNAS

More TrueNAS Tutorialshttps://lawrence.technology/truenas-tutorials/Related IX Systems Forum Posthttps://www.truenas.com/community/threads/how-to-setup-vlans...

www.youtube.com

But note when doing this how to: By messing with bridge0 you might have already broken stuff.

 

[Andy McClements]

Er.. thanks for the reply.. I think..

(BTW, I'm a 57 year-old ex university network admin so I do have some idea about networking. So you think I did something in the FreeNAS GUI which it allowed me to do without warning me, that is not adequately covered in the frankly pisspoor iXSystems handbook. Well, I am not going to beat myself up about that. Sorry..)

Thanks for the link to a Youtube video. It wouldn't have ocurred to me to use Youtube as a resource for using FreeNAS. I will see if I can endure the video long enough to learn something.

Edit: the video linked, links to a good old forum post on a related topic, I'll try that first before resorting to the video :)

 

[ornias]

Er.. thanks for the reply.. I think..

Why the "I think"?
I don't blame you for not getting VLAN's working (welcome to the club of people not managing to get it working due to the piss-poor manual).

I'm a 57 year-old ex university network admin so I do have some idea about networking

I am a lot of what people call "annoying", but I don't estimate skill (or lack thereoff) based on age, race, sex or gender.
But as an ex-network admin you should've run an ifconfig and noticed bridge0 already existed before trying to mess with it ;)

The problem here is not networking in general, the problem is that somehow the stack FreeNAS uses doesn't work well with having IP's on multiple virtual interfaces that are connected. Although it that might be the case with more operating systems actually (albeit less obvious) because I had similar issues with proxmox once...

So you think I did something in the FreeNAS GUI which it allowed me to do without warning me, that is not adequately covered in the frankly pisspoor iXSystems handbook

Well, I think freenas is smart enough that removing the bridge0 and rebooting should fix the problems caused :)
(remember to do so before following said guide/instructions)

Well, I am not going to beat myself up about that. Sorry..)

You don't have to beat up yourself over it. making mistakes is human. The bridge0 thing was not the important part of what I was trying to tell... The important part was the part about the interfaces and the video I linked

Thanks for the link to a Youtube video. It wouldn't have ocurred to me to use Youtube as a resource for using FreeNAS. I will see if I can endure the video long enough to learn something.

Well, its not that long if you skip to the relevant bits ;)

Edit: the video linked, links to a good old forum post on a related topic, I'll try that first before resorting to the video :)

That forum post should've been called:
"How to overcomplicate VLAN setup within FreeNAS 11.3"

It's more like a deep dive with background info and some info that is usefull to bugtrace.

The TLDR instructions are:
- Make sure you have this stack for each vlan you want to use (including vlan0/1), under "interfaces":
Physical (no ip) -> Agragation (no ip) -> VLAN (no ip) -> bridge (Optional: IP)

- Go to jails, On the tab, select the jail and click "edit"

- Go to the networking tab and "interfaces" replace "vlan0:bridge0" with the bridge connected to the vlan listed above.

This is my config:


here you can see I have 3 physical interfaces, connected to 1 Link aggragation, all without IP's.
Attached to that are multiple VLANS (all without IP's).
To each vlan is attached 1 Bridge (with the same number as the vlan)
Putting an IP on the bridge is optional, if the bridge is used by FreeNAS itself it needs an IP, if it's only used by jails or VM's it doesn't have to have a IP itself.

Then under a random jail this is my config:


Notice under "interfaces" it states "vnet0:bridge31"

Hope this helps you out :)

 

[colmconn]

This may help: https://www.youtube.com/watch?v=l6OsF5ppQnU Though I've not tried it.

 

[ornias]

This may help: https://www.youtube.com/watch?v=l6OsF5ppQnU Though I've not tried it.

Thats the same video I just linked...
It might be worthwhile reading up a bit in a topic before replying to prevent repeating what's already said ;)

 

[Andy McClements]

Hope this helps you out :)

Hi, thanks for all that. I backed out of the bad config no problem. I then was able to get the jail created and starting up OK, without the previously reported errors. This was by using the GUI to create a new bridge (bridge40) with just the VLAN (vlan40) as member. Then, create the Jail in advanced mode, selecting DHCP & VNET in Basic Properties, and in Network Properties, entering 'vnet0:bridge40'.

I just wanted to share a few further observations, if you have any more insights, I'd be glad to hear them.

According to 'ifconfig -a':

1. 'bridge0' does not exist
2. 'bridge40' not only has Vlan40 as member (as I have configured), but also, 'vnet0.1', AND the parent interface of Vlan40, which is a LAGG.

I'm finding the Jail is being connected to the VLAN which is configured to pass untagged frames on the LAGG, and not the desired 'vlan40'.

It may be that I'm hitting a limitation which requires the VLAN parent interface to be carrying NO untagged traffic ?

 

[ornias]

Ohh lordy, Thats what I warned for, you didn't back out of the old config issue correctly... You assumed you did based on the GUI.
If you don't have bridge0 i'm afraid it's defaulting to your newly created bridge40.

A clean (not messed up) FreeNAS config, has by default a (Hidden from gui, visable with ifconfig -a ) bridge0, this is the default bridge and A LOT of freenas features depend on it. If you don't have it your setup is fucked.


I think FreeNAS is trying to compensate for you removing/messing with the default Bridge0 and is making your new bridge the default, this is going to cause A LOT of problems down the road.

I suggest just grabbing the backup of your freenas config and restore that one. (I assume you made one before messing with the networking, as you are a former professional)

*edit*
How I know this? Well your new Bridge wouldn't get a default LAGG assigned, that one should be auto assigned to the default (hidden) bridge0.

 

[Andy McClements]

Hi, thanks for the reply but I'm not sure you're correct there, the way I reverted the config was to reboot from my previous 11.2-U8 boot partition, delete the 'messed up' 11.3 environment, and re-run the upgrade, creating a fresh 11.3 boot image with the previous 'un-messed-up' config. Yes, naturally I have config backups also, which is another option.

Anyway, surely one of the points of a GUI is to validate user inputs ? If 'messing' with bridge0 is such a no-no, why did the GUI allow it ? It certainly does validation in many other respects.

Edit: Yup, just rebooted into my previous 11.2 config, which has been running undisturbed for months,'ifconfig -a' lists no 'bridge0'.

 

[ornias]

I'm not sure you're correct there,

I solved these problems before.
Your Link-agg shouldn't have been auto connecting to your self-created bridge.

the way I reverted the config was to reboot from my previous 11.2-U8 boot partition, delete the 'messed up' 11.3 environment, and re-run the upgrade, creating a fresh 11.3 boot image with the previous 'un-messed-up' config. Yes, naturally I have config backups also, which is another option.

I'm not sure this would work. It might've, be appearently: it didn't.
The reason I suggested a reinstall with a config backup is so we can be sure your previous tryouts don't mess up anything along the road. Been there, done that.

Anyway, surely one of the points of a GUI is to validate user inputs ? If 'messing' with bridge0 is such a no-no, why did the GUI allow it ? It certainly does validation in many other respects.

Well, lets make a summary:
- You have a not working configuration
- I have a working configuration

Considering what you told me, it looks like our configs are pretty similar.

I hit the "oops I messed with Bridge0 and now things go weird" problem before about 2-3 times, not touching bridge0 makes sure I don't hit it again. So I guess thats one of the reasons my config works ;)

If you have a nice clean config, the how-to guide from Lawrence Systems is VERY good.
With my notes (also available under the video, with a little more precision), you should be able to get your link aggregation working too.

But the thing is: As long as you messed with Bridge0, I can't give guarentees....

 

[Andy McClements]

Thanks again for continued input. I just checked a FreeBSD 11.3 box sitting in the same rack as this FreeNAS box, 'ifconfig -a' reports no 'bridge0' device, just the loopback and primary NIC's.

I then did as you suggested, and reverted the FreeNAS box to 11.2-U6 which it was running until yesterday morning, until I upgraded > 11.2-U8 > 11.3-U1. I then restored the config backup taken when going 11.2-U6 > 11.2-U8. 'ifconfig -a' reports no 'bridge0'.

TBH, based on my knowledge and experience, I see no problem with this, as a bridge only is required if needing a link between two interfaces and a typical server without VNET would not require this.

My plan now, is to re-create the LAGG, as a pure trunk with no untagged traffic, recreate the VLAN's, and try again...

 

[Patrick M. Hausen]

There are two approaches I am aware of. If you go straight ahead "the traditional FreeNAS way" you rarely have to manage bridge interfaces manually. You simply go along and configure in this order:

1. physical - "up" if more higher level stuff is to follow
2. lagg - if applicable
3. VLAN - if applicable, then put your IP address for host system connectivity here if desired (could be an additional VLAN for VMs/jails only ...)

Then when you attach a jail to the desired VLAN all the necessary bridging is autoconfigured.

The downside is that this is "wrong" as documented in the FreeBSD handbook. The IP address is supposed to go onto the bridge interface. All of this implying that you use a single interface/bridge combination for host (FreeNAS) and VM/jail communication simultaneously. If not, you do not configure an IP address at all. Simply attach the jails to the VLAN and bridges will be created as necessary. Problem solved.

I personally do mess with bridge0 in setups where there is only one interface for hosts, all jails and VMs and no VLANs like my home network. To have the IP address on the bridge like the FreeBSD documentation says. But I do know quite well what I am doing here and at least for now the "wrong" way works, too - FreeNAS has been doing this since the introduction of VNET jails. (put IP address on interface and bridge on the "outside")

HTH,
Patrick

P.S. If I am not mistaken you do not have a bridge0 interface if you have neither a jail nor a VM.

 

[ornias]

There are two approaches I am aware of. If you go straight ahead "the traditional FreeNAS way" you rarely have to manage bridge interfaces manually. You simply go along and configure in this order:

1. physical - "up" if more higher level stuff is to follow
2. lagg - if applicable
3. VLAN - if applicable, then put your IP address for host system connectivity here if desired (could be an additional VLAN for VMs/jails only ...)

Then when you attach a jail to the desired VLAN all the necessary bridging is autoconfigured.

The downside is that this is "wrong" as documented in the FreeBSD handbook. The IP address is supposed to go onto the bridge interface. All of this implying that you use a single interface/bridge combination for host (FreeNAS) and VM/jail communication simultaneously.

If not, you do not configure an IP address at all. Simply attach the jails to the VLAN and bridges will be created as necessary. Problem solved.

I never got the "auto configuring of bridging" to work with more than a single vlan connection, thats also why Tom Lawrence made his video and one of the conclusions of the topic he based his video on..

I personally do mess with bridge0 in setups where there is only one interface for hosts, all jails and VMs and no VLANs like my home network. To have the IP address on the bridge like the FreeBSD documentation says. But I do know quite well what I am doing here and at least for now the "wrong" way works, too - FreeNAS has been doing this since the introduction of VNET jails. (put IP address on interface and bridge on the "outside")

Yeah, Thats why I ignored the comment about there not being a warning... There are cases you might want to mess with it... But the thing is "knowing the consequences" ;)

P.S. If I am not mistaken you do not have a bridge0 interface if you have neither a jail nor a VM.

Shoot, I was wrong there indeed!
(lagg still shouldn't auto connect to bridge40 though :P )

 

[Patrick M. Hausen]

I never got the "auto configuring of bridging" to work with more than a single vlan connection, thats also why Tom Lawrence made his video and one of the conclusions of the topic he based his video on..

That's odd - we have that in production here with 11.3-U1 (started with one of the alpha nightlies, because we needed the FreeBSD 11.3 kernel, production or not) and multiple VLANs and VMs assigned to them. Never touched a bridge interface.

 

[ornias]

That's odd - we have that in production here with 11.3-U1 (started with one of the alpha nightlies, because we needed the FreeBSD 11.3 kernel, production or not) and multiple VLANs and VMs assigned to them. Never touched a bridge interface.

Well, creating the bridge interface (phys<->lagg<->vlan<->bridge) myself (just like Tom does and the topic he links to advices) "just works" (tm), so i'm quite happy now that I FINALLY got VLANs to work.


Although I also needed host access to some of the VLANS, so I needed to create some bridges manually anyway to put an IP on them.

 

[Andy McClements]

Thanks guys. So, I've deleted and re-created my old VLANs and LAGG, and re-created the LAGG with no L3 configured on it, and added the 2 L3 VLAN's and the new VLAN ('vlan40') I want to use for the Jails with no IP configured.

As per the video linked above, I've created a new bridge 'bridge40', with just 'vlan40' as member.

I've created a new jail, using DHCP & VNET, and under Interfaces, entered 'vnet0:bridge40'.

The jail starts OK, BUT it as assigned a DHCP lease from the primary LAN interface of the box. Sure enough, 'ifconfig -a' on the host box reports:

bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN2 Jail bridge
ether 02:85:9a:7a:3f:28
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 20000
member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 20000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: plex as nic: epair0b
options=8<VLAN_MTU>
ether 38:ea:a7:14:fa:09
hwaddr 02:5f:10:00:0c:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

'vlan30' is being added as a member of the bridge and is taking precedence when the Jail is leased an address.

This is contrary to the bridge configuration I've assigned, so I'd be very open to suggestions as to WTF is going on..

TIA

 

[ornias]

Give every VLAN its own bridge... I guestimate that should prevent it from auto-adding vlan30 to bridge40

 

[Andy McClements]

It's an unresolved bug, already reported:

https://jira.ixsystems.com/browse/NAS-105329

For me, (as an ex network admin) it seems quite a serious problem. In a production environment you certainly do not want bridges being created spuriously between network segments.

I'm not really keen on speculatively hacking around with my box further to find a workaround, as I've wasted enough time this already TBH.

 

[Andy McClements]

I've created a Jira account and added a comment to the bug. It seems quite a serious security fault to have a jail (running potentially untrusted software) being connected to an incorrect network.

If you feel like upvoting the bug, please do !

 

[ornias]

For me, (as an ex network admin) it seems quite a serious problem. In a production environment you certainly do not want bridges being created spuriously between network segments.

Yeah thats definately not a "low priority bug" imho.

I'm not really keen on speculatively hacking around with my box further to find a workaround, as I've wasted enough time this already TBH.

Did you try creating a bridge for every vlan? Thats should solve the bug.
THe bug should be triggered when there is a vlan without bridge.

Why? I've the same setup (almost precisely as you can see), with the only difference being I've a (placeholder) bridge for every vlan ;)