VM and Jail bridge attachment at boot

[ad31677]

I have a multihome TrueNAS box (see signature) with networking roughtly like this:

oce0 ----> LAGG0 ----- bridge0 [ ADMIN NET xxx.xxx.161.0/24 ]
igb0 --------/

oce1 ----> LAGG1 ----> bridge1 [ User NET xxx.xxx.162.0/24 ] <----- vnet0 [plex jail]
igb1 --------/ \------------------------------------ vnet1 [Linux VM in bhyve]

After a reboot of the system, vnet0 to the Jail and vnet1 to the VM basically keep coming up on bridge0. This is even although they are supposed to be bridged to bridge1 and the management UI shows this. This also happens if I reboot the Linux VM - it gets wrongly reconnected to bridge0.

I can log in manually and fix it of course use ifconfig lagg0 deletem vnet0 && ifconfig lagg1 addm vnet0 which is easy enough but a bit of a PITA. I suppose I could automatically get around it with a cron job too.

I've seen there's a few posts about this kind of issue previously but I didn't yet find a workable (for me) solution and wondered if this has been resolved yet, maybe in TrueNAS 13 or TrueNAS Scale. So anyway..

Thanks in advance,
Aidan

 

[sretalla]

It's probably worth reading this post carefully:https://www.truenas.com/community/threads/multiple-interfaces-in-jails.92364/post-639747

I saw something similar and maybe more related to your post, but can't seem to find it with the search function, so this will have to be the first clue for now. Note the mention of the vnet_default_interface setting to none (as that's likely to be the one that grabs the NIC with bridge0).

Also note you could do things backwards to simplify your life a lot... run your admin net on bridge1 and all other stuff on bridge0, which would not need any config.

 

[ad31677]

Thanks for the reply, Sretalla. I did see that post you referred to.

I haven't actually tried the bit about moving the address assignment from the lagg to the bridge (as I would have done this kind of thing manually in Linux) or the bit about setting the vnet default interface to "none". I will give those ago tonight.

I could indeed do things backwards but that would break things for the Jail on lagg0/bridge0 (admin). That'll be the jail I didn't mention for the sake of brevity.

 

[Patrick M. Hausen]

Jails - assigning separate NIC

Hello, I've been struggling for a couple of days to assign NIC to a jail using GUI (11.3). I know that using: sudo iocage create -r 11.3-RELEASE ip4_addr="igb2|172.16.1.3/26" defaultrouter="172.16.1.1" -n test2 I'll create jail having igb2 assigned to it; but once I just save its settings in...

www.truenas.com

I write "MUST" like in RFC language, because networking will break if you have IP addresses on bridge member interfaces. Bridge members are supposed to be strictly layer 2. Sometimes you need to specify "up" in the options field, so I made that a habit to be safe.

 

[VR46]

Check the vnet_default_interface of your jail. I had a somewhat similar issue where lagg1 kept getting added to my bridge0 because it's where the default gateway is. Changing that setting from auto to none solved my issue. You may be able to set to bridge1.

 

[ad31677]

Hi, I haven't had the time to work on this issue recently due to heavy work load. I did try setting the jail interface to none but then TrueNAS 12 wouldn;t stsart at all. This needs more investigation on my part, really.

Is there something I need to do by a channel other than the UI?

 

[Patrick M. Hausen]

Everything can be set in the UI. It helps to know one's way around iocage to diagnose things. Which setting precisely do you mean by "jail interface"? You need to set vnet_default_interface to "none", nothing else.

 

[ad31677]

Everything can be set in the UI. It helps to know one's way around iocage to diagnose things. Which setting precisely do you mean by "jail interface"? You need to set vnet_default_interface to "none", nothing else.

That's what I was referring to, sorry, just couldn't remember the field name on the UI when I was writing it last night.

 

[Patrick M. Hausen]

OK, let me repeat that - you set vnet_default_interface to "none" for one jail and your entire TrueNAS host refuses to start? In which way? What are all other settings of the jail? You can get a complete list with iocage get all <jailname>.