Jails loosing connection when adding virtual interface

[alleus]

Hi there awesome TrueNAS community!

Yesterday, I inadvertently caused some downtime in a bunch of services I'm hosting in my homelab running TrueNAS. I was doing some preparations for upcoming network changes I'm planning (introducing some VLANs for different segments of my home network). I went into Network / Interfaces / Add and added a VLAN interface with name vlan90, Vlan tag 90 and Parent interface set to my primary interface used by TrueNAS itself and all jails (note that I hadn't yet configured the switch port connected to the interface). I tested out the interface changes, and I could still reach the server fine and confirmed the changes. Shortly after, I started receiving monitor alerts for basically all of my hosted services that are running in separate jails. I tried pinging the DHCP assigned IPs of my jails, and realised none of them were reachable. I went into the shell for some of the jails with iocage console and none of the jails seemed to have any connection to the outside. All jails still were running properly, and according to both iocage in TrueNAS and ifconfig inside the jails they still had DHCP assigned IP addresses as before. I promptly removed the newly created VLAN interface in TrueNAS and applied the change, but the jails were still not reachable. I attempted to renew IP addresses in the jails with service dhclient restart epair0b but it just froze and never received any signals from my DHCP server. I had to restart every jail one by one to get them to regain connectivity to the rest of the network.

What can I do to prevent something like this to happen again? I have some suspicions, but it's a bit scary to change things up causing downtime and extra work with manually restarting 23 jails and making sure they run properly...

My current setup is:

• TrueNAS CORE 12.0-U8.1 connected via a single interface to my network with the built in Intel 82576 interfaces on my old Dell server (yeah, I should update TrueNAS, but was half way over to SCALE when I realised the lack of a good substitution for FreeBSD Jails and haven't had time to dive back into this)
• TrueNAS has a static IP
• Jails are configured with DHCP Autoconfigure IPv4 and I'm assigning them IP addresses in a DHPC server separated from TrueNAS (the jails are assigned IP addresses on the same /24 network as both the TrueNAS static IP and the network gateway). Screenshot of sample configuration below.

One guess I have is that I could do some more specific settings on the jails (disabling Autoconfigure IPv4 and setting vent_default_interface as well as IPv4 Interface) to make the jail connections more resilient to VLAN changes on the TrueNAS host. Another, more wild, guess is that things are improved in TrueNAS 13.

 

[sretalla]

Don't expect to be doing any connecting with apps/jails/VMs while making interface-level changes to the NAS.

On a change, the network stack is torn down completely and built back up on the host, so there's no avoiding connectivity loss to dependent functions.

Some can re-connect automatically, others not after such a change.

The "safest and best" way to do it is:

Stop all your apps/jails/VMs

Make your networking changes

Change your apps/jails/VMs accordingly after changes are applied and start them again.