Jail Networking

[NugentS]

I set up a jail today. Release 12.2-Release-p6
I used VNET, and TrueNAS picked Berkley Packet Filter
I set the networking up as per the screenshot below

I then added allow raw_sockets so I could use ping and traceroute from the jail

When I start this jail - my TrueNAS drops off the network completely and I have to walk to the machine and switch it off. I cannot even ping the IPMI interface - so I am assuming that as the switches seem to go beserk (and are also unreachable) that TrueNAS is blasting crap out onto the network and bringing it down

What I am trying to achieve - and this is what I thought vnet was for is to have an independant IP stack in the jail so I could use different DNS/a different default gateway in the jail

First Question: Have I got the concept correct
Second Question: If "yes" to above any idea what the hell is going wrong?

To clarify I have 5 NICs on the server

1. 10Gb Main Network, SAMBA Shares
2. 10Gb Storage Network - different switch, different network. Switch is attached to main network for management purposes
3. 1Gb Jail Switch on main network
4. 1Gb Unconnected
5. IPMI

Oh and I already have a jail on that nic that doesn't use vnet or Berkeley and works - but I cannot change the GW address - it just uses the TrueNAS one.

 

[zfs get]

Hi, I'd like to preface my post by saying that I don't know the subject matter well-enough to be able to offer a straight solution, but I'll do my best to help, at least some.

Now, what I do know for a fact is that you can create independent network stacks in FreeBSD jails. Each jail can have its own IPv4/6 localhost, Its own gateway, NAT, firewalls, network interfaces, IP addresses and so on, but (obviously) It's important that the jails work in concert with the host. A while back, I built a system of VNET jails inside a FreeBSD KVM, it was a concept build, to try out how well it would host (and separate) a number of different servers/network services, so It's certainly doable.

Before you go about wrestling with VNET jails, I'd suggest you disconnect and "export"/archive the non-VNET jail you already have running on the machine, (if you can), to give you a clear slate. Then I'd advise you check that the particular NIC you intend to associate with the jail works well with FreeBSD's network drivers. Once you're sure of that, I'd also consider enabling Disable Hardware Offloading under your desired NIC's settings in Network>Interfaces>Edit, but be sure to disable this feature down the road if you later grow convinced this is not the issue as it can limit performance. At the end, I'd also suggest you enable STP (Spanning Tree Protocol) on all your switches that support the feature.

When all that is done, go ahead and create a jail, just like before, just be sure to tick the box that allows it to have its own localhost ("assign_localhost". The newly created jail should be assigned an IP inside the scope of the host machine, It should also get a localhost different than the host machine, say 127.0.0.2. After the jail is created and up & running, edit its /etc/rc.conf to add a gateway different to the one on your host machine as well as the /etc/resolv.conf to add your custom DNS addresses. It's been about two years since I've worked with things like this, besides I never went about it on TrueNAS, so I'm a little fuzzy on the details but something along these lines should work. If this ends up failing or working badly, something tells me that, together with a different gateway and resolv.conf, you also might have to put the jail in its own, separate subnet, though I'm not entirely sure.

Good luck, I hope this was of some help!

I went and tried my settings on a recent TrueNAS machine of mine and although I don't have the setup necessary to really put things to the test, I did notice something: You should customize your network settings from the TrueNAS menu since editing the files I mentioned above manually, directly in the jail, doesn't seem to foster permanent changes. It is likely that TrueNAS's jail management software edits /etc/resolv.conf etc. on every jail startup to prevent conflict.
I succeeded in changing the DNS server used by the jail from my network's default by editing the "resolver" setting in the jail options. I don't have a second working gateway I could try (and I can't be bothered to set one up), but setting the appropriate ("IPv4 Default Route" or something like that) to something that's wrong but inside the correct network did fail to route outside, so that's promising. Setting the Default Route to an IP outside the subnet produced a VNET error, which is only reasonable. I also have hangups about the jail's localhost having the same IP as the host's localhost, but that might not be an issue. I do recall FreeBSD (iocage) giving the jail localhosts a different address in the localhost-reserved space.

 

[NugentS]

The main 10Gb NICs are Chelsio, the 1Gb are Intel - so that shouldn't be a problem.
Hardware Offloading is also disabled
STP is on on all switches

 

[sretalla]

What are the IP address details of the host?

 

[NugentS]

Nic 1: 192.168.38.32 /24
Nic 2: 172.16.16.10 /24
Nic 3: None - I have not assigned an IP in TrueNAS
Nic 4: None

 

[sretalla]

And which is the one you assigned as the VNET default interface? Nic 3?

 

[NugentS]

192.168.38.40/24

 

[zfs get]

So, If I'm reading you right, you went through what I suggested yet the problem still persists in the exact same way? Did you disable the other, non-VNET, jail?

Could you paste the output of "tail -n 100 /var/log/messages" after the reboot?

 

[NugentS]

Update: I now have two vnet jails that seem to co-exist. Not sure what I did / why as I am sure I have tried this stuff before

But I just tried to build a new jail, with the other two disabled - and have just come back from a walk to the server reset button.

The tail command just shows the server booting which I suspect isn't helpful. I do have a couple of things to try though

 

[NugentS]

Just tried to build nextcloud - mostly for amusement. Walked to the reset button.
I can manually create a jail now (vnet) and it works - I have no idea what is happening here, but this seems far from stable. I cannot use the plugins (at least the ones I have tried as they just crash the server)
I had no issues under 11.3 (I think it was) that I remember - not that I used plugins much.

 

[sretalla]

192.168.38.40/24

Which NIC?

 

[NugentS]

NIC 3 - the 1Gb Nic for Jails

 

[NugentS]

I am not currently at my house - so will not be able to experiment for a few days.
But I am correct that a vnet is an entirely separate network stack - like a separate PC and thus I can have a vnet on a second interface on the same network as the NAS primary interface.

 

[sretalla]

NIC 3 - the 1Gb Nic for Jails

OK, so that highlights problem #1 for me... you're trying to have 2 NICs in the same TrueNAS server joined to the same subnet, which is an absolute no according to @jgreco .

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

 

[NugentS]

Not sure I am. The Jail has the address - not the TrueNAS box. It 's my understanding that having a jail with a vnet is a valid configuration. TrueNAS has one IP address, the jail has the second
Somewhere I think @jgreco was asked this question and confirmed this was good. Can't find where and of course I am assuming I am doing it the right way.

The concept is that TN does NOT have two NIC's on the network. It actually only has one. TN does not have an IP on the second NIC

 

[jgreco]

Not sure I am. The Jail has the address - not the TrueNAS box. It 's my understanding that having a jail with a vnet is a valid configuration. TrueNAS has one IP address, the jail has the second
Somewhere I think @jgreco was asked this question and confirmed this was good. Can't find where and of course I am assuming I am doing it the right way.

The concept is that TN does NOT have two NIC's on the network. It actually only has one. TN does not have an IP on the second NIC

That should be fine. Someone else complained that TrueNAS wouldn't create the second network without assigning an IP address to TrueNAS; I don't know if that's true, because I've generally been able to specify "up" in the auxiliary ifconfig parameters and that was fine. If you are really forced to assign an IP, pick an unused RFC1918 network. i.e. if you are using 10.0.0.0/8 addresses, assign em1 as 192.168.2.2/24 or something like that.

 

[NugentS]

I have managed 2 jails on one nic with vnet and no IP on that TN interface.
Will be back home at the weekend to crash the server a few times