Jail network loop

[PetrZ]

Hi, I am facing a problem, I thought it's probably jail bridge loop. It occurred after update of FreeNAS to 11.2-U1. I was running some jails on 11.1-U6 without any issue. After update, when I start any jail, it seems there is broadcast storm and whole segment is flooded. There are 4 GbE connected to same switch - 1x ILO, 2x onboard NIC as lagg for balancing, 1x PCI-E NIC. DHCP is used for PCI-E NIC and jails, lagg is static IP. I didn't found root cause yet. When I start FreeNAS with stopped jails, there is no problem. When I start jail, loop occur and only disconnecting PCI-E NIC cable or reboot helps. I just found this Multiple network interfaces on a single subnet topic, by @jgreco.

Any ideas before I will start dumping and analyzing traffic is welcomed.

Some good practices for using jails as DMZ as well. I'm considering adding another one NIC dedicated for DMZ. On Debian system with KVMs I have two bridges, one with LAN, one with DMZ and assign them to different VMs. Didn't try that with iocage yet, I am still noob in *BSD world. Do I understand it well, that I can't have 2 ports lagg for NAS LAN traffic in IP subnet A, ILO in IP subnet A, jails for internal use in IP subnet A and DMZ jails in IP subnet B? Do I need to use just one logical interface (lagg) per physical subnet, regardless IP subnet? So I can't have NIC dedicated to jail, but I have to use shared interface with rest of the system and if I need separate NIC for DMZ, I have to use separate physical subnet (other switch or VLAN)? ILO is probably other story, as it's actually different system, just sitting in same box, right?

 

[jgreco]

No. Your LAG interface is the layer immediately above the hardware interfaces.

It seems like you might be trying to put two networks in the same broadcast domain. This is really not a good thing to do. I'm picking that up from your rather mixed use of terms including "subnet."

Ethernet IP networks are networks. Each is supposed to be in a separate broadcast domain. That can be created using separate physical switches, or it can be virtual LANs (VLANs). Either way, each network should have an address range that all clients agree upon.

A common mistake is for people to connect all their interfaces to a single switch and then go "there, done" because they don't understand what they're doing. Hence documentation like my multiple network interfaces discussion. But an almost-equally common mistake is for them to put multiple NETWORKS on a single switch (without VLANs). This is also a no-no. Some people will argue "but it works." And it may, at least more or less, but it's fail-y for reasons that share some similarity to the multiple network interfaces discussion. Unfortunately it makes life miserable for purists like me who prefer to build networks correctly.

What might be happening to you, best I can guess, is that when you bring up your jail, the bridging setup somehow ends up creating a loop, which causes a broadcast storm. The bridging code does have the capability to do RSTP, but this would cause a port to shut down, which isn't what you want. Since I really don't use the FreeNAS jail facility, I am not that up on exactly what it's doing. Seems like a strong possibility though.

So, instead, can I suggest this:

Remove the IP assignment from the LAG.
Drop the PCI-E NIC for the time being. Probably not worth the effort.
On your switch, create one VLAN for each network you wish to support.
Make sure the switch is configured for LACP on the FreeNAS ports. In most cases this creates a new "virtual" interface on the switch that you need to use for the remaining configuration.
Modify that "virtual" switch interface to use tagged VLAN to the FreeNAS box, and make sure the networks you want FreeNAS to access are presented.
Now, on the FreeNAS box:
Make sure that the LAG is enabled with LACP.
Add a VLAN interface, using the LAG as the parent and the appropriate tag for your storage traffic. Then configure that interface with the IP that you had previously removed from the LAG. At this point things should seem to be "working" for the main NAS.
Add a second VLAN interface, again using the LAG as the parent, and the tag you assigned for your DMZ. You should then bridge this to your jail. Now that should work too.

If you get real ambitious you might be able to add your PCI-E NIC to the LAG mix, but this is probably a bad idea. I've found FreeBSD is twitchy about mixing ethernet device drivers in a LAG. Using this strategy, you may add as many VLAN's as you wish, and you can also add another ethernet interface or two to the LACP bundle without a problem.

 

[PetrZ]

Thank you very much for your suggestions, @jgreco.
I will try to set it exactly this way as soon as I'll reach the office. Just btw: Separate NIC for DMZ was considered for scenario if I will build physically separated subnet (dedicated switch) for DMZ latter. For some reason I feel physical separation better than just using VLANs, although I believe that VLAN implementation is also secure.

 

[jgreco]

Thank you very much for your suggestions, @jgreco.
I will try to set it exactly this way as soon as I'll reach the office. Just btw: Separate NIC for DMZ was considered for scenario if I will build physically separated subnet (dedicated switch) for DMZ latter. For some reason I feel physical separation better than just using VLANs, although I believe that VLAN implementation is also secure.

Yes, a separate switch would be fine, as would be a properly configured VLAN. If you are actually experiencing broadcast storms, however, something's amiss.

 

[Tarrant]

I'm in a similar boat, but my question is a bit different. 11.3 release, created a jail, and ended up with such a spectacular dhcp discovery broadcast storm (803K packets...) that I accomplished my first internal DDoS on my own network.

I'm aware of all the configuration failures on my part to result in such a thing, but the problem is FreeNAS is now in a boot loop. How can I start FreeNAS with the jail disabled despite having configured the jail to auto-start? Failing that, any other suggestions for recovery?

And my apologies in advance for what is likely a trivial question. But advice would be hugely appreciated.