No functional networking in jails if VM is started first

[jyavenard]

So I virtualised a linux PC to run in my TrueNAS box after upgrading the latter to a supermicro H11SSL-NC motherboard with an AMD Epyc 7302.

I noticed after a reboot of TrueNAS that if I set the VM to autostart, then my jails wouldn't have networking active. Jails set to use DHCP wouldn't even start then as they can't retrieve an IP address.

If I manually start the VM however, once TrueNAS has booted, it's all fine.

Has anyone experienced something similar?

Something bhyve is doing to the ethernet bridged configuration seems to be conflicting with what iocage is doing.
Note that if the VM is started after the jails have been setup, I can stop/start jails just fine: no networking issue.

If I can't resolve this, is there a way to make the VM starts only once all the jails have started?

TIA

 

[garm]

Jail networking is a bag of skates as we say in Sweden.. I have separate bridges for jails and VM on separate VLANs just to make sure they behave. And only a few jails can handle DHCP. Most are on assigned IPs.

 

[sretalla]

What's your bridge setup?

 

[jyavenard]

What's your bridge setup?

Only whatever TrueNAS configured automatically for both the jail and the VM.

I didn't create any bridges

 

[sretalla]

OK, perhaps it will help to take control of that.

On core, the default bridge will be bridge0, so what you would want to do is stop all your jails, setting them not to start on boot (same for the VMs), then reboot.

Then in Network | Interfaces, create a new bridge called bridge0, adding your NIC as a member.

Before you test the changes, remove your IP address from the NIC (or take away DHCP), turn off hardware offload and add that IP to the Bridge (... preferably don't enable DHCP on the bridge... it will work, but don't).

Then test the changes and make sure to save them within the timeout assuming all went well.

if you now have a bridge with your TrueNAS IP address on it and a NIC with no IP, you can go in and configure the VM and jails to use that bridge rather than the NIC.

That should then work reliably.

Set all jails and the VMs to start on boot and reboot... hopefully all good from there.

 

[jyavenard]

Thanks. Will try as soon as possible.
I don't use DHCP for the primary network interface, but I do use IPv6 auto-configuration.

turn off hardware offload

is there such thing for the interfaces themselves?

That should then work reliably.

Any particular reasons why no first manually creating the bridge is causing an issue?
Should I lodge a bug ?

 

[sretalla]

is there such thing for the interfaces themselves?

For some NICs (usually Intel will have it), yes.

Any particular reasons why no first manually creating the bridge is causing an issue?

Nothing I can see from here... but I have no view on your system at all, so I'd just be guessing.

Should I lodge a bug ?

You can do that if you feel it's a bug.

 

[jyavenard]

For some NICs (usually Intel will have it), yes.

There's a big warning about reduction in performance. How much of a hit would it take? This box has been running for over a decade with just jails and the network not having any special configuration.
I think I will move the VM to use a different NIC instead, that may be simpler and will break less thing.

My kids have been complaining of sudden loss of internet network speed, which seem to coincide on when I started the VM last week. I wonder if it's causing some flooding somewhere that causes a overall network performance degradation.

 

[jyavenard]

if you now have a bridge with your TrueNAS IP address on it and a NIC with no IP, you can go in and configure the VM and jails to use that bridge rather than the NIC.

Ok. I got to follow those instructions.
however, I don't see an option to configure the jails to use the bridge created

it only shows "none", "auto" or "igb0" , it's igb1 I'd like

I was thinking.
Would it be possible to have both igb0 and igb1 be connected to the same LAN (no VLAN).
Have igb0 set to 192.168.10.x ; don't have any IP assigned to igb1 and member of bridge0
and have all jails / VMs use igb1 instead.

That way HW acceleration for the NAS itself is on, and only off for the VMs/Jails .

 

[sretalla]

Would it be possible to have both igb0 and igb1 be connected to the same LAN (no VLAN).
Have igb0 set to 192.168.10.x ; don't have any IP assigned to igb1 and member of bridge0
and have all jails / VMs use igb1 instead.

That way HW acceleration for the NAS itself is on, and only off for the VMs/Jails .

Yes, that can work.

however, I don't see an option to configure the jails to use the bridge created

it only shows "none", "auto" or "igb0" , it's igb1 I'd like

Set it to none, but set under the Network section at the bottom:

Interfaces:
vnet0:bridge0

 

[jyavenard]

ok sounds good.
So I have igb0 without any ip address assigned.
bridge0 with igb0 member, set to previous IP address
igb1 without any ip address

All jails set to use none as vnet_default_interface but in network properties "vnet0:bridge0"

VMs set to use igb1.

So far so good. On reboot all started, no connectivity issues anymore.
So there are now two bridges. The one I manually created bridge0 with igb0 and bridge1 which contains igb1

Thank you heaps for your help. Couldn't have done it without you.