Jails not accessible locally after reboot

[PeterSM]

I have an issue where jails are not accessible after a reboot. I have been running jails successfully since Frreenas 9, I have SMB shares, Jails and VMs all running on the same box that recently upgraded to Truenas Core TrueNAS-13.0-U4. Sometimes when I reboot SMB and VMs are running and accessible but all the jails which are up are not accessible. They are marked as up in the GUI. Restarting the jail has no effect the only solution is to reboot the system. Recently I had to reboot 3 times before the jails were accessible. This has happened on earlier releases so is not related to the recent update. Where do I start to look for a solution?
NIC is Intel I210.
Any thoughts are appreciated!

 

[sretalla]

Having a look at ifconfig when things are working and when not would probably be helpful... I suspect some complications coming from bridges somewhere.

 

[PeterSM]

Thanks for the response.
It is working now, so I have saved the ifconfig output. I will run ifconfig again next time I have an issue and compare the two.

 

[Dopamin3]

This exact same thing happens to me from time to time, and restarting the server only fixes it sometime. I found the root cause for me, and it's that bridge0 and bridge1 randomly swap.

Take a look at ifconfig on your TrueNAS Core host. Take note of bridge0 and bridge1 to see which one has the right associated interfaces.

For each of your jails do:
iocage get interfaces *jailname*
it will return a response like vnet0:bridge0
iocage stop *jailname*
iocage set interfaces=vnet0:bridge1 *jailname*
iocage start *jailname*

 

[PeterSM]

Thanks Dopamin3. Will using iocage to set the vnet:bridge be permanent or will I have to reset it after each boot?

 

[Dopamin3]

Thanks Dopamin3. Will using iodate to set the vnet:bridge be permanent or will I have to reset it after each boot?

It does set it permanently, however in my experience my TrueNAS Core host randomly swaps bridge0 and bridge1 quite often after reboots. So at some point you might have to change it from bridge1 back to bridge0 or vice versa.

Honestly I think this is an issue with 13.0-U4 (or a recent release) because I never ever had this issue when it was still "FreeNAS" instead of "TrueNAS Core"

 

[sretalla]

my TrueNAS Core host randomly swaps bridge0 and bridge1 quite often after reboots.

Are you letting that happen automatically, or have you set the bridges up in Network | Interfaces (in which case, swapping around would be a real problem as you've specified what should be what)?

 

[Dopamin3]

Are you letting that happen automatically, or have you set the bridges up in Network | Interfaces (in which case, swapping around would be a real problem as you've specified what should be what)?

I never set the bridge interfaces manually. Here is a screenshot from Network / Interfaces in the TrueNAS Core 13.0-U4 GUI.

And here is an ifconfig output from the host:

Code:

root@RyzenNAS:~ # ifconfig
igb0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 70:85:c2:5a:9f:42
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 00:07:43:5b:95:20
        media: Ethernet none
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
cxl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 00:07:43:5b:95:28
        inet 10.13.37.10 netmask 0xffffff00 broadcast 10.13.37.255
        media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:2a:14
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: vnet0.7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000
        member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: vnet2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: vnet1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000
        member: cxl1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:04:51:ff
        hwaddr 58:9c:fc:10:f9:4e
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 2212
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:d1
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:55:27:78
        hwaddr 58:9c:fc:10:08:37
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 2282
vnet2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:5f:ac:b3
        hwaddr 58:9c:fc:00:2b:00
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 2350
vnet0.5: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: *** as nic: epair0b
        options=8<VLAN_MTU>
        ether 70:85:c2:31:41:ea
        hwaddr 02:4e:8b:8d:84:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.6: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: *** as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:33:ee:98
        hwaddr 02:a3:03:67:ae:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.7: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: *** as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:07:43:c8:bc:a3
        hwaddr 02:5f:2d:d5:d9:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.8: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: *** as nic: epair0b
        options=8<VLAN_MTU>
        ether 70:85:c2:8b:86:16
        hwaddr 02:71:38:8e:8b:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>

 

[sretalla]

OK, so it's simple to fix your problem... stop all jails/VMs and manually define bridge0 and bridge1. Like that you'll get the same result after every reboot.

If you need broadcast to work properly, you can even use the opportunity to move your TrueNAS IP (which should be static) to the bridge at the same time.