Second Jail looses connectivity

[daretar]

Hello :)

I have an issue where every jail (except the first jail I ever created) looses connection to anything besides the truenas (core) server itself. For me it looks like it looses connection to the gateway, but I might be wrong with that assumption.

My configuration is:
NIC: Intel X540-T2 -> bonded to failover
Ifconfig:

Code:

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether a0:36:9f:21:59:d8
    media: Ethernet autoselect
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
ix1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether a0:36:9f:21:59:d8
    hwaddr a0:36:9f:21:59:da
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Lagg for failover
    options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether a0:36:9f:21:59:d8
    inet 192.168.178.15 netmask 0xffffff00 broadcast 192.168.178.255
    laggproto failover lagghash l2,l3,l4
    laggport: ix0 flags=1<MASTER>
    laggport: ix1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:cf:1e:83:03:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: dev as nic: epair0b
    options=8<VLAN_MTU>
    ether 26:4b:fe:6f:45:b3
    hwaddr 02:9d:78:23:6f:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: jenkins as nic: epair0b
    options=8<VLAN_MTU>
    ether 26:4b:fe:04:84:0a
    hwaddr 02:2c:5f:c8:26:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

iocage list

Code:

+-----+---------+-------+--------------+------+
| JID |  NAME   | STATE |   RELEASE    | IP4  |
+=====+=========+=======+==============+======+
| 1   | dev     | up    | 12.2-RELEASE | DHCP |
+-----+---------+-------+--------------+------+
| 2   | jenkins | up    | 12.2-RELEASE | DHCP | --> looses connectivity
+-----+---------+-------+--------------+------+

netstat -rn on dev (jail 1)

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 192.168.178.1 UGS epair0b
127.0.0.1 link#1 UH lo0
192.168.178.0/24 link#3 U epair0b
192.168.178.35 link#3 UHS lo0

Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#1 U lo0
fe80::1%lo0 link#1 UHS lo0
ff02::/16 ::1 UGRS lo0

netstat -rn on jenkins (jail 2) -> no ping to anything but TrueNAS

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.178.1      UGS     epair0b
127.0.0.1          link#1             UH          lo0
192.168.178.0/24   link#3             U       epair0b
192.168.178.50     link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

For testing I also activated DHCP (no change). I also created a third jail for testing. That jail also looses connectivity after 30-60 minutes. Sometimes faster sometimes slower. I created the jails exactly the same way.

Is there anything I am missing?

 

[daretar]

I guess I did a bad job explaining what happens :D
So after 30-50 Minutes:

1. I still can ping the jail from inside my network
2. From inside the jail (iocage console) I can't ping google
3. From inside the jail I can't ping the gateway/dns etc.
4. From inside the jail I can ping TrueNAS

All this only happens on jails that are not the first jail I ever created. The order of which I start the jails doesn't seem to matter. I don't have any IP conflicts.
I use the current release TrueNAS CORE 12.0-U2