FreeNAS-11.3-U4 jails networking stops working after a while

wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
Hey guys,

I have a strange problem with my freenas server lately and I really hope you can help me. Here ist the thing: since upgrading zo 11.3-U3 and above all of my jails become unresponsive networkingwise after some time (several hours after boot). This can be recovered by a reboot of my whole server which is quite annoying.

All my jails have static IPs in the 192.168.4.0/24 subnet with 192.168.4.10 as their default gateway (pfsense box). All other clients in this subnet don't have any problems and even my freenast host system does not have any problem with its static IP 192.168.4.87. My two linux VMs on the freenas box aren't affected either. They continue to work just fine. Only my jails become "offline" after some hours runtime of my freenas server.

I already tried (without success):
  • turning off "hardware offloading" for all of my nics including 'ix0' (which is the nic my jails use in 'bridge1')
  • using DHCP and letting pfsense assign the static IPs
  • useing a different NIC as base for the vnet-Interfaces of my jails
Can you please help me? If there is anything wrong I am not capeable of recognizing it anymore :-(
Thank you very much in advance!

Here are my hardware specs:
CPU: Ryzen 3700x
RAM: 32GB ECC DDR4 Samsung (M391A2K43BB1-CTD)
Mainboard: Asrock X470D4U
NIC: Intel X550-T2 10GBE
HBA: LSI SAS 9207-8i in IT-Mode

Jails overview (ignore "Test", "build" and "folding_at_home" -> unimportant testing jails):
Code:
+-----+-----------------+-------+--------------+-----------------+
| JID |      NAME       | STATE |   RELEASE    |       IP4       |
+=====+=================+=======+==============+=================+
| -   | Test            | down  | 11.3-RELEASE | 192.168.4.79/24 |
+-----+-----------------+-------+--------------+-----------------+
| -   | build           | down  | 11.2-RELEASE | DHCP            |
+-----+-----------------+-------+--------------+-----------------+
| 1   | emby            | up    | 11.2-RELEASE | 192.168.4.82    |
+-----+-----------------+-------+--------------+-----------------+
| -   | folding_at_home | down  | 11.3-RELEASE | 192.168.4.84    |
+-----+-----------------+-------+--------------+-----------------+
| 2   | gitlab          | up    | 11.2-RELEASE | 192.168.4.83    |
+-----+-----------------+-------+--------------+-----------------+
| 3   | heimdall        | up    | 11.3-RELEASE | 192.168.4.81    |
+-----+-----------------+-------+--------------+-----------------+
| 4   | plex            | up    | 11.2-RELEASE | 192.168.4.85    |
+-----+-----------------+-------+--------------+-----------------+
| 5   | subsonic        | up    | 11.2-RELEASE | 192.168.4.86/24 |
+-----+-----------------+-------+--------------+-----------------+


ifconfig output:
Code:
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: pfsense LAN
    options=a400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether a0:36:9f:3f:df:1c
    hwaddr a0:36:9f:3f:df:1c
    inet 192.168.4.87 netmask 0xffffff00 broadcast 192.168.4.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Grottenolm LAN
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:3f:df:1e
    hwaddr a0:36:9f:3f:df:1e
    inet 192.168.7.77 netmask 0xffffff00 broadcast 192.168.7.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
    status: active
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Cable LAN
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether a8:a1:59:2b:39:6e
    hwaddr a8:a1:59:2b:39:6e
    inet 192.168.2.77 netmask 0xffffff00 broadcast 192.168.2.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a8:a1:59:2b:54:ce
    hwaddr a8:a1:59:2b:54:ce
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:4f:b1:06:61:3e
    hwaddr 02:4f:b1:06:61:3e
    nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to Xubuntu
    options=80000<LINKSTATE>
    ether 00:bd:2c:14:f9:00
    hwaddr 00:bd:2c:14:f9:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2081
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:12:11:83:a3:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 20000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to Xubuntu
    options=80000<LINKSTATE>
    ether 00:bd:ad:14:f9:01
    hwaddr 00:bd:ad:14:f9:01
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2081
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:12:11:83:a3:01
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000
    member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: emby as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:79:aa:5f
    hwaddr 02:37:90:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to UbuntuServerVM
    options=80000<LINKSTATE>
    ether 00:bd:27:24:f9:02
    hwaddr 00:bd:27:24:f9:02
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2583
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: gitlab as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:b1:cf:58
    hwaddr 02:37:90:00:0d:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: heimdall as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:82:0e:41
    hwaddr 02:37:90:00:0e:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:b1:cf:58
    hwaddr 02:37:90:00:0f:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: subsonic as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:0d:0f:d3
    hwaddr 02:37:90:00:10:0a
    inet6 fe80::ff:60ff:fe0d:fd3%vnet0.5 prefixlen 64 scopeid 0x10
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
 
Last edited:
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
I suspect this is a thermal issue; however, it's not clear given the info you've provided what's going on. Is there anything logged in /var/log/middlewared.log on what's going on with those jails when they lock up?
 
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
thanks for your suggestions. no there is nothing in the /var/log/middlewared.log about jails. right now my jails are locked up again for about 164 minutes. Because I am using the monitoring server check-mk I am able to tell the time the jails go offline quite exactly (+/-5min). But I could not find any relevant logentries for this time. Is there anywhere such a thing like a networking or jail log?

Update:
I suspect this is a thermal issue; however, it's not clear given the info you've provided what's going on
Click to expand...
although I can understand this suggestion, I doubt it is the case since my NIC and mainboard have good airflow from three 120mm noctua FANs in my chassis.
Is there anything else than /var/log/middlewared.log I could check?
 
Last edited:
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
All the relevant logs are in /var/log:
  • console.log: console messages
  • daemon.log: messages from system processes
  • debug.log
  • messages
 
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
Hm no sry in the logs you mentioned was nothing relevant for this problem. I guess it must have something to do with vnet and/or bridge-Interface settings since only the bridged jail vnet-interfaces are affected by this problem while the tap-interfaces of my VMs continue to work even if alle the jails become offline after a while. Should I file a bugreport in jira here?
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Can you output ifconfig -a?
 
hescominsoon

hescominsoon

Patron
Joined
Jul 27, 2016
Messages
456
sounds like a hardware issue...maybe. what nics are you using?
 
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
hescominsoon said:
sounds like a hardware issue...maybe. what nics are you using?
Click to expand...
I use one of the onboard 1Gbit intel nics of my mainboard (igb0) and both of the 10Gbit Intel X550-T2 (ix0 & ix1). On one of the 10gbit nics are my jails and my VMs. All NICs are configured with static IPs. If it was an hardware issue my VMs and my other services like samba shares would be affected as well won't they? Also only the vnet interfaces of my jails are offline, not the physical ones :(

What baffles me is that the vnet interfaces work fine for several hours after a reboot of my server and then suddenly stop working. Without any explanation in the logs.

Samuel Tai said:
Can you output ifconfig -a?
Click to expand...
Code:
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: pfsense LAN
    options=a400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether a0:36:9f:3f:df:1c
    hwaddr a0:36:9f:3f:df:1c
    inet 192.168.4.87 netmask 0xffffff00 broadcast 192.168.4.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Grottenolm LAN
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:3f:df:1e
    hwaddr a0:36:9f:3f:df:1e
    inet 192.168.7.77 netmask 0xffffff00 broadcast 192.168.7.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
    status: active
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Cable LAN
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether a8:a1:59:2b:39:6e
    hwaddr a8:a1:59:2b:39:6e
    inet 192.168.2.77 netmask 0xffffff00 broadcast 192.168.2.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a8:a1:59:2b:54:ce
    hwaddr a8:a1:59:2b:54:ce
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:4f:b1:06:61:3e
    hwaddr 02:4f:b1:06:61:3e
    nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to Xubuntu
    options=80000<LINKSTATE>
    ether 00:bd:97:0c:f9:00
    hwaddr 00:bd:97:0c:f9:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2085
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:12:11:83:a3:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 20000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to Xubuntu
    options=80000<LINKSTATE>
    ether 00:bd:0d:0d:f9:01
    hwaddr 00:bd:0d:0d:f9:01
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2085
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:12:11:83:a3:01
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 18 priority 128 path cost 2000
    member: vnet0.9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vnet0.8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vnet0.7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to UbuntuServerVM
    options=80000<LINKSTATE>
    ether 00:bd:92:1c:f9:02
    hwaddr 00:bd:92:1c:f9:02
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2578
vnet0.2: flags=8d43<UP,BROADCAST,RUNNING,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: gitlab as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:b1:cf:58
    hwaddr 02:37:90:00:0d:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: subsonic as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:0d:0f:d3
    hwaddr 02:37:90:00:10:0a
    inet6 fe80::ff:60ff:fe0d:fd3%vnet0.5 prefixlen 64 scopeid 0x10
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:b1:cf:58
    hwaddr 02:37:90:00:0f:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.7: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: emby as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:79:aa:5f
    hwaddr 02:37:90:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.8: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: heimdall as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:82:0e:41
    hwaddr 02:37:90:00:0e:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.9: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Test as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:20:e9:66
    hwaddr 02:37:90:00:11:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: build as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:d4:9c:9f
    hwaddr 02:37:90:00:12:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
 
Last edited:
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
I finally saw something in the logs:

Code:
Jul 29 12:23:39 fw-server-freenas snmpd[1515]: Name of an interface changed. Such interfaces will keep its old name in IF-MIB.
Jul 29 12:23:40 fw-server-freenas syslog-ng[7429]: syslog-ng starting up; version='3.23.1'
Jul 29 12:23:40 fw-server-freenas kernel: pid 1174 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)


This log entries appeard just after I rebooted the server a few minuetes ago to get my jails working again...
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
You may want to reverse the assignment of your bridges, and make bridge0 the bridge for all your jails and VMs, and bridge1 for everything else. I suspect iocage is unhappy being forced to use bridge1.
 
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
Ok interessting idea. I'll try that! These bridge-Interfaces were created automatically by freenas. Can you tell me how to change the assignment of physical interfaces to the bridges so that these changes will survive a reboot?
 
Last edited:
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
For the jails, it's pretty simple, just change vnet_default_interface to the NIC (igb0, in your case, which joined bridge0) on the bridge you want. For the VMs, you'll have to change the nic_attach defined in the NIC interface device beneath the VM.
 
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
But igb0 is just a 1gbit interface in a different subnet than ix0. I thought I could change the assignedment of interfaces to the bridges themself. For example: instead of having igb0 on bridge0 I'd like to have ix0 on bridge0. This way I could sill use my 10Gig NIC for all my jails and freenas itself but just on another bridge. Is that possible?
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Yes, but you may need to physically create the bridges under Network->Interfaces.
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
If you want to continue to rely on the auto-assignment, I saw tap0 for Xubuntu on bridge0 (due to its nic_attach being igb0), and everything else on bridge1. If you change this to ix0, then it's likely everything will be created on bridge0 on ix0 at the next boot.
 
Last edited:
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
Ok thanks for your advice. I just reorganized my whole networking. I deleted my complete network config, recreated every interface and switched the onboard interface from igb1 to igb0. I also created the two bridge interfaces bridge0 and bridge1 in the freenas webinterface. Then I assigned ix0 to bridge0 and igb0 to bridge1 and changed all jails to vnet0:bridge0.

Whats interessting here is that I could no longer attach my jail or VM virtual NICs to ix0 directly instead I had to attach them to the appropriate bridge-interface bridge0. I wonder if this led to the problem in the first place :rolleyes:

So far everything seems to work, but we'll see if this fixed it in a couple of hours runtime.

*Update*

Unfortunately all jails went offline again when a cloud sync task of mine started... But I saw that FreeNAS 11.3-U4.1 is available now. So I installed that and since the reboot my jails are online again. I noticed that the "epair*"-Interface messages printed on the freenas console when a jail comes up changed slightly: Before the update to 11.3-U4.1 they were all called "epair0a/epair0b". After the update they are called "epair1a, epair1b, epair2a, etc.".
So... I am just waiting again if the problem appears.

*Update#2*

Here we go again - no change :-( My jails went offline 2hours ago again. Ok then. I'll try to use a different PCIe Port for my Intel Nic (ix0 & ix1) then. If that does not help either I guess I have to file a bug report and wait for TrueNAS 12.
 
Last edited:
wannabe_nerd

wannabe_nerd

Dabbler
Joined
Jul 25, 2020
Messages
13
Meanwhile I migrated most of my jails (heimdall, emby, plex) to docker containers inside my Ubuntu Server VM. In fact just my gitlab and my subsonic jail are currently active. And guess what? They are running stable for more than 3 days now! So what's going on here?

Current jail situation:
Code:
+-----+----------+-------+--------------+-----------------+
| JID |   NAME   | STATE |   RELEASE    |       IP4       |
+=====+==========+=======+==============+=================+
| -   | build    | down  | 11.2-RELEASE | DHCP            |
+-----+----------+-------+--------------+-----------------+
| -   | emby     | down  | 11.2-RELEASE | 192.168.4.82/24 |
+-----+----------+-------+--------------+-----------------+
| 5   | gitlab   | up    | 11.2-RELEASE | 192.168.4.83/24 |
+-----+----------+-------+--------------+-----------------+
| -   | heimdall | down  | 11.3-RELEASE | 192.168.4.81/24 |
+-----+----------+-------+--------------+-----------------+
| -   | plex     | down  | 11.2-RELEASE | 192.168.4.85/24 |
+-----+----------+-------+--------------+-----------------+
| 8   | subsonic | up    | 11.2-RELEASE | 192.168.4.86    |
+-----+----------+-------+--------------+-----------------+


Maybe it has something to do with the fact, that most of my jails are based on the release 11.2 and only my most recent jail (heimdall) on 11.3? I can't see any other difference.
 
hescominsoon

hescominsoon

Patron
Joined
Jul 27, 2016
Messages
456
wannabe_nerd said:
Meanwhile I migrated most of my jails (heimdall, emby, plex) to docker containers inside my Ubuntu Server VM. In fact just my gitlab and my subsonic jail are currently active. And guess what? They are running stable for more than 3 days now! So what's going on here?

Current jail situation:
Code:
+-----+----------+-------+--------------+-----------------+
| JID |   NAME   | STATE |   RELEASE    |       IP4       |
+=====+==========+=======+==============+=================+
| -   | build    | down  | 11.2-RELEASE | DHCP            |
+-----+----------+-------+--------------+-----------------+
| -   | emby     | down  | 11.2-RELEASE | 192.168.4.82/24 |
+-----+----------+-------+--------------+-----------------+
| 5   | gitlab   | up    | 11.2-RELEASE | 192.168.4.83/24 |
+-----+----------+-------+--------------+-----------------+
| -   | heimdall | down  | 11.3-RELEASE | 192.168.4.81/24 |
+-----+----------+-------+--------------+-----------------+
| -   | plex     | down  | 11.2-RELEASE | 192.168.4.85/24 |
+-----+----------+-------+--------------+-----------------+
| 8   | subsonic | up    | 11.2-RELEASE | 192.168.4.86    |
+-----+----------+-------+--------------+-----------------+


Maybe it has something to do with the fact, that most of my jails are based on the release 11.2 and only my most recent jail (heimdall) on 11.3? I can't see any other difference.
Click to expand...
aha..that could be it...have you tried updating your jails to the 11.3 codebase?
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "FreeNAS-11.3-U4 jails networking stops working after a while"

Similar threads

N
Help getting 10 gig Peer to Peer working
Replies
16
Views
7K
Redcoat
Redcoat
phynias
upgrade to 11.2-U3 and now jails have no network access
Replies
2
Views
2K
phynias
phynias
kdragon75
  • Locked
Broken Jail Networking 11.1U6
Replies
2
Views
2K
kdragon75
kdragon75
rodofrn
  • Locked
Very slow transfer speeds.
Replies
17
Views
7K
Elliot Dierksen
Elliot Dierksen
C
  • Locked
Jail has no NIC after recent update
2
Replies
28
Views
9K
Crispin
C
Top