Jails useless after upgrade

[kazooless]

Hello,

Last year I upgraded from Truenas from 11.x to 12.x. After doing so, my 4 jails would be unable to access the network after being booted for a few minutes. My network devices would also be unable to access the jails. But in the first few minutes, the connectivity is there and working.

I had a lot of changes in my life last year, so I just put the server in the closet for a while. I took it out this week and upgraded to 13.x. Then I upgraded all of the jails to 13.x. I get the same problem. I created a brand new jail, let it sit, and sure enough, it loses network connectivity too. The main console is still accessible via WebUI & SSH. When I console in to a jail, the jail is able to ping the Internet.

When I check the arp table on my Mac computer, I do get a weird entry for the jails:

Code:

? (10.0.1.214) at (incomplete) on en0 ifscope [ethernet]

The arp tables in the jails look fine. I only have Apple devices here, except for the router, which is a TP-Link. I have iPhones, iPads, HomePod minis, An Apple TV & MacBooks. I've search for hours on this problem with little to show. I did, however, find one person who found a bug with arp & iocage vnets, but I can't imagine I'm the only other person with an Apple network & TrueNAS jails. If it truly is some sort of bug preventing any Apple device from communicating with jails, then I would expect a lot of chatter about it here in the forums.

Any ideas?

 

[Patrick M. Hausen]

Do you have hardware offloading disabled on your physical network interface?

 

[kazooless]

It was not disabled. I disabled it now but it doesn't seem to be making any difference.

 

[kazooless]

I permanently added the arp entries to my Mac laptop. As soon as I did, my Mac could ping again. But I can't do this on an AppleTV as far as I know. It makes me think this thread is still relevant: https://www.truenas.com/community/threads/arp-replies-loss-in-vnet.77027/

Is anyone with a Mac, the latest version of TrueNAS and jails, running their jails with no problems? I'm afraid of wasting time installing from scratch to find the problem still exists. Plus the prospect of losing all my historical data is not appealing to me.

 

[Patrick M. Hausen]

Yes, I run dozens of jails on TrueNAS without any problem. Do you have a statically configured bridge interface? Did you move the IP configuration from the physical interface to the bridge? Did you in any way mess with the TrueNAS firewall manually?

Best we start with the output of` ifconfig on the TrueNAS host and the output of iocage get all <jailname> for one of the jails.

BTW: did you reboot after disabling hardware offloading? You must reboot after any changes in the NAS network settings to reconnect the jails.

 

[kazooless]

Thank you for your reply. I'm really not sure if my bridge interface is statically configured. It was years ago on a much earlier version of FreeNAS I was running when I created these. I don't remember purposely moving any IP configuration. Haven't touched a firewall but I don't think it is running one.

From the TrueNAS console:

Code:

root@freenas:/etc # ifconfig

igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: Primary-LAN

    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>

    ether 0c:c4:7a:9a:a3:b6

    inet 10.0.1.200 netmask 0xffffff00 broadcast 10.0.1.255

    media: Ethernet autoselect (1000baseT <full-duplex>)

    status: active

    nd6 options=9<PERFORMNUD,IFDISABLED>

igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500

    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>

    ether 0c:c4:7a:9a:a3:b7

    media: Ethernet autoselect

    status: no carrier

    nd6 options=9<PERFORMNUD,IFDISABLED>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384

    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>

    inet6 ::1 prefixlen 128

    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3

    inet 127.0.0.1 netmask 0xff000000

    groups: lo

    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pflog0: flags=0<> metric 0 mtu 33160

    groups: pflog

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

    ether 58:9c:fc:10:07:61

    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15

    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200

    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 10 priority 128 path cost 2000

    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 9 priority 128 path cost 2000

    member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 8 priority 128 path cost 2000

    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 7 priority 128 path cost 2000

    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 6 priority 128 path cost 2000

    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

            ifmaxaddr 0 port 1 priority 128 path cost 20000

    groups: bridge

    nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: associated with jail: couchpotato as nic: epair0b

    options=8<VLAN_MTU>

    ether 02:ff:60:1c:24:41

    hwaddr 02:7d:cf:40:9b:0a

    groups: epair

    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

    status: active

    nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: associated with jail: new-plex as nic: epair0b

    options=8<VLAN_MTU>

    ether 02:ff:60:0d:6c:18

    hwaddr 02:4d:20:68:c0:0a

    groups: epair

    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

    status: active

    nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: associated with jail: plex as nic: epair0b

    options=8<VLAN_MTU>

    ether 02:ff:60:14:fa:09

    hwaddr 02:85:b5:b9:27:0a

    groups: epair

    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

    status: active

    nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.4: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: associated with jail: sabnzbd as nic: epair0b

    options=8<VLAN_MTU>

    ether 02:ff:60:ef:4d:38

    hwaddr 02:42:d9:5b:1f:0a

    groups: epair

    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

    status: active

    nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.6: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

    description: associated with jail: sickrage as nic: epair0b

    options=8<VLAN_MTU>

    ether 02:ff:60:c2:11:83

    hwaddr 02:3c:a8:e9:0d:0a

    groups: epair

    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

    status: active


    nd6 options=9<PERFORMNUD,IFDISABLED>

And one of the jails:

Code:

root@freenas:/etc # iocage get all plex
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:1
bpf:0
children_max:0
cloned_release:11.2-RELEASE-p15
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:10.0.1.1
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:plex
host_hostuuid:plex
host_time:1
hostid:d09c4e6a-b0c3-11e4-b46b-00224d83cf02
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:10.0.1.214/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:0
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/plex/data
jail_zfs_mountpoint:none
last_started:2022-09-07 03:02:43
localhost_ip:none
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:13.1-RELEASE-p2
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:02ff6014fa09,02ff6014fa0a
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

I did reboot after disabling. What's weird is when I go back in to edit, the checkmark is still there but without choosing anything, you're prompted as if you are disabling it again.

 

[Patrick M. Hausen]

OK, please do the following:

1. disable boot for all jails and for all VMs (if you have any)
2. reboot your system - this gets rid of the bridge0 interface
3. manually create a bridge interface named "bridge0", put igb0 as member
4. remove the IP address from igb0 but put "up" into the options field
5. put the IP address on the bridge0 interface
6. test and save your network configuration - you might want to put a tine value longer than 60 seconds into that timeout field, e.g. 300
7. you should be able to reconnect after a short disconnection

For each jail:

1. iocage set vnet_default_interface=none <jailname>
2. iocage set interfaces=vnet0:bridge0 <jailname>

Check with ifconfig on the host that the IP address is indeed on the bridge0 interface, then start your jails.

HTH,
Patrick

 

[kazooless]

Thanks so much for your time. I really appreciate it.

I followed your instructions. Unfortunately, it didn't fix it. Instead, now the primary interface also has the problem (it didn't before). Again, if I add the Mac addresses to the arp table manually, then I can access them again. Any other ideas?

Here is the current output, in case I missed something:

Code:

root@freenas:~ # ifconfig
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Primary-LAN
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 0c:c4:7a:9a:a3:b6
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:9a:a3:b7
    media: Ethernet autoselect
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:07:61
    inet 10.0.1.200 netmask 0xffffff00 broadcast 10.0.1.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: couchpotato as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:1c:24:41
    hwaddr 02:7d:cf:40:9b:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:14:fa:09
    hwaddr 02:4d:20:68:c0:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sabnzbd as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:ef:4d:38
    hwaddr 02:85:b5:b9:27:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.4: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sickrage as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:c2:11:83
    hwaddr 02:42:d9:5b:1f:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
root@freenas:~ #

Code:

root@freenas:~ # iocage get all plex
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:1
bpf:0
children_max:0
cloned_release:11.2-RELEASE-p15
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:10.0.1.1
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:plex
host_hostuuid:plex
host_time:1
hostid:d09c4e6a-b0c3-11e4-b46b-00224d83cf02
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:10.0.1.214/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:0
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/plex/data
jail_zfs_mountpoint:none
last_started:2022-09-08 02:00:53
localhost_ip:none
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:13.1-RELEASE-p2
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:02ff6014fa09,02ff6014fa0a
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:none
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off
root@freenas:~ # 

 

[Patrick M. Hausen]

Now I start to suspect your switch. Does it have some "anti spoofing" feature?

 

[kazooless]

I thought I replied to this. I am wondering the same too, though the other posts I mentioned do seem to describe this problem.

ARP replies loss in VNET

Apple devices (and possibly other vendors) lose connection to VNET jails because the jail ARP replies are not going through. These replies correspond to ARP requests that are padded with more than 18 zeros, like macOS or iOS are sending (looks like VNET has a bug that corrupts ARP replies when...

www.truenas.com

The Apple TV seems to be able to connect to Plex without issue since the last change. The iPhone can't ping it. The Mac can still ping as long as the static arp entry is there.

I'm going to try a different router to see if it makes any difference. I'm using a TP Archer AX50 and see nothing about anti-spoofing on the local network. It does have the ability to bind an IP to a MAC but that hasn't made any difference.

 

[kazooless]

@Patrick M. Hausen, you were right. I switched out my Archer for the provided Frontier router and the problem is gone. The WiFi signal in the house is nowhere near as good, but at least I can access my jails, LOL.

Thanks for the help. Is there a way to mark this as SOLVED?