No internet access from jails, DNS and DHCP failure?

[AMomchilov]

Hey there!

My previously-working plugins broke at some point several months ago. I don't think it's update related, and I don't know of any other correlated events that might explain what happened.

If I configure my plugins' jails to start with DCHP, they fail to start from the web UI. If I start them in the CLI with, I can see that there's a DHCP failure:

Code:

% sudo iocage start Heimdall
No default gateway found for ipv6.
* Starting Heimdall
  + Started OK
  + Using devfs_ruleset: 1000 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
  + Acquiring DHCP address: FAILED, address received: 0.0.0.0/8

Stopped Heimdall due to DHCP failure

If I set a manual IP address, I can get it to start, but I see some odd behaviour:

Code:

% sudo iocage start Heimdall
# ...

% sudo iocage console Heimdall

root@Heimdall:~ # ping 10.0.0.01
PING 10.0.0.01 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.21: icmp_seq=0 ttl=63 time=0.392 ms # (╯°□°）╯︵ ┻━┻ Why is 10.0.0.21 answering?

root@Heimdall:~ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 10.0.0.21: icmp_seq=0 ttl=51 time=11.704 ms # ?!?!?!

root@Heimdall:~ # traceroute 1.1.1.1 # Takes a loooong time to complete
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
 1  10.0.0.1 (10.0.0.1)  0.205 ms  0.239 ms  0.200 ms
 2  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  11.066 ms  14.599 ms  8.961 ms
 3  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  8.970 ms  9.287 ms  19.403 ms
 4  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  14.700 ms  8.905 ms  10.142 ms
 5  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  11.075 ms  10.375 ms  8.264 ms
 6  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  11.683 ms  10.123 ms  17.810 ms
 7  * * *
 8  1.1.1.1 (1.1.1.1)  7.673 ms  11.948 ms  11.694 ms # Actually pings 1.1.1.1 and not 10.0.0.21 ?

root@Heimdall:~ # ping google.com
# no response, DNS is borked?

root@Heimdall:~ # host google.com # Nope, dns is fine ...?
google.com has address 142.251.33.174
google.com has IPv6 address 2607:f8b0:400b:803::200e
google.com mail is handled by 10 smtp.google.com.


root@Heimdall:~ # arp -a
? (10.0.0.1) at 7a:45:58:ea:ba:55 on epair0b expires in 1168 seconds [ethernet]  # It sees my router!
? (10.0.0.28) at 9e:d1:f7:74:b3:e9 on epair0b permanent [ethernet]               # This is the jail itself
? (10.0.0.21) at 9e:d1:f7:cd:ba:ad on epair0b expires in 1087 seconds [ethernet] # This is the TrueNAS host

I can ping my jail from other machines on the network, too, which I take as a sign that my issue is not with my router or Proxmox configuration (I have TrueNAS running in a VM). I'm confident that it's actually my jail responding, because the ping is only successful when the jail is actually running.

Code:

$ traceroute 10.0.0.28
traceroute to 10.0.0.28 (10.0.0.28), 64 hops max, 52 byte packets
 1  kserve    (10.0.0.21)  4.846 ms  2.276 ms  3.047 ms # This is the TrueNAS host
 2  10.0.0.28 (10.0.0.28)  3.123 ms  3.120 ms  3.272 ms # This is the jail running on it

$ ping 10.0.0.28
PING 10.0.0.28 (10.0.0.28): 56 data bytes
64 bytes from 10.0.0.28: icmp_seq=0 ttl=63 time=2.412 ms

Some other stats:

• My default gateway is a Unify Dream Machine SE on 10.0.0.1
• My subnet is 10.0.0.1/16
• My TrueNAS instance is called "kserve", on 10.0.0.21
• My jails are usually on 10.0.0.22 and 10.0.0.23, but I moved them to 10.0.0.28 and 10.0.0.29 as a trouble-shooting step
• I'm running TrueNAS CORE 13.0-U3.1
• I'm not sure if it makes a difference, but this Truenas instance is virtualized inside on a Proxmox virtualization server.

Could anyone suggest some troubleshooting steps, or have any ideas what might be happening here?

Show : Detailed TrueNAS configuration

• Network
  • Network Summary
    
    
  • Global Configuration
    
    
  • Interfaces
    
    
    
    
    
    
  • ifconfig
    
    
    Code:

    vtnet0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500    description: The Proxmox virtual NIC
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 9e:d1:f7:cd:ba:ad
        inet 10.0.0.21 netmask 0xffff0000 broadcast 10.0.255.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
    tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.30.0.1 --> 10.30.0.2 netmask 0xffff0000
        groups: tun
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 1468
    bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ee:4a
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
    vnet0.12: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: Heimdall as nic: epair0b
        options=8<VLAN_MTU>
        ether 9e:d1:f7:74:b3:e8
        hwaddr 02:c9:58:ae:1a:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
    

• Jails
  • Heimdall
    
    
    
    
    
    Code:

    root@Heimdall:~ # ifconfiglo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
    epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 9e:d1:f7:74:b3:e9
        hwaddr 02:c9:58:ae:1a:0b
        inet 10.0.0.28 netmask 0xffff0000 broadcast 10.0.255.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>