Jails on Core 13.0-U5.2 do not connect to the internet

[BuffTofu]

"When in doubt, use tcpdump."

(unknown network engineer, possibly me )

tcpdump has been running for nearly an hour now. What can I get out of it?

 

[Patrick M. Hausen]

E.g. run tcpdump -i bridge0 -n port 53 on the NAS host and try a name lookup inside a jail.

And post the output of ifconfig on the NAS host, please.

 

[BuffTofu]

E.g. run tcpdump -i bridge0 -n port 53 on the NAS host and try a name lookup inside a jail.

And post the output of ifconfig on the NAS host, please.

That command doesn't seem to do anything, or I am not waiting long enough. This is ifconfig output: root@truenas[~]# ifconfig re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: NetworkInterface options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 54:be:f7:51:83:ed inet 192.168.0.250 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0<> metric 0 mtu 33160 groups: pflog bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:82:40 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.48 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 2000 member: vnet0.47 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 2000 member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet 192.168.1.1 --> 192.168.1.2 netmask 0xffffff00 groups: tun nd6 options=9<PERFORMNUD,IFDISABLED> Opened by PID 1932 vnet0.47: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: homebridge as nic: epair0b options=8<VLAN_MTU> ether 56:be:f7:96:5c:6d hwaddr 02:55:5d:8c:2c:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> vnet0.48: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: Odasrv as nic: epair0b options=8<VLAN_MTU> ether 56:be:f7:36:6d:c6 hwaddr 02:ca:ff:93:e6:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> root@truenas[~]#

 

[Patrick M. Hausen]

Have you disabled hardware offloading for re0?

Also you should move the IP address from re0 to bridge0, but that's a different story and probably not the cause of your problems.

 

[BuffTofu]

Have you disabled hardware offloading for re0?

Also you should move the IP address from re0 to bridge0, but that's a different story and probably not the cause of your problems.

Disabling does nothing. Could you explain the second thing?

 

[Whattteva]

Curious, did you try rebooting with earlier version that you upgraded from and see if the DNS still works there also?

 

[BuffTofu]

Curious, did you try rebooting with earlier version that you upgraded from and see if the DNS still works there also?

I’ll try that first thing tomorrow.

 

[Patrick M. Hausen]

Disabling does nothing. Could you explain the second thing?

Did you reboot after disabling hardware offloading? Jails tend to lose connectivity after changes to the network settings. I've developed the habit to reboot the system after any change.

A bridge member interface must not have a layer 3 address in FreeBSD. This requirement has been grossly violated by FreeNAS/TrueNAS for years. You need to reboot with all jails disabled, manually create a bridge0 interface with re0 as the member. Remove IP address from re0 only putting "up" into the options field. Apply IP address to bridge0 interface.

If that worked well, then configure your jails:

vnet_default_interface: none
interfaces: vnet0:bridge0

Then re-enable your jails.

I cannot imagine how that would fix your particular problem with DNS but it's mandatory if you ever want to use multicast applications and/or IPv6.

 

[BuffTofu]

Remove IP address from re0 only putting "up" into the options field. Apply IP address to bridge0 interface.

What does that mean? Additionally, here are the config pages for one of the jails. It could be likely I just messed up or misinterpreted some setting.

 

[BuffTofu]

Curious, did you try rebooting with earlier version that you upgraded from and see if the DNS still works there also

I tried downgrading but it gives me this: Error: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body rv = await self.method(*([self] + args)) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 981, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/update.py", line 389, in file await self.middleware.call('update.install_manual_impl', job, destfile, dest_extracted) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1279, in call return await self._call( File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1247, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/update_/install_freebsd.py", line 72, in install_manual_impl if self.install_impl(job, dest_extracted) is None: File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/update_/install_freebsd.py", line 37, in install_impl raise CallError(f'Unable to downgrade from {old_version} to {new_version}') middlewared.service_exception.CallError: [EFAULT] Unable to downgrade from TrueNAS-13.0-U5.3 to TrueNAS-13.0-U5

 

[Whattteva]

How are you doing the downgrade? Are you doing it through System -> Boot -> activate (from the 3 dots)??

 

[BuffTofu]

How are you doing the downgrade? Are you doing it through System -> Boot -> activate (from the 3 dots)??

Is that what I'm supposed to do? I uploaded the file in System -> Update

 

[Davvo]

Is the raw package jail setting checked?

 

[Whattteva]

Is that what I'm supposed to do? I uploaded the file in System -> Update

No, don't do it that way. The reason why upgrades are near risk-free on ZFS is due to having boot environments. Basically, it makes a full system snapshot before the upgrade and you can always return to that point in time. Do it through System -> Boot and activate the last version that worked.

Through this menu, you can jump back and forth between environments seamlessly.

 

[BuffTofu]

No, don't do it that way. The reason why upgrades are near risk-free on ZFS is due to having boot environments. Basically, it makes a full system snapshot before the upgrade and you can always return to that point in time. Do it through System -> Boot and activate the last version that worked.

Through this menu, you can jump back and forth between environments seamlessly.

I didn’t know that. I went to U3 and it changed nothing.

 

[Whattteva]

I didn’t know that. I went to U3 and it changed nothing.

You have to reboot after you change it. Did you reboot?

 

[BuffTofu]

You have to reboot after you change it. Did you reboot?

Yes

 

[winnielinnie]

Can you create a new test jail, and see if DNS works under the test jail?

 

[BuffTofu]

Can you create a new test jail, and see if DNS works under the test jail?

Does not work

 

[winnielinnie]

Are you using any custom scripts or firewall rules?