Jails on Core 13.0-U5.2 do not connect to the internet

[Patrick M. Hausen]

iX' implementation violated FreeBSD's documented architecture for the bridge implementation since day one of VNET jails. I don't understand why they do it that way. Using vnet_default_interface: auto performs some "magic" that frequently seems to work (still ending with an illegal configuration) and sometimes doesn't.

The FreeBSD docs clearly state that a bridge member interface MUST NOT have an IP address.

Fortunately this can all be fixed by configuration in the TrueNAS context.

 

[Davvo]

I'm using a bridge, what should I check to confirm everything is per standard? It works fine, but I prefer doing things by the manual.

 

[Patrick M. Hausen]

Did you explicitly configure the bridge in Network > Interfaces? What are you using it for?

 

[Davvo]

Did you explicitly configure the bridge in Network > Interfaces? What are you using it for?

Yes. I am assigning it to my jails.

 

[Patrick M. Hausen]

Does your NAS host have an IP address in that same network? If yes, is it assigned to the bridge or to the physical member interface?

 

[Davvo]

Does your NAS host have an IP address in that same network? If yes, is it assigned to the bridge or to the physical member interface?

All jails have the bridge0 as network interface.

 

[Patrick M. Hausen]

Is igb0 a member of bridge0? If yes, you must remove the IP address from igb0. Simply untick the "DHCP" flag and instead just put "up" in the options field.

Caveat: you will not be able to reach your NAS via 192.168.1.120, anymore - only via 192.168.1.124 as statically configured on bridge0. This is the only correct configuration, altough you are of course free to pick any IP address in that network. So you might want to change the bridge to .120 if you have clients relying on that address being available.

The two problems you currently have:

• any bridge member interface MUST NOT have an IP address
• two interfaces can never have different IP addresses from the same network

HTH,
Patrick

 

[ElCas]

I'm having the same issue as reported and have followed along with the instructions but I can't seem it get my jail to connect/reach anything. Ping with IP address I get No route to host. If I ping with CName I get Unknown host. I have no issues pinging with IP address or CName in the main Server.

I'm still a noob with TrueNAS so forgive me if I missed something obvious.

Jail info

Jail Network

Global Network Config

Interfaces

Main ifconfig

em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
em1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
hwaddr 0c:c4:7a:04:35:47
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: lagg0
options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
inet 192.168.0.9 netmask 0xffffff00 broadcast 192.168.0.255
laggproto loadbalance lagghash l2,l3,l4
laggport: em0 flags=4<ACTIVE>
laggport: em1 flags=4<ACTIVE>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:ff:e6
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 10000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

jail /etc/resolf.conf

domain local
search local
nameserver 192.168.0.1

jail ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:c4:7a:5a:b8:7e
hwaddr 02:06:5c:84:d1:0b
inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>

 

[Patrick M. Hausen]

I recommend not using DHCP for your NAS host but static configuration.

Then you need to create a bridge interface with lagg0 as the only member:

- disable autostart of your jail
- reboot
- Network > Interfaces > Add - type bridge, name bridge0, member interface lagg0, IP address 192.168.0.9/24
- disable DHCP on lagg0 - the static address on bridge0 will take over
- test and save - increase the timeout for test, because you will lose connectivity for a minute or so. You can clear the ARP cache on your desktop after clicking "test" to speed this up
- enter a DNS server and default gateway in Network > Global Settings, make sure your NAS has got Internet access

Now in your jail:

- vnet_default_interface: none
- IPv4 interface: vnet0
- IPv4 address: 192.168.0.4
- IPv4 netmask: /24
- IPv4 default router: keep as is
- interfaces: vnet0:bridge0 - keep as is
- resolver: keep as is

That should do it.

 

[ElCas]

I recommend not using DHCP for your NAS host but static configuration.

It's not really DHCP, the router has it on a static IP. I just left it DHCP so that if need to change the IP I just do it at the router and it reflects automatically on the NAS.

Then you need to create a bridge interface with lagg0 as the only member:

- disable autostart of your jail

They already were (I only fire them up when I need them hence why I just found out I could't reach the anything. )

- reboot
- Network > Interfaces > Add - type bridge, name bridge0, member interface lagg0, IP address 192.168.0.9/24
- disable DHCP on lagg0 - the static address on bridge0 will take over
- test and save - increase the timeout for test, because you will lose connectivity for a minute or so. You can clear the ARP cache on your desktop after clicking "test" to speed this up
- enter a DNS server and default gateway in Network > Global Settings, make sure your NAS has got Internet access

Now in your jail:

- vnet_default_interface: none
- IPv4 interface: vnet0
- IPv4 address: 192.168.0.4
- IPv4 netmask: /24
- IPv4 default router: keep as is
- interfaces: vnet0:bridge0 - keep as is
- resolver: keep as is

That should do it.

I made all changes but the jail still can't reach anything. Anything else I can try to make the jails be able to reach out?


main ifconfig

em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=4812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

em1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=4812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
hwaddr 0c:c4:7a:04:35:47
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: lagg0
options=4812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 0c:c4:7a:04:35:46
laggproto loadbalance lagghash l2,l3,l4
laggport: em0 flags=4<ACTIVE>
laggport: em1 flags=4<ACTIVE>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Briidge for everything
ether 58:9c:fc:10:ff:e6
inet 192.168.0.9 netmask 0xffffff00 broadcast 192.168.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 10000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: reseller as nic: epair0b
options=8<VLAN_MTU>
ether 0e:c4:7a:5a:b8:7d
hwaddr 02:21:06:93:6b:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

jail ifconfigs

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:c4:7a:5a:b8:7e
hwaddr 02:e8:44:bf:23:0b
inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>

New NAS configs

Jail Configs

 

[Patrick M. Hausen]

Looks good - only thing that comes to mind: disable hardware offloading for em0 and em1 ...

Then try to ping 192.168.0.9 from within the jail, then 192.168.0.1, then we'll check DNS if necessary.

 

[ElCas]

Looks good - only thing that comes to mind: disable hardware offloading for em0 and em1 ...

Done.

Then try to ping 192.168.0.9 from within the jail, then 192.168.0.1, then we'll check DNS if necessary.

Pinging 192.168.0.9 response as expected. Pinging 192.168.0.1 returns Host is down.

 

[Patrick M. Hausen]

Could you take that lagg out of the equation? Just make em0 a member of the bridge and put "up" into the options field of em0?

Loadbalance is not guaranteed to work, only LACP is ...

 

[ElCas]

Could you take that lagg out of the equation? Just make em0 a member of the bridge and put "up" into the options field of em0?

Loadbalance is not guaranteed to work, only LACP is ...

That seems to have done the trick.

So now that lagg only has em1 does that mean it will not be used anymore by the NAS?

 

[Patrick M. Hausen]

Correct. I am running everything on a lagg without problems, do you have an LACP capable switch? Were you perchance running loadbalanced lagg to two different switches? That's not working as you found out.

 

[ElCas]

Correct. I am running everything on a lagg without problems, do you have an LACP capable switch? Were you perchance running loadbalanced lagg to two different switches? That's not working as you found out.

Both NICs are on the same switch so in theory it should be fine as long as I don't start making a lot of request.

 

[Patrick M. Hausen]

You can use one NIC for your NAS services and one for the jails. Delete the lagg, remove 192.168.0.9 from bridge 0 and put it on em1 ... put "up" into the options field of bridge0 instead.

Or use LACP.

 

[ElCas]

You can use one NIC for your NAS services and one for the jails. Delete the lagg, remove 192.168.0.9 from bridge 0 and put it on em1 ... put "up" into the options field of bridge0 instead.

Or use LACP.

Since my switch is a stock Linksys WRT-1200AC working as a router/switch, LACP is not an option for me

I made the changes but I am getting invalid argument when setting up the bridge

File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 57, in bridge_setup
iface.add_member(member)
File "netif.pyx", line 1211, in netif.BridgeInterface.add_member
File "netif.pyx", line 1229, in netif.BridgeInterface.bridge_cmd
OSError: [Errno 22] Invalid argument

 

[Patrick M. Hausen]

That's a bit weird, sorry, without actually trying myself on your system it's difficult to diagnose from here. Possibly switch em0 and em1? Or it requires multiple steps like deleting the bridge first - which means the jail must be down - then assigning IP configuration to e.g. em0 again - even with DHCP if you prefer, then recreating the bridge with em1 as member and only "up" as described.

Anyway what I forgot to mention: if you go that route assigning one interface to the NAS itself and one to the bridge for jails, always make 100% sure to set vnet_default_interface to "none". If you leave it to "auto" TrueNAS might perform "magic" assignments of member interfaces to bridges that lead to a bridging loop and bring down your network by what is called a broadcast storm.

Another use is of course to leave the IP address at the bridge interface and make both interfaces members. That gives you two possible topologies. First you could connect both of them to your switch infrastructure, two different switches even, for redundancy. BUT YOU ABSOLUTELY MUST ENABLE STP FOR THAT. Or you will create a loop leading to a broadcast storm ...

STP can be enabled by putting "stp em0 stp em1" into the options field of the bridge interface.

Second feature would be to connect another device like e.g. a printer that shares a cabinet with the NAS so you need to run only one cable to that location. Or if your NAS has got a 10G and a 1G interface, bridge them both, plug your single workstation Mac or PC via 10G into the 10G interface to work on those videos, and connect the 1G to your switch/router/Internet infrastructure.

Just some ideas.

HTH,
Patrick

 

[ElCas]

That's a bit weird, sorry, without actually trying myself on your system it's difficult to diagnose from here. Possibly switch em0 and em1? Or it requires multiple steps like deleting the bridge first - which means the jail must be down - then assigning IP configuration to e.g. em0 again - even with DHCP if you prefer, then recreating the bridge with em1 as member and only "up" as described.

Anyway what I forgot to mention: if you go that route assigning one interface to the NAS itself and one to the bridge for jails, always make 100% sure to set vnet_default_interface to "none". If you leave it to "auto" TrueNAS might perform "magic" assignments of member interfaces to bridges that lead to a bridging loop and bring down your network by what is called a broadcast storm.

Another use is of course to leave the IP address at the bridge interface and make both interfaces members. That gives you two possible topologies. First you could connect both of them to your switch infrastructure, two different switches even, for redundancy. BUT YOU ABSOLUTELY MUST ENABLE STP FOR THAT. Or you will create a loop leading to a broadcast storm ...

STP can be enabled by putting "stp em0 stp em1" into the options field of the bridge interface.

Second feature would be to connect another device like e.g. a printer that shares a cabinet with the NAS so you need to run only one cable to that location. Or if your NAS has got a 10G and a 1G interface, bridge them both, plug your single workstation Mac or PC via 10G into the 10G interface to work on those videos, and connect the 1G to your switch/router/Internet infrastructure.

Just some ideas.

HTH,
Patrick

Thanks for all your help. Since it's working now and I don't seem to be having any issues I'll just leave as it and manually set em1 to the bridge should something happen to em0.