SOLVED Subnets gateways

listhor · Mar 24, 2020

I have configured a few interfaces in FreeNAS:

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.17.0.9 netmask 0xffffff00 broadcast 172.17.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg4095: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.5 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.4 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.62 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    groups: vlan
vlan13: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Dzieci
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.3.14 netmask 0xfffffff0 broadcast 172.16.3.15
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 13 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
vlan15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: VPN
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.5.4 netmask 0xfffffff0 broadcast 172.16.5.15
    inet 172.16.5.5 netmask 0xfffffff0 broadcast 172.16.5.15
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 15 vlanpcp: 1 parent interface: lagg4095
    groups: vlan

Routing

Code:

Routing tables
Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.1.1         UGS      vlan11
10.55.0.0/16       link#2             U          vmx0
10.55.1.2          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0
172.16.0.0/24      link#3             U          vmx1
172.16.0.9         link#3             UHS         lo0
172.16.1.0/26      link#8             U        vlan11
172.16.1.2         link#8             UHS         lo0
172.16.1.3         link#8             UHS         lo0
172.16.1.4         link#8             UHS         lo0
172.16.1.5         link#8             UHS         lo0
172.16.1.62        link#8             UHS         lo0
172.16.3.0/28      link#9             U        vlan13
172.16.3.14        link#9             UHS         lo0
172.16.5.0/28      link#11            U        vlan15
172.16.5.4         link#11            UHS         lo0
172.16.5.5         link#11            UHS         lo0
172.17.0.0/24      link#5             U          igb2
172.17.0.9         link#5             UHS         lo0

I would expect that communication to and from specific interface goes through its respective gateway (which is not configurable). But it's not that case, everything goes back through default gateway.

Code:

traceroute -i vlan15 1.1.1.1         
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
 1  172.16.1.1 (172.16.1.1)  0.311 ms  0.195 ms  0.158 ms
(....)

If I remove default gateway, than FreeNAS looses all connectivity.
vlan15 supposed to be going through VPN gateway configured in my router, but since data in FreeNAS goes through its default gateway - everything leaks out...

I found this thread: https://www.ixsystems.com/community/threads/gui-not-accessible-without-default-gateway.78747/
So, it means that there is no other way, no hope left??

Yorick · Mar 24, 2020

This is normal routing behavior. You have no specific route for 1.1.1.1, so that goes by the 0.0.0.0 (default) route. vlan15 has 172.16.5.0/28, things in that subnet would be reached via that interface.

Think of this as you would any router. That it's a FreeNAS box doesn't change basic routing behavior.

I think more to the point is: What are you actually looking to achieve? What's the use case? "I want to route this here" is a technical solution to a use case. So, what's your use case? Why route 1.1.1.1 through vlan15?

In general, there are a couple simple routing mechanisms that can be used. Designs that use as many mechanisms as needed, but no more, are better, because simpler.

Destination routing. To reach prefix N, go to next-hop Y. Special cases: Equal Cost Multi-Pathing, and "fallback" routes with different priorities.

VRFs. Interfaces E1 through E3 are in VRF 1, Interfaces E4 through E6 in VRF 2. Routing is contained to things within a VRF. Cross-VRF routing can be configured. This is useful if different services need to take different routes.

Policy based routing, sometimes called source routing. When criteria are met, go to next-hop Y.

Not really routing but a way to avoid overly complex routing schemes: http proxy. For all http traffic, use this proxy. That means all I need is a /32 destination route on the "Internet" link, while I can still default route to the rest of my network via the "trusted" link. More a thing in enterprises, incredibly useful for banks.

Which of these methods FreeNAS supports I don't know. Let's see what you are looking to achieve, first, and whether the VLAN-heavy design you have here is sufficiently simple.

listhor · Mar 24, 2020

Yorick said:
I think more to the point is: What are you actually looking to achieve? What's the use case? "I want to route this here" is a technical solution to a use case. So, what's your use case? Why route 1.1.1.1 through vlan15?
(...)
Destination routing. To reach prefix N, go to next-hop Y. Special cases: Equal Cost Multi-Pathing, and "fallback" routes with different priorities.

VRFs. Interfaces E1 through E3 are in VRF 1, Interfaces E4 through E6 in VRF 2. Routing is contained to things within a VRF. Cross-VRF routing can be configured. This is useful if different services need to take different routes.

Policy based routing, sometimes called source routing. When criteria are met, go to next-hop Y.

Not really routing but a way to avoid overly complex routing schemes: http proxy. For all http traffic, use this proxy. That means all I need is a /32 destination route on the "Internet" link, while I can still default route to the rest of my network via the "trusted" link. More a thing in enterprises, incredibly useful for banks.
(...)

Thanks for explanation. 1.1.1.1 was just an example for traceroute. What I would like to achieve is that every piece of data originated within subnet/interface goes out through its gateway - or at least for VPN vlan; it's mainly for jails and apps installed there. Thus, I think I'm talking about SNAT. How to achieve that in FreeNAS? Static routes in GUI are rather for DNAT?

Yorick · Mar 24, 2020

I'll throw up my hands, I have no idea "why" this design, so I can't make meaningful suggestions. That's cool, others can come along and ask smarter questions or come with suggestions.

Kcaj · Mar 24, 2020

You need to rethink how you are configuring your jails looking at all the aliases assigned to VLAN11. If I am not mistaken you are choosing it the jails IPv4 Interface?

Just read this, its very possible to configure what you are trying to achieve, you just need to slow down and read and understand what you are doing.

listhor · Mar 24, 2020

Kcaj said:
You need to rethink how you are configuring your jails looking at all the aliases assigned to VLAN11. If I am not mistaken you are choosing it the jails IPv4 Interface?
Just read this, its very possible to configure what you are trying to achieve, you just need to slow down and read and understand what you are doing.

I've read Freenas vlan part of manual but there's not much there.
I have slightly different interfaces setup - GUI is binded to dedicated interface (vmx1) and not lagg which is used (or suppose to be used) for data link. Anyway, using Tunables I couldn't create bridge0 - there was nothing like that showed in ifconfig after restart.
When trying to change jails network setup I couldn't assign IPV4 interface of vnet0, always comes up as vnet0:bridge0 and after changing other jail's setting I've ended up with:

Code:

  route: writing to routing socket: Network is unreachable
add net default: gateway 172.16.5.1 fib 0: Network is unreachable

And still I don't understand why it is so overcomplicated instead of simply using each subnet/interface default gateway... Or, I'm not smart enough :)

listhor · Mar 25, 2020

listhor said:
And still I don't understand why it is so overcomplicated instead of simply using each subnet/interface default gateway... Or, I'm not smart enough :)

I kept trying and got smarter, haha. Jail's settings I had to apply (difference from this tutorial is there's no IPV4 interface chosen) :

Code:

VNET on Berkley Packet Filter on
IPV4 Interface - "----" #NOTHING - no interface
IPV4 Address - 10.0.1.156   #Option only needed if not using DHCP - Assign desired IP address for the machine -- ie 10.0.1.156 for VLAN1, 10.0.40.156 for VLAN40
IPV4 Netmask - 24               #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1  #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0   #match this for your appropriate VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - search local,nameserver 10.0.1.1,nameserver 1.1.1.1,nameserver 1.0.0.1  (See ***Note below)
vnet_default_interface - auto

listhor · Mar 25, 2020

@Kcaj maybe you'll be able to help once again. Above works as long as I don't touch network settings - if I reload them or save something new, I loose connectivity to all vnet networked jails - I need to restart them to be able to connect to their services... Maybe it has something to do with lack of bridge0 interface (Tunables don't create it)...
ifconfig:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.17.0.9 netmask 0xffffff00 broadcast 172.17.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg4095: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.1.62 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    groups: vlan
vlan13: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Dzieci
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 13 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
vlan15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: NordVPN
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 15 vlanpcp: 1 parent interface: lagg4095
    groups: vlan
vlan14: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zaufana
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 14 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
bridge15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vpn
    ether 02:11:2a:4b:ab:0f
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.16 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vlan15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000000
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zarządzanie
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 1 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zasoby
    ether 02:11:2a:4b:ab:0b
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.18 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vnet0.17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 18 priority 128 path cost 2000
    member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
bridge13: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: dzieci
    ether 02:11:2a:4b:ab:0d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan13 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 10000
bridge14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zaufana
    ether 02:11:2a:4b:ab:0e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan14 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 10000
vnet0.16: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: vpn as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:c0:3a:d8
    hwaddr 02:e9:d0:00:0e:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.17: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:14:fa:09
    hwaddr 02:e9:d0:00:12:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.18: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: associated with jail: nextcloud as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:ba:b5:81
    hwaddr 02:e9:d0:00:13:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

Kcaj · Mar 25, 2020

I personally dont use bridge0 with my VLAN setup, or any turnables and I dont have an issue with loosing connection to my jails as you describe. First thing I would try is to see if your jails can communicate out when you have this problem again.

Could the link aggregation interface be the culprit, when the configuration reloads?

listhor · Mar 25, 2020

Kcaj said:
I personally don't use bridge0 with my VLAN setup, or any turnables and I don't have an issue with loosing connection to my jails as you describe. First thing I would try is to see if your jails can communicate out when you have this problem again.

Could the link aggregation interface be the culprit, when the configuration reloads?

Tunables don't create bridge0 anymore, and even after creating it manually nothing changes - still jails are affected by network restart. I'm not so sure whether lagg can be hold responsible. Communication between Freenas and LAN is not affected. Only jails are "isolated".
When inside jails and issuing netstat -r command, I receive following output but with delay of about 30 - 40 seconds:

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.1.1         UGS     epair0b
localhost          link#1             UH          lo0
172.16.1.0/26      link#2             U       epair0b
172.16.1.3         link#2             UHS         lo0

Above output content is as normal. To me, it looks like vnet is affected but ifconfig output is as same as before network restart.
In syslog I can see some abnormal mdns service output:

25.03.2020 22:06:57 Błąd freenas daemon mDNSResponder Client application[-1]() registered 2 identical instances of service freenas._http._tcp.local. port 5000.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNSPlatformSourceAddrForDest: connect 1.1.1.1 failed errno 51 (Network is unreachable)

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNSResponder (FreeBSD 11.0) (Feb 21 2020 19:35:46) starting

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080109FD80 01 28 _webdav._tcp.local. PTR freenas._webdav._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080109E980 01 25 _smb._tcp.local. PTR freenas._smb._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080109D580 01 25 _ssh._tcp.local. PTR freenas._ssh._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801095180 01 25 _ssh._tcp.local. PTR freenas._ssh._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801088980 01 25 _nfs._tcp.local. PTR freenas._nfs._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801089D80 01 36 _middleware-ssl._tcp.local. PTR freenas._middleware-ssl._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080108B180 01 32 _middleware._tcp.local. PTR freenas._middleware._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080108C580 01 27 _iscsi._tcp.local. PTR freenas._iscsi._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080108D980 01 27 _https._tcp.local. PTR freenas._https._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 000000080108ED80 01 27 _https._tcp.local. PTR freenas._https._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801090180 01 26 _http._tcp.local. PTR freenas._http._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801091580 01 26 _http._tcp.local. PTR freenas._http._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801092980 01 33 _device-info._tcp.local. PTR freenas._device-info._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNS_FinalExit failed to send goodbye for: 0000000801093D80 01 27 _adisk._tcp.local. PTR freenas._adisk._tcp.local.

25.03.2020 22:06:55 Błąd freenas daemon mDNSResponder mDNSResponder (FreeBSD 11.0) (Feb 21 2020 19:35:46) stopping

I'm not sure where to start looking for the root cause...

Important Announcement for The TrueNAS Community.

SOLVED Subnets gateways

listhor

Contributor

Yorick

Wizard

listhor

Contributor

Yorick

Wizard

Kcaj

Contributor

listhor

Contributor

listhor

Contributor

listhor

Contributor

Kcaj

Contributor

listhor

Contributor

Similar threads