SOLVED Subnets gateways

listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
I have configured a few interfaces in FreeNAS:
Code:
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.17.0.9 netmask 0xffffff00 broadcast 172.17.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg4095: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.5 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.4 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.62 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    groups: vlan
vlan13: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Dzieci
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.3.14 netmask 0xfffffff0 broadcast 172.16.3.15
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 13 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
vlan15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: VPN
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.5.4 netmask 0xfffffff0 broadcast 172.16.5.15
    inet 172.16.5.5 netmask 0xfffffff0 broadcast 172.16.5.15
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 15 vlanpcp: 1 parent interface: lagg4095
    groups: vlan

Routing
Code:
Routing tables
Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.1.1         UGS      vlan11
10.55.0.0/16       link#2             U          vmx0
10.55.1.2          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0
172.16.0.0/24      link#3             U          vmx1
172.16.0.9         link#3             UHS         lo0
172.16.1.0/26      link#8             U        vlan11
172.16.1.2         link#8             UHS         lo0
172.16.1.3         link#8             UHS         lo0
172.16.1.4         link#8             UHS         lo0
172.16.1.5         link#8             UHS         lo0
172.16.1.62        link#8             UHS         lo0
172.16.3.0/28      link#9             U        vlan13
172.16.3.14        link#9             UHS         lo0
172.16.5.0/28      link#11            U        vlan15
172.16.5.4         link#11            UHS         lo0
172.16.5.5         link#11            UHS         lo0
172.17.0.0/24      link#5             U          igb2
172.17.0.9         link#5             UHS         lo0

I would expect that communication to and from specific interface goes through its respective gateway (which is not configurable). But it's not that case, everything goes back through default gateway.
Code:
traceroute -i vlan15 1.1.1.1         
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
 1  172.16.1.1 (172.16.1.1)  0.311 ms  0.195 ms  0.158 ms
(....)

If I remove default gateway, than FreeNAS looses all connectivity.
vlan15 supposed to be going through VPN gateway configured in my router, but since data in FreeNAS goes through its default gateway - everything leaks out...

I found this thread: https://www.ixsystems.com/community/threads/gui-not-accessible-without-default-gateway.78747/
So, it means that there is no other way, no hope left??
 
Yorick

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
This is normal routing behavior. You have no specific route for 1.1.1.1, so that goes by the 0.0.0.0 (default) route. vlan15 has 172.16.5.0/28, things in that subnet would be reached via that interface.

Think of this as you would any router. That it's a FreeNAS box doesn't change basic routing behavior.

I think more to the point is: What are you actually looking to achieve? What's the use case? "I want to route this here" is a technical solution to a use case. So, what's your use case? Why route 1.1.1.1 through vlan15?

In general, there are a couple simple routing mechanisms that can be used. Designs that use as many mechanisms as needed, but no more, are better, because simpler.

Destination routing. To reach prefix N, go to next-hop Y. Special cases: Equal Cost Multi-Pathing, and "fallback" routes with different priorities.

VRFs. Interfaces E1 through E3 are in VRF 1, Interfaces E4 through E6 in VRF 2. Routing is contained to things within a VRF. Cross-VRF routing can be configured. This is useful if different services need to take different routes.

Policy based routing, sometimes called source routing. When criteria are met, go to next-hop Y.

Not really routing but a way to avoid overly complex routing schemes: http proxy. For all http traffic, use this proxy. That means all I need is a /32 destination route on the "Internet" link, while I can still default route to the rest of my network via the "trusted" link. More a thing in enterprises, incredibly useful for banks.

Which of these methods FreeNAS supports I don't know. Let's see what you are looking to achieve, first, and whether the VLAN-heavy design you have here is sufficiently simple.
 
listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Yorick said:
I think more to the point is: What are you actually looking to achieve? What's the use case? "I want to route this here" is a technical solution to a use case. So, what's your use case? Why route 1.1.1.1 through vlan15?
(...)
Destination routing. To reach prefix N, go to next-hop Y. Special cases: Equal Cost Multi-Pathing, and "fallback" routes with different priorities.

VRFs. Interfaces E1 through E3 are in VRF 1, Interfaces E4 through E6 in VRF 2. Routing is contained to things within a VRF. Cross-VRF routing can be configured. This is useful if different services need to take different routes.

Policy based routing, sometimes called source routing. When criteria are met, go to next-hop Y.

Not really routing but a way to avoid overly complex routing schemes: http proxy. For all http traffic, use this proxy. That means all I need is a /32 destination route on the "Internet" link, while I can still default route to the rest of my network via the "trusted" link. More a thing in enterprises, incredibly useful for banks.
(...)
Click to expand...
Thanks for explanation. 1.1.1.1 was just an example for traceroute. What I would like to achieve is that every piece of data originated within subnet/interface goes out through its gateway - or at least for VPN vlan; it's mainly for jails and apps installed there. Thus, I think I'm talking about SNAT. How to achieve that in FreeNAS? Static routes in GUI are rather for DNAT?
 
Yorick

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
I'll throw up my hands, I have no idea "why" this design, so I can't make meaningful suggestions. That's cool, others can come along and ask smarter questions or come with suggestions.
 
K

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
You need to rethink how you are configuring your jails looking at all the aliases assigned to VLAN11. If I am not mistaken you are choosing it the jails IPv4 Interface?

Just read this, its very possible to configure what you are trying to achieve, you just need to slow down and read and understand what you are doing.
 
listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Kcaj said:
You need to rethink how you are configuring your jails looking at all the aliases assigned to VLAN11. If I am not mistaken you are choosing it the jails IPv4 Interface?
Just read this, its very possible to configure what you are trying to achieve, you just need to slow down and read and understand what you are doing.
Click to expand...
I've read Freenas vlan part of manual but there's not much there.
I have slightly different interfaces setup - GUI is binded to dedicated interface (vmx1) and not lagg which is used (or suppose to be used) for data link. Anyway, using Tunables I couldn't create bridge0 - there was nothing like that showed in ifconfig after restart.
When trying to change jails network setup I couldn't assign IPV4 interface of vnet0, always comes up as vnet0:bridge0 and after changing other jail's setting I've ended up with:
Code:
  route: writing to routing socket: Network is unreachable
add net default: gateway 172.16.5.1 fib 0: Network is unreachable

And still I don't understand why it is so overcomplicated instead of simply using each subnet/interface default gateway... Or, I'm not smart enough :)
 
listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
listhor said:
And still I don't understand why it is so overcomplicated instead of simply using each subnet/interface default gateway... Or, I'm not smart enough :)
Click to expand...
I kept trying and got smarter, haha. Jail's settings I had to apply (difference from this tutorial is there's no IPV4 interface chosen) :
Code:
VNET on Berkley Packet Filter on
IPV4 Interface - "----" #NOTHING - no interface
IPV4 Address - 10.0.1.156   #Option only needed if not using DHCP - Assign desired IP address for the machine -- ie 10.0.1.156 for VLAN1, 10.0.40.156 for VLAN40
IPV4 Netmask - 24               #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1  #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0   #match this for your appropriate VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - search local,nameserver 10.0.1.1,nameserver 1.1.1.1,nameserver 1.0.0.1  (See ***Note below)
vnet_default_interface - auto
 
listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
@Kcaj maybe you'll be able to help once again. Above works as long as I don't touch network settings - if I reload them or save something new, I loose connectivity to all vnet networked jails - I need to restart them to be able to connect to their services... Maybe it has something to do with lack of bridge0 interface (Tunables don't create it)...
ifconfig:
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.17.0.9 netmask 0xffffff00 broadcast 172.17.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg4095: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.1.62 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    groups: vlan
vlan13: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Dzieci
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 13 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
vlan15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: NordVPN
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 15 vlanpcp: 1 parent interface: lagg4095
    groups: vlan
vlan14: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zaufana
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 14 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
bridge15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vpn
    ether 02:11:2a:4b:ab:0f
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.16 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vlan15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000000
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zarządzanie
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 1 vlanpcp: 0 parent interface: lagg4095
    groups: vlan
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zasoby
    ether 02:11:2a:4b:ab:0b
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.18 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vnet0.17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 18 priority 128 path cost 2000
    member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
bridge13: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: dzieci
    ether 02:11:2a:4b:ab:0d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan13 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 10000
bridge14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: zaufana
    ether 02:11:2a:4b:ab:0e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan14 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 10000
vnet0.16: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: vpn as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:c0:3a:d8
    hwaddr 02:e9:d0:00:0e:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.17: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: associated with jail: plex as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:14:fa:09
    hwaddr 02:e9:d0:00:12:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.18: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: associated with jail: nextcloud as nic: epair0b
    options=8<VLAN_MTU>
    ether ac:1f:6b:ba:b5:81
    hwaddr 02:e9:d0:00:13:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
 
K

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
I personally dont use bridge0 with my VLAN setup, or any turnables and I dont have an issue with loosing connection to my jails as you describe. First thing I would try is to see if your jails can communicate out when you have this problem again.

Could the link aggregation interface be the culprit, when the configuration reloads?
 
listhor

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Kcaj said:
I personally don't use bridge0 with my VLAN setup, or any turnables and I don't have an issue with loosing connection to my jails as you describe. First thing I would try is to see if your jails can communicate out when you have this problem again.

Could the link aggregation interface be the culprit, when the configuration reloads?
Click to expand...
Tunables don't create bridge0 anymore, and even after creating it manually nothing changes - still jails are affected by network restart. I'm not so sure whether lagg can be hold responsible. Communication between Freenas and LAN is not affected. Only jails are "isolated".
When inside jails and issuing netstat -r command, I receive following output but with delay of about 30 - 40 seconds:
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.1.1         UGS     epair0b
localhost          link#1             UH          lo0
172.16.1.0/26      link#2             U       epair0b
172.16.1.3         link#2             UHS         lo0

Above output content is as normal. To me, it looks like vnet is affected but ifconfig output is as same as before network restart.
In syslog I can see some abnormal mdns service output:
25.03.202022:06:57BłądfreenasdaemonmDNSResponderClient application[-1]() registered 2 identical instances of service freenas._http._tcp.local. port 5000.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNSPlatformSourceAddrForDest: connect 1.1.1.1 failed errno 51 (Network is unreachable)
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNSResponder (FreeBSD 11.0) (Feb 21 2020 19:35:46) starting
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080109FD80 01 28 _webdav._tcp.local. PTR freenas._webdav._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080109E980 01 25 _smb._tcp.local. PTR freenas._smb._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080109D580 01 25 _ssh._tcp.local. PTR freenas._ssh._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801095180 01 25 _ssh._tcp.local. PTR freenas._ssh._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801088980 01 25 _nfs._tcp.local. PTR freenas._nfs._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801089D80 01 36 _middleware-ssl._tcp.local. PTR freenas._middleware-ssl._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080108B180 01 32 _middleware._tcp.local. PTR freenas._middleware._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080108C580 01 27 _iscsi._tcp.local. PTR freenas._iscsi._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080108D980 01 27 _https._tcp.local. PTR freenas._https._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 000000080108ED80 01 27 _https._tcp.local. PTR freenas._https._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801090180 01 26 _http._tcp.local. PTR freenas._http._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801091580 01 26 _http._tcp.local. PTR freenas._http._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801092980 01 33 _device-info._tcp.local. PTR freenas._device-info._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNS_FinalExit failed to send goodbye for: 0000000801093D80 01 27 _adisk._tcp.local. PTR freenas._adisk._tcp.local.
25.03.202022:06:55BłądfreenasdaemonmDNSRespondermDNSResponder (FreeBSD 11.0) (Feb 21 2020 19:35:46) stopping
Click to expand...
I'm not sure where to start looking for the root cause...
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Subnets gateways"

Similar threads

listhor
Multiple interfaces and shared IPs
Replies
7
Views
4K
listhor
listhor
IamSpartacus
  • Locked
LAG/LACP Media Status = Down?
Replies
7
Views
6K
mwolfe08
M
R
  • Locked
Networking SNMP
Replies
5
Views
3K
tvsjr
T
I
jails bridge0 not coming up at boot.
Replies
1
Views
2K
ikke
I
S
Jails can't browse/ping
2 3
Replies
46
Views
15K
paradoxiom
paradoxiom
Top