TrueNAS will not communicate outside of its VLAN

[aidentherobot]

Hi yall,

I recently upgraded my TrueNAS server with a 10gbe card however after doing so my TrueNAS box cannot ping or even see the other Lans on the network. I tested my switch with another machine and I was able to ping every subnet as well as hosts on those subnets. I have reinstalled the OS on the Machine but that has not helped. My NAS is on the 10.0.4.0/22 network and If I want to ping a host on the 10.0.18.0/24 network It will complain saying that it cannot find a route to the host. This is very weird and I have never had this happen before on this machine I am really stumped on this if anybody has an idea that would be helpful.

 

[Patrick M. Hausen]

Go to Network > Global Settings and enter the correct default gateway for that VLAN.

 

[NinthWave]

Go to Network > Global Settings and enter the correct default gateway for that VLAN.

I have the same problem.

TrueNAS is at 10.0.0.6 and there is an "allow all rule" on this subnet but still, TN cannot ping anything on 10.0.10.x except the jails and VM hosted on it. It cannot be pinged either from devices on the 10.0.10.x subnet.

[EDIT]
It can ping google, it can ping the cameras at 10.0.40.x but it cannot ping 10.0.10.x

I have jails and 1 VM running at 10.0.10.x
I know the cable is disconnected at en1 but I am trying to reach devices outside the host and the cameras are reachable.

 

[Samuel Tai]

What netmask have you set?

 

[NinthWave]

What netmask have you set?

On pfSense, it 10.0.0.0/24 and 10.0.10.0/24

 

[Patrick M. Hausen]

Please provide a diagram of your network including switch(es) if present.

 

[NinthWave]

Please provide a diagram of your network including switch(es) if present.

This is what I got on hand but I can draw something less cluttered if needed.

 

[Patrick M. Hausen]

That's quite a mouthful Running a network this complex you should be familiar with troubleshooting methods like running a packet trace with tcpdump. I'll try to help, of course.

Please add the output of ifconfig and netstat -rn on your TrueNAS host. Then we can discuss the connection of your TrueNAS to the rest of the network.

 

[NinthWave]

TrueNAS will not communicate outside of its VLAN

Hi yall, I recently upgraded my TrueNAS server with a 10gbe card however after doing so my TrueNAS box cannot ping or even see the other Lans on the network. I tested my switch with another machine and I was able to ping every subnet as well as hosts on those subnets. I have reinstalled the OS...

www.truenas.com

I think it's the asymetric routing you suspected last year. I hid it under the carpet by bringing the VM/Jails in the native subnet. But now that that I want to put those server apps on another machine, I have to face it.

 

[NinthWave]

Code:

truenas% ifconfig
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 84:2b:2b:51:53:8a
        inet 10.0.0.6 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 84:2b:2b:51:53:8c
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IOT
        options=80000<LINKSTATE>
        ether 84:2b:2b:51:53:8c
        groups: vlan
        vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: bce1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Surveillance
        options=80000<LINKSTATE>
        ether 84:2b:2b:51:53:8c
        groups: vlan
        vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: bce1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:e5:2a
        inet 10.0.10.2 netmask 0xffffff00 broadcast 10.0.10.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
        member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ab:7c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:54:42
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: seafile as nic: epair0b
        options=8<VLAN_MTU>
        ether 86:2b:2b:3b:01:7e
        hwaddr 02:c5:dc:b8:8c:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: embyserver as nic: epair0b
        options=8<VLAN_MTU>
        ether 86:2b:2b:c0:af:79
        hwaddr 02:58:9f:95:a9:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:35:d9:4c
        hwaddr 58:9c:fc:10:08:26
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        Opened by PID 43344

I find it odd that bce0 shows as VLAN

Code:

truenas% netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.0.1           UGS        bce0
10.0.0.0/24        link#1             U          bce0
10.0.0.6           link#1             UHS         lo0
10.0.10.0/24       link#7             U      bridge10
10.0.10.2          link#7             UHS         lo0
127.0.0.1          link#3             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

 

[Patrick M. Hausen]

I find it odd that bce0 shows as VLAN

I don't understand.

 

[NinthWave]

From what I gathered last year when I upgraded the network, one can not mix tagged and intagged trafic on NIC.
The jails and VM are all linked to bce1 while bce0 only get untagged traffic from the local switch. So I don't understand why there is a vlan status at bce0.

 

[Patrick M. Hausen]

What do you mean by VLAN status? The flags? That only means hardware offloading support for VLAN tags is enabled for bce0. Which is the default. If there are not tagged frames arriving at bce0, that won't change anything. You do have hardware offloading for bce1 disabled, right?

 

[Patrick M. Hausen]

The reason is that your cable is disconnected at bce1. You have an address in the network 10.0.10.0/24 set on bridge10. That means to your NAS the network is locally connected. It won't send traffic to that network out to the default gateway, it will send it to bridge10. That's why you can reach your jails/VMs on that network.

You need to either connect bce1 to a trunk port on your switch - then your NAS can reach the VLANs on that interface directly without firewall intervention. Or if you want to isolate the NAS host from the VLANs and the jails on them and pass everything through the firewall then you must remove the IP address from the bridge interface. You cannot have both. How is your NAS supposed to tell? IP address on bridge --> network is local, no traffic going to default gateway.

 

[NinthWave]

You do have hardware offloading for bce1 disabled, right?

Yes

 

[NinthWave]

The reason is that your cable is disconnected at bce1. You have an address in the network 10.0.10.0/24 set on bridge10. That means to your NAS the network is locally connected. It won't send traffic to that network out to the default gateway, it will send it to bridge10. That's why you can reach your jails/VMs on that network.

You need to either connect bce1 to a trunk port on your switch - then your NAS can reach the VLANs on that interface directly without firewall intervention. Or if you want to isolate the NAS host from the VLANs and the jails on them and pass everything through the firewall then you must remove the IP address from the bridge interface. You cannot have both. How is your NAS supposed to tell? IP address on bridge --> network is local, no traffic going to default gateway.

That did it. I recconnected the cable and traffic has resumed.

This is so logical. The TN is dowsntairs with the future server. I disconnected the cable form TN host to connect it to the future server.

Thanks!

 

[Patrick M. Hausen]

Keep in mind that while TrueNAS does not implement a true vSwitch like ESXi it is still perfectly possible to isolate the services and control plane from the jails and VMs. The latter are perfectly satisfied with a bridge interface without any IP address, VLAN or not. It's all layer 2. So you can connect your jails to VLAN x in your switched infrastructure but not the TrueNAS host. If you so desire.

 

[NinthWave]

Last thing. Because I did set this last year:
The bridge10 do need an IP because jails and VM are endpoint in VLAN10 ?

 

[Patrick M. Hausen]

No, it needs that IP address if and only if the NAS host needs to talk to anything in that VLAN without going through your pfSense.

FreeBSD bridge is an unmanaged switch, no VLAN support. Plug bce1 in port based VLAN 10 on your switch, bridge without IP address, all jails on that bridge will end up in VLAN 10.

Need more VLANs? Create VLANs on TrueNAS and bce1, put one bridge on each VLAN, plug into trunk port on switch. Again, IP address on these bridge interfaces only if the TN host itself needs a connection to that VLAN. Unmanaged switches don't have IP addresses at all and still work. It's all layer 2.

Once sorted out it's all rather trivial. Then it offers lots of opportunities to shoot yourself in the foot, because layer 2 loops ...