Problems with SMB Transfers when VLANs are used

[coolrunnings]

I've got a couple of Dell R530's set up with TrueNAS Core and decided to try using the VLAN features to save some ports as 10G is limited in my environment. Each of the two TrueNAS servers is connected to an 8 port Unifi Aggregation switch via some Ubiquiti media converters. The servers have x540 T2 cards in them. Each port on the switch is set up as a trunk port with all the VLANs I intend to use tagged on it. The VLANs are working correctly on everything except TrueNAS.
I've set up a VLAN in TrueNAS, attached it to the correct interface, and set the Priority Code Point to Best Effort (default). I've assigned it a static IP in the correct subnet for that VLAN. Once I apply the changes, test them, and make them permanent, everything looks to be in order. I can ping. I can access the management interface over that VLAN. However, when I try to transfer files to it over SMB from a couple Windows 11 machines, it seems to be able to traverse the directory structure (albeit often slowly). It then times out with an error: Error 0x8007003B an unexpected network error occurred. Here are the steps I've done to troubleshoot it:
1. I tried a continuous ping: no dropped packets there.
2. I checked permissions and even gave a share full control for everyone. Exact same error.
3. I tried a different network cable (brand new). Same problem.
4. I tried removing the VLAN and just copying direct to the interface. This works as expected.
5. I tried adding the VLAN back in and accessing the SMB share via the IP address of the physical interface: same error.
6. I tried removing the VLAN, setting up a 1G port (Broadcom instead of Intel in this case), and adding the VLAN to that interface. I get the same error when trying to transfer files.
7. I rebooted all PCs and servers involved and tried again. Same error.
8. I tried connecting to a 48 port Unifi Pro switch instead of the 10G switch (yes I tagged the ports the same). Same error.
9. I tested all these things on both TrueNAS servers with the same behavior.
10. I did a network capture with Wireshark but not exactly sure how to interpret the data in there. I can provide this to anyone willing to help.
Another thing I find odd is that the web interface seems to time out abnormally fast when connected over the VLAN interface. I didn't notice this when not using VLANs. I'm baffled at this point and about to give up on using VLANs with TrueNAS. It's sticking in my craw though that I can't figure this out. Anyone have some suggestions here?

 

[Samuel Tai]

Are you LAGGing both ports of the x540 together? Is your switchport expecting all traffic to be tagged, or do you have an untagged native VLAN? On the TrueNAS side, only VLAN interfaces are tagged, so this could be tagged vs untagged mismatch.

 

[coolrunnings]

Are you LAGGing both ports of the x540 together? Is your switchport expecting all traffic to be tagged, or do you have an untagged native VLAN? On the TrueNAS side, only VLAN interfaces are tagged, so this could be tagged vs untagged mismatch.

No LAGG. The switch port is set up with VLAN1 as native (untagged) and all other VLANs tagged. I do have an IP address assigned to the physical interface in subnet for my native VLAN and an address assigned to the VLAN in the subnet for that VLAN. What's not computing for me is why pinging worked and access to the web interface worked but not SMB.

 

[Samuel Tai]

No LAGG. The switch port is set up with VLAN1 as native (untagged) and all other VLANs tagged. I do have an IP address assigned to the physical interface in subnet for my native VLAN and an address assigned to the VLAN in the subnet for that VLAN. What's not computing for me is why pinging worked and access to the web interface worked but not SMB.

OK, then you've got a routing problem. Are your SMB clients in the same VLAN as your SMB VLAN? Do you have multiple default gateways defined on TrueNAS? It's very likely you're dealing with an asymmetric routing scenario, where traffic to TrueNAS for SMB is going out a different path to the clients. If you can, try forcing TrueNAS to only use the SMB VLAN for everything.

 

[coolrunnings]

For current testing purposes I've got a machine on the native VLAN (VLAN1) and a machine on another VLAN (in this case VLAN10). Once VLANs are added to the interface in TrueNAS I can't get SMB to work on any VLAN or on the native VLAN.

If it's a routing issue, any idea why ping and the web interface would work?

My understanding was that you could only have a single default gateway assigned in TrueNAS. I
Here's how my network is set up:
VLAN1 - 10.0.0.0/24
VLAN10 - 10.50.10.0/24
VLAN20 - 10.50.20.0/24
etc.
Router is at .1 for each interface in pfSense.
TrueNAS is set to have a default gateway of 10.0.0.1
Physical interface (ix0) is set up with an IP of 10.0.0.153/24
VLAN 20 is set up with an IP of 10.50.20.5 and is set up with a parent interface of ix0.

I can ping from a client on the 10.0.0.0/24 network without problem and can access the web interface from that client going to 10.0.0.153 or 10.50.20.5.

From another client on VLAN10 I can also ping and pull up the web interface at 10.0.0.153 and 10.50.20.5.

From either client I can't get SMB to work correctly. At times after a reboot it will work for maybe 30 seconds then just stop all together and give the error I mentioned above. Since pinging works, I'm at a loss how to troubleshoot it as a routing issue... Sorry to be verbose. Hoping the detail will clarify. Thank you for your help.

 

[Samuel Tai]

Definitely a routing issue, as I suspect pfSense doesn’t have routes nor connections to VLANs 10 & 20. Your DNS server probably also doesn’t have entries for TrueNAS’s other IPs.

 

[coolrunnings]

pfSense has both VLANs 10 and 20 set up with appropriate rules allowing traffic across each. Any other devices on each VLAN can ping across each and access data across each.

DNS shouldn't come into play as we're strictly using IP addresses not hostnames at this stage.

If I connect a hypervisor (Hyper-V or Proxmox) to this switchport with this same profile it works exactly as expected. If it were a routing issue outside of TrueNAS, how could that be?

How do I troubleshoot this from within TrueNAS? Do you see anything incorrect with how I've laid out my configuration that stands out?

 

[Elliot Dierksen]

You did not list the IP address(es) that TrueNAS is using, but I am guessing that it has IP addresses on several VLAN's. I suspect your issue is that some of the traffic is going through the firewall, and some of it is not. The client may send the traffic through the firewall, but TrueNAS will respond from the interface on a directly connected network. The firewall will reset the connection because it doesn't see all the traffic. You can test this out by having your TrueNAS only use a single IP address on one VLAN.

 

[coolrunnings]

You did not list the IP address(es) that TrueNAS is using, but I am guessing that it has IP addresses on several VLAN's. I suspect your issue is that some of the traffic is going through the firewall, and some of it is not. The client may send the traffic through the firewall, but TrueNAS will respond from the interface on a directly connected network. The firewall will reset the connection because it doesn't see all the traffic. You can test this out by having your TrueNAS only use a single IP address on one VLAN.

In my testing I have an IP address on the LAN (native VLAN) and on at least one VLAN. The idea was to make the storage accessible on the same VLANs my systems use so they wouldn't have to traverse the firewall to get to the data on the file server. However, even having VLANs on TrueNAS at all means that the interface they are associated with doesn't work correctly with SMB. If I have a VLAN set up on ix0, then I try to access the files on TrueNAS from the native, non VLAN'ed IP from a machine on the same subnet, it doesn't work. If I remove the VLAN with its address, then everything works. That seems counter-intuitive to me.

 

[Elliot Dierksen]

My network is similar to yours. I have 3 main VLAN's which are my home network, my work network, and a storage network. I have a firewall that does the routing between them. Nothing on the storage network uses SMB, only NFS. SMB does listen on all interfaces, but there are no clients on the storage network. I can do this by name because I have different name to IP zones configured in BIND so it always gives the local interface IP to the clients. As @Samuel Tai pointed out, this is a routing problem. Primarily it is a routing problem because of the firewall, and the firewall is only seeing one side of the connection. If this were a router or layer 3 switch, it wouldn't matter. Even with traffic "permitted", the firewall still expects to see both sides of the connection. If it doesn't, it will reset the connection. Period, end of story. I have worked with firewalls from Cisco, Firepower, Check Point, Watchguard and probably some others I can't remember. They all work that way by design. The fact that it works when you test using only a single IP illustrates this. ICMP (ping) works because it is connection-less. SMB is using TCP, so additional rules apply.

I think what is tripping you up is you are thinking that all traffic will go to the default gateway. That is true ONLY if it doesn't match some other route. Longest match is the first criteria on IP routing. I'll give you an example from my network. The routing table is below.

Code:

freenas2% netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.180.3.1         UGS       lagg1
10.180.3.0/24      link#7             U         lagg1
10.180.3.27        link#7             UHS         lo0
127.0.0.1          link#5             UH          lo0
192.168.250.0/24   link#4             U          cxl1
192.168.250.27     link#4             UHS         lo0
192.168.252.0/24   link#3             U          cxl0
192.168.252.27     link#3             UHS         lo0
192.168.253.0/24   link#8             U         vlan1
192.168.253.27     link#8             UHS         lo0

If a client on 10.180.3.0/24 tries to make a connection to connect to the 192.168.253.27 IP of freenas2, that will fail. Traffic from 10.180.3.0 is permitted to 192.168.253.0, but the return traffic would go out the directly connected 10.180.3.27 interface. The firewall would not see both sides of the TCP 3 way handshake, and would reset the connection.

 

[coolrunnings]

Thanks for pointing me in the right direction. What was actually tripping me up is that I was using the native VLAN and the default gateway was on that same interface (in my case 10.0.0.1). When adding VLANs, I isolated it that I could access the server when on the same VLAN but not on the native VLAN. I tried putting the default gateway on a different VLAN and now I can access files from the native one as well as each other additional VLANs. Thank you for the in-depth explanation on how this works. I realize I need to spend more time learning networking cause this threw me pretty good! I'm planning to use a pihole for my DNS locally here. It'll be interesting to see if I can figure out how to accomplish what you're doing with the IP zones giving the local IP based on which network one is coming from. Thanks again!

 

[Elliot Dierksen]

You could probably google this, but here is an excerpt from my named.conf file showing how I use different zone files based on the source IP of the client.

Code:

acl home {
        192.168.253.0/24;
};

acl ebd-work {
        10.180.3.0/24;
};

view "home" {
        match-clients    { 127.0.0.0/8; home; };

        zone "oau.org"  {
                type master;
                file "master/oau.db";
        };
};

view "ebd-work" {
        match-clients    { ebd-work; };

        zone "oau.org"  {
                type master;
                file "master/oau.10.db";
        };
};

 

[coolrunnings]

That is helpful. Thanks!