File transfer issues between machines on different vLANs

[mrripley]

Over the last two days I've been working on rearchitecting my network. I'm configured multiple vLANs, setup a LAGG group for FreeNAS and am moving VMs and physical machines to the newly created vLANs. So far everything has gone relatively well. Today though I ran into the first issue that I haven't been able to solve. There is an issue transferring data between FreeNAS and any machine, physical or virtual, when they aren't on the same vLAN. At first I assumed that this was an issue with intra-vLAN routing and was the fault of pfSense. However I can transfer files between other machines that are in different vLANs without issue, just not FreeNAS. This seems to rule out routing as the issue.

When starting a transfer initially everything looks fine:

But then this will happen:

The speed drops to basically nothing, the transfer stalls. Eventually leading to this:

Hitting Try Again never succeeds. It will continue to die at the same point, within 1-2% every time:

If I do the transfer on the same vLAN it completes with no issues:

I've searched on all the topics I can think of but haven't been able to find anything that is similar. There's no new hardware involved. The FreeNAS box is physical with a Supermicro board, Intel NICs. The switch is a Cisco 3560. The pfSense box is also a physical box, also Supermicro/Intel. The VMs run on Supermicro blades with Intel NICs. My desktop is an ASUS board with Intel NIC. So there shouldn't be an issue with hardware. I'm at a total loss.

Network setup:

 

[hugovsky]

Can you copy if connected directly to freenas?

 

[Elliot Dierksen]

What are the source and destination IP addresses, and what does the port-channel interface look like in the 3560? What protocol is the LAGG using (LACP, etc)?

 

[mrripley]

Can you copy if connected directly to freenas?

I'm not sure what you mean by directly connected. Can you elaborate?

What are the source and destination IP addresses, and what does the port-channel interface look like in the 3560? What protocol is the LAGG using (LACP, etc)?


interface Port-channel1
description FreeNAS LACP Group
switchport access vlan 11
switchport trunk encapsulation dot1q
switchport mode trunk



interface GigabitEthernet0/1
description FreeNAS
switchport access vlan 11
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet0/2
description FreeNAS
switchport access vlan 11
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!



#sho lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi0/1 SA 32768 0025.90da.8236 14s 0x0 0x8B 0x1 0x3D
Gi0/2 SA 32768 0025.90da.8236 11s 0x0 0x8B 0x2 0x3D



#sh etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port


Number of channel-groups in use: 6
Number of aggregators: 6

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) LACP Gi0/1(P) Gi0/2(P)


FreeNAS IPs
- 172.16.10.25
- 172.16.11.25

Tested transfers to
- 172.16.10.3
- 172.16.11.53

172.16.10.25 -> 172.16.10.3 Works
172.16.11.25 -> 172.16.10.3 Fails

172.16.11.25 -> 172.16.11.53 Works
172.16.10.25 -> 172.16.11.53 Fails

I should also mention that copying anything to FreeNAS cross vLAN fails almost immediately. It doesn't even transfer some of the data before failing.

 

[Elliot Dierksen]

FreeNAS IPs
- 172.16.10.25
- 172.16.11.25

Tested transfers to
- 172.16.10.3
- 172.16.11.53

172.16.10.25 -> 172.16.10.3 Works
172.16.11.25 -> 172.16.10.3 Fails

172.16.11.25 -> 172.16.11.53 Works
172.16.10.25 -> 172.16.11.53 Fails

I should also mention that copying anything to FreeNAS cross vLAN fails almost immediately. It doesn't even transfer some of the data before failing.

What is doing your routing between VLAN's? You mentioned pFsense as a firewall. If so, that is the problem. Firewalls HATE HATE HATE only seeing one side of the conversation, and that is what is happening. For the sake of argument, I am going to assume that the firewall has IP's 172.16.10.1 and 172.16.11.1.

In the first failure scenario, host 172.16.10.3 send its request for 172.16.11.25 to its default gateway of 172.16.10.1. That gateway forwards that request out its 172.16.11.1 interface to 172.16.11.25 (FreeNAS). FreeNAS responds to 172.16.10.3 out its 172.16.10.25 interface because it is a directly connected interface. The firewall only sees half the conversation, it eventually resets or drops the connection. I bet you will find that inter-vlan connections work if you drop one of the IP addresses on FreeNAS.

 

[hugovsky]

Can you copy if connected directly to freenas?

I meant what happens if you connect freenas and desktop PC with one cable. NIC to NIC. I was suspecting of what @Elliot Dierksen just said. Routing vlans thru firewall... always a bad ideia. I've been there also.

 

[mrripley]

What is doing your routing between VLAN's? You mentioned pFsense as a firewall. If so, that is the problem. Firewalls HATE HATE HATE only seeing one side of the conversation, and that is what is happening. For the sake of argument, I am going to assume that the firewall has IP's 172.16.10.1 and 172.16.11.1.

In the first failure scenario, host 172.16.10.3 send its request for 172.16.11.25 to its default gateway of 172.16.10.1. That gateway forwards that request out its 172.16.11.1 interface to 172.16.11.25 (FreeNAS). FreeNAS responds to 172.16.10.3 out its 172.16.10.25 interface because it is a directly connected interface. The firewall only sees half the conversation, it eventually resets or drops the connection. I bet you will find that inter-vlan connections work if you drop one of the IP addresses on FreeNAS.

I meant what happens if you connect freenas and desktop PC with one cable. NIC to NIC. I was suspecting of what @Elliot Dierksen just said. Routing vlans thru firewall... always a bad ideia. I've been there also.

Thank you guys. Seems like you both had the right idea. Removing one interface allowed me to transfer files between FreeNAS and a machine in a different vLAN. I have to rethink my plan a little bit with regards to how I'm going to handle drive mappings, I won't be able to use a single DNS entry, but at least I don't have to scrap all the work I've already done.