Weird TCP errors, FreeNAS-11.1-U4, Supermicro X11 and Intel NICs

[rungekutta]

Ok I need help...

Setup: Supermicro X11SSM-F, Intel i3, 32GB ECC Samsung RAM
2x onboard Intel NICs + 1x IPMI
3x WD RED + 3x Seagate NAS 3TB in RAIDZ2
FreeNAS-11.1-U4 running on bare metal.

So I recently replaced my humble Airport Extreme router with OpnSense running on a fanless Qotom Q355G4 MiniPC with 4 Intel NICs. Idea being to increase performance (I've got 1Gb WAN) and enable a LAN/DMZ setup to isolate home LAN from VMs with ports open to the Internet.

The FreeNAS machine has one NIC on the LAN and one on the DMZ. Later and when I'm done it will probably be on LAN only, although I still need to access some NFS shares from the DMZ so not quite decided yet.

Anyhow so far so good except I've run into a very odd network problem which seems isolated to FreeNAS. Weird TCP errors and/or dropped packets.

When I iperf on same subnet or across subnets (through OpnSense (LAN and DMZ, vice versa, for testing)) I get consistent >900Mbit as expected, when I try between MacOS and Windows clients vs Linux VMs (in ESXi). So the network infrastructure and the firewalling (across subnets) looks fine. However as FreeNAS gets involved, things get weird.

iPerf in UDP mode (-u) looks ok. However with TCP things go horribly wrong (in this case from MacOS client):


foo$ ./iperf3 -c 192.168.200.10
Connecting to host 192.168.200.10, port 5201
[ 4] local 192.168.1.174 port 49559 connected to 192.168.200.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 210 KBytes 1.71 Mbits/sec
[ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.01-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.00-5.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.01-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.00-7.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.01-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.00-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.01-10.00 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 210 KBytes 172 Kbits/sec sender
[ 4] 0.00-10.00 sec 65.0 KBytes 53.3 Kbits/sec receiver


(DMZ is 192.168.1.0/24, LAN is 192.168.200.0/24)

iPerf against the NIC on the same subnet however, also in the case of FreeNAS, behaves as expected. So the problems only occur when crossing subnets, in either direction.

Similarly, if I try to scp a file across subnets and FreeNAS is involved, the transfer stalls and times out but not always immediately. And I can connect to the web front end, so some stuff does get through.

I have the following tunables, to set Cubic instead of newreno and slightly larger default send and receive buffers:

net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.recvspace=613200
net.inet.tcp.sendspace=613200


Haven't tried reverting back to newreno but tried messing with the buffers including the more conservative defaults, but no difference.

I've disabled pf scrubbing in OpnSense and trimmed it down to a pretty minial setup. It runs on FreeBSD 11.1 like FreeNAS.

I'm at a total loss (no pun intended!) from here... so would greatly appreciate some pointers for further troubleshooting!

Many thanks in advance.

 

[rogerh]

So if you are trying to connect from subnet A to the NIC on the FreeNAS box which is on subnet B, then presumably your initial connection goes via the router. But when FreeNAS TCP stack wants to reply to a packet you have sent from a host on subnet A, is it not going to try and send it via FreeNAS's subnet A NIC? Which will upset your original host on subnet A because it is waiting for a packet from FreeNAS's subnet B NIC?

I may have this entirely wrong, but it seems to me you can't easily contact a host with two subnets on the "wrong" one for the sending host - why do you want to?

 

[c32767a]

So if you are trying to connect from subnet A to the NIC on the FreeNAS box which is on subnet B, then presumably your initial connection goes via the router. But when FreeNAS TCP stack wants to reply to a packet you have sent from a host on subnet A, is it not going to try and send it via FreeNAS's subnet A NIC? Which will upset your original host on subnet A because it is waiting for a packet from FreeNAS's subnet B NIC?

I may have this entirely wrong, but it seems to me you can't easily contact a host with two subnets on the "wrong" one for the sending host - why do you want to?

You are more or less correct. The IP addressing will be "correct", but the packets will come back through an unexpected path. And that exposes the real issue which is that there's a stateful firewall on one of the 2 paths, so it's likely UDP would pass through apparently "OK"(stateless) and TCP would experience nondeterministic behavior based on which way the session is going (stateful).

Short answer, don't make the kernel's IP stack do unnatural things.. Use the interface IPs on the subnet that's local to your clients.

 

[rogerh]

I remain to be convinced that a host which receives a packet on nicA and replies from nicB will use nicA's as the from address. If it doesn't, then the packet will appear to the original sender (to nicA) as irrelevant. The host is not routing, just originating a packet here. BICBW

 

[rungekutta]

Thanks very much for the help guys, believe that was exactly what was going on. Ended up leaving FreeNAS with one nic on dmz and one on lan but most services (ssh, smb) only listen on lan. Nfs is open in dmz but with access restricted to a few hosts that need it. And all access to FreeNAS directly on the relevant interface without any routing. Thanks!