PC can ping FreeNAS, FreeNAS can't ping PC

[Merlijn]

Hi there,

Years ago I used FreeNAS very actively, but as time progressed I started to use software like Napp-IT. Now I am building a new NAS and I thought it would be nice and fun to use FreeNAS again. But the problem is that it isn't going as smooth as I had hoped so I hope some of you guys can help me with some issues.

I have multiple issues, one is that I can't ping my PC from the FreeNAS-server and the other is very slow write and read speeds. But before I elaborate on this, let's give the environment I am testing this in. I hope it's all clear enough, but if something is missing please tell me so I can provide it.

Hardware
The FreeNAS-server consists of the following:

• SuperMicro X11SSH-LN4F
• Intel Core i3-6100
• 16 GB of ECC ram
• 2 x Samsung 840 Evo + Samsung 830 (stripe)
• 1 x Samsung 850 Evo
• Intel X520-DA2 10 gigabit NIC
• Intel FTLX8571D3BCV-IT SFP+ LC module

The PC consists of a Core i5, the same NIC and SFP+ module, a couple of SSD's and a 12GB ramdisk (for testing the 10g). The server and client are directly connected with a OM3 LC-LC cable and are on a different subnet (I use 10.0.0.1/255.0.0.0 on my network while I use 192.168.1.1 (PC) and 192.168.1.2 (server)/255.255.255.0 for the NAS-network.

Problem 1
The second problem of slow write and read speeds might have to do with the ping problem, so let's do ping first.

The PC/client can ping to the FreeNAS-server perfectly. See:

Code:

c:\>ping 192.168.1.2

Pinging 192.168.1.2 with 32 bytes of data:
Reply from 192.168.1.2: bytes=32 time<1ms TTL=64
Reply from 192.168.1.2: bytes=32 time<1ms TTL=64
Reply from 192.168.1.2: bytes=32 time<1ms TTL=64
Reply from 192.168.1.2: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.1.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

When I try to do the same from the FreeNAS-server to the client I get:

Code:

[root@NAS10G] ~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes

--- 192.168.1.1 ping statistics ---
801 packets transmitted, 0 packets received, 100.0% packet loss

These are the outputs of ifconfig / ipconfig:

Code:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS                       UM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:87:2d:98
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active

Code:

Ethernet adapter Ethernet 3:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::ccf6:40e8:9111:eda9%23
   IPv4 Address. . . . . . . . . . . : 192.168.1.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Does anyone know what can be wrong or where to look for the cause of the problem?

Problem 2
Unfortunately I get rather slow read and write speeds from and to the NAS. If I copy a file from a CIFS share to my client then it will max out at around 160 MB/s. The other way around will max at around 260 MB/s.

I use a ramdisk on my client and have 2 volumes in the FreeNAS server. Volume VOL1SSD has one SSD, Volume VOL3SSD has 3 striped SSD's. I was hoping that 3 striped SSD's would be fast enough to saturate the 10g connection, that's why there are three.

Some dd commands:

Code:

Single SSD volume WRITE
[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL1SSD/DS1SSD/ddfile bs=1024k count=20000
20971520000 bytes transferred in 37.189942 secs (563903002 bytes/sec) = 537 MB/s

[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL1SSD/DS1SSD/ddfile bs=2048k count=10000
20971520000 bytes transferred in 37.588777 secs (557919721 bytes/sec) = 532 MB/s

[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL1SSD/DS1SSD/ddfile bs=4096k count=5000
20971520000 bytes transferred in 37.932039 secs (552870885 bytes/sec) = 527 MB/s

Multi SSD volume WRITE
[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL3SSD/DS10G/ddfile bs=1024k count=20000
20971520000 bytes transferred in 22.074732 secs (950023762 bytes/sec) = 906 MB/s

[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL3SSD/DS10G/ddfile bs=2048k count=10000
20971520000 bytes transferred in 21.870604 secs (958890754 bytes/sec) = 914 MB/s

[root@NAS10G] ~# dd if=/dev/zero of=/mnt/VOL3SSD/DS10G/ddfile bs=4096k count=5000
20971520000 bytes transferred in 23.286719 secs (900578563 bytes/sec) = 858 MB/s

So I don't think the storage is the bottleneck, there is something wrong in the network. I read the post of the jgreco about MTU size and that 9000 won't affect performance much but adds a lot of complexity/possible issues and stuff so I have left it at 1500.

I guess it's better to first troubleshoot the network so that it can ping, because maybe the issue with ping are also in one or another way the cause of the slow transfer speed.

Does anyone have some pointers on what I can look at/what I can do?

Thanks in advance!

 

[joeschmuck]

How about giving it a gateway.

 

[Merlijn]

How about giving it a gateway.

Hi,

Thanks for your reply. I probably don't understand, so a few questions. What gateway should it get? I searched on how to make a direct connection between FreeNAS and my client, and things like this turned up:

If you only want a direct connection, you can just connect the 2 machines using UTP cables.
And then just assign them a static IP address in the same subnet, e.g.: 192.168.0.1 & 192.168.0.2 with mask 255.255.255.252

I did that with 192.168.1.1 and 192.168.1.2 and 255.255.255.0. There is no router, so what is the gateway then?

Furthermore, I haven't configured a gateway partially because of the notes in the online manual:

IPv4 Default Gateway IP address typically not set (see NOTE below)

Note: in many cases, a FreeNAS® configuration does not include default gateway information as a way to make it more difficult for a remote attacker to communicate with the server. While this is a reasonable precaution, such a configuration does not restrict inbound traffic from sources within the local network. However, omitting a default gateway will prevent the FreeNAS® system from communicating with DNS servers, time servers, and mail servers that are located outside of the local network.

 

[joeschmuck]

Doh, you may be correct. Do you have a firewall on your PC? Could it be blocking the ping requests?

Just so I understand, you can transfer files back and fourth without issue other than your speed concerns, right? If so then I'd suspect the firewall thing.

I can't give any advice on the throughput thing, I have no first hand knowledge of 10Gb connections but there are some folks here who do have experience and hopefully they will chime in.

 

[SweetAndLow]

Try using iperf to test your network throughout. If ping is failing it sounds like a client firewall issue.

 

[gpsguy]

My guess, is that a firewall on the PC is blocking the pings.

What OS are you running on the PC? If it's Windows 10, have you read/seen this message: https://forums.freenas.org/index.php?threads/x520-speed-issue.43695/page-3#post-298547

I'd suggest you read the entire thread.

 

[Merlijn]

Doh, you may be correct. Do you have a firewall on your PC? Could it be blocking the ping requests?

Just so I understand, you can transfer files back and fourth without issue other than your speed concerns, right? If so then I'd suspect the firewall thing.

I can't give any advice on the throughput thing, I have no first hand knowledge of 10Gb connections but there are some folks here who do have experience and hopefully they will chime in.

I checked Windows firewall (I don't have any other firewall) and all ICMP is allowed in private and domain. My client has two networks (also my local LAN) and through that it can be pinged as you should expect. I don't have any other firewall that I know off. By the way, yes I can transfer files back and forth without issues other then the slowness.

 

[Merlijn]

My guess, is that a firewall on the PC is blocking the pings.

What OS are you running on the PC? If it's Windows 10, have you read/seen this message: https://forums.freenas.org/index.php?threads/x520-speed-issue.43695/page-3#post-298547

I'd suggest you read the entire thread.

Thanks a lot for the handy link, I read the whole topic and used all of the settings in the post you mentionend. Now I can transfer files from the ramdisk (client) to the NAS with 500-850 MB/s (it starts at 850 but drops to ~550), that is a lot better than it was :). The other way around (from the NAS to the ramdisk) is only around 250-300 MB/s though. Do I have to do the same kind of modifications on the NIC in FreeNAS to improve this and if so, where can I do that?

What is weird by the way: the transfer speeds on my single SSD share are a lot more consistent than the stripe with 3 SSD's. the single SSD share is even faster on avarage. How is this possible?

To summerize in speed:
- NAS -> CLIENT from 160 MB/s to 250-300 MB/s (3 SSD stripe share) / 300 MB/s (single SSD share)
- CLIENT -> NAS from 260 MB/s to 500-600 MB/s (3 SSD stripe share) / 600-700 MB/s (single SSD share)

 

[Merlijn]

Try using iperf to test your network throughout. If ping is failing it sounds like a client firewall issue.

Also look at the edits, I made some dumb mistakes when I started with iPerf, but came further later on ;).

Hi and thanks for your response. I did not have any experience with iPerf so I hope I did it right. I installed it on the Windows 8.1 client and allowed port 5001 on both TCP and UDP for inbound and outbound connections in the 'advanced' Windows firewall. I then used the following commands:

- On the server iperf -s
- On the client iperf.exe -c 192.168.1.2
- On the client iperf.exe -s
- On the server: iperf -c 192.168.1.1

In both situations the iPerf server was waiting for a connection while the iPerf client gave errors like:

Code:

D:\Program Files\Applications\iPerf>iperf3.exe -c 192.168.1.2
iperf3: error - unable to connect to server: Connection refused

EDIT 1:
I read on the docs of iPerf that the standard port is 5001, but the Windows application defaults to 5201 so I have to use the -p 5001 parameter in order to get them to communicate. Another handy parameter is '-f m' so you will get it in mega instead of kilo. Will return with (hopefully) some results.

EDIT 2:
With iPerf it's also impossible to connect to the client. The results from the client to FreeNAS are really... euhm... really disappointing :D.

Code:

[root@NAS10G] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 59432
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-14.4 sec  14.0 Bytes  7.76 bits/sec
[  5] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 59439
[  5]  0.0-196.7 sec  14.0 Bytes  0.57 bits/sec

I had to abort the iPerf in CMD (Windows) with ctrl+c because it didn't do anything. According to the manuel there should be reporting intervals every 10 seconds, so the first time I aborted after 14 sec and the second time I tried it for a lot longer. Strangely both are at 14 Bytes, but I don't understand how this is possible since the transfer speeds from the CIFS share are a lot higher and functional.

So I guess that the cause of not being able to iPerf from NAS -> client is the same cause that I can't ping from NAS -> client. But does that also explain the werid iPerf results I have from client to NAS? Does anyone have an idea?

EDIT 3:
I googled and found this thread: https://github.com/esnet/iperf/issues/339
I assumed FreeNAS had iPerf 3 since it is the latest version, but FreeNAS has iPerf 2 so I have to try it again with 2.

By the way, I edit all my findings and stuff so people that might have the same hardware/setup/trouble/learning curve as me can run through it. If you guys find it annoying let me know ;).

EDIT 4:
It worked with version 2! See:

Code:

[root@NAS10G] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ ID] Interval      Transfer    Bandwidth
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 59584
[  4]  0.0-10.0 sec  9.70 GBytes  8.33 Gbits/sec
[  5] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 59589
[  5]  0.0-60.0 sec  62.1 GBytes  8.90 Gbits/sec

Code:

D:\Program Files\Applications\iPerf\iperf-2.0.9-win64>iperf.exe -c NAS10G -p 5001 -f m -t 60 -i 5
------------------------------------------------------------
Client connecting to NAS10G, TCP port 5001
TCP window size: 0.20 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 59589 connected with 192.168.1.2 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 5.0 sec  4564 MBytes  7656 Mbits/sec
[  3]  5.0-10.0 sec  4855 MBytes  8146 Mbits/sec
[  3] 10.0-15.0 sec  5473 MBytes  9183 Mbits/sec
[  3] 15.0-20.0 sec  5452 MBytes  9146 Mbits/sec
[  3] 20.0-25.0 sec  5493 MBytes  9216 Mbits/sec
[  3] 25.0-30.0 sec  5465 MBytes  9168 Mbits/sec
[  3] 30.0-35.0 sec  5445 MBytes  9136 Mbits/sec
[  3] 35.0-40.0 sec  5449 MBytes  9141 Mbits/sec
[  3] 40.0-45.0 sec  5402 MBytes  9063 Mbits/sec
[  3] 45.0-50.0 sec  5436 MBytes  9121 Mbits/sec
[  3] 50.0-55.0 sec  5176 MBytes  8685 Mbits/sec
[  3] 55.0-60.0 sec  5421 MBytes  9095 Mbits/sec
[  3]  0.0-60.0 sec  63634 MBytes  8896 Mbits/sec

The other way around (NAS to client) still does not work. So from the NAS to client ping and iPerf both don't work. So I am back where I started, any suggestions?

 

[Merlijn]

Hi again,

After some sleep I wanted to check the firewall again because three people here referenced that in this thread. I am not that familiair with Windows firewall but I figured out why pings weren't allowered. I allowed all ICMP packages to arrive on domain and private networks, but not on public. Since the connection I have with the NAS is seen by Windows as a public network (and I can't find where to change that) the firewall blocked all traffic. I now allowed ICMP also on public networks so it works. Here are the iPerf results:

Show : NAS -> Client (TCP window size of 64k)

Code:

[root@NAS10G] ~# iperf -c 192.168.1.1 -p 5001 -t 60 -i 5 -w 64k
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 65.0 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.1.2 port 29378 connected with 192.168.1.1 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 5.0 sec  762 MBytes  1.28 Gbits/sec
[  3]  5.0-10.0 sec  780 MBytes  1.31 Gbits/sec
[  3] 10.0-15.0 sec  799 MBytes  1.34 Gbits/sec
[  3] 15.0-20.0 sec  796 MBytes  1.34 Gbits/sec
[  3] 20.0-25.0 sec  792 MBytes  1.33 Gbits/sec
[  3] 25.0-30.0 sec  782 MBytes  1.31 Gbits/sec
[  3] 30.0-35.0 sec  788 MBytes  1.32 Gbits/sec
[  3] 35.0-40.0 sec  799 MBytes  1.34 Gbits/sec
[  3] 40.0-45.0 sec  783 MBytes  1.31 Gbits/sec
[  3] 45.0-50.0 sec  799 MBytes  1.34 Gbits/sec
[  3] 50.0-55.0 sec  802 MBytes  1.35 Gbits/sec
[  3] 55.0-60.0 sec  797 MBytes  1.34 Gbits/sec
[  3]  0.0-60.0 sec  9.26 GBytes  1.33 Gbits/sec

Show : NAS -> Client (TCP window size of 512k)

Code:

[root@NAS10G] ~# iperf -c 192.168.1.1 -p 5001 -t 60 -i 5 -w 512k
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.168.1.2 port 29376 connected with 192.168.1.1 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 5.0 sec  3.61 GBytes  6.19 Gbits/sec
[  3]  5.0-10.0 sec  3.64 GBytes  6.25 Gbits/sec
[  3] 10.0-15.0 sec  3.64 GBytes  6.26 Gbits/sec
[  3] 15.0-20.0 sec  3.65 GBytes  6.27 Gbits/sec
[  3] 20.0-25.0 sec  3.64 GBytes  6.26 Gbits/sec
[  3] 25.0-30.0 sec  3.65 GBytes  6.26 Gbits/sec
[  3] 30.0-35.0 sec  3.64 GBytes  6.26 Gbits/sec
[  3] 35.0-40.0 sec  3.64 GBytes  6.26 Gbits/sec
[  3] 40.0-45.0 sec  3.65 GBytes  6.26 Gbits/sec
[  3] 45.0-50.0 sec  3.60 GBytes  6.18 Gbits/sec
[  3] 50.0-55.0 sec  3.62 GBytes  6.21 Gbits/sec
[  3] 55.0-60.0 sec  3.62 GBytes  6.22 Gbits/sec
[  3]  0.0-60.0 sec  43.6 GBytes  6.24 Gbits/sec

Show : Client -> NAS (TCP window size of 64k)

Code:

D:\Program Files\Applications\iPerf\iperf-2.0.9-win64>iperf.exe -c NAS10G -w 64k
-i 5 -t 60 -p 5001
------------------------------------------------------------
Client connecting to NAS10G, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[  3] local 192.168.1.1 port 63272 connected with 192.168.1.2 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 5.0 sec  5.46 GBytes  9.37 Gbits/sec
[  3]  5.0-10.0 sec  5.36 GBytes  9.21 Gbits/sec
[  3] 10.0-15.0 sec  5.39 GBytes  9.26 Gbits/sec
[  3] 15.0-20.0 sec  5.36 GBytes  9.21 Gbits/sec
[  3] 20.0-25.0 sec  5.34 GBytes  9.17 Gbits/sec
[  3] 25.0-30.0 sec  5.39 GBytes  9.27 Gbits/sec
[  3] 30.0-35.0 sec  5.40 GBytes  9.27 Gbits/sec
[  3] 35.0-40.0 sec  5.36 GBytes  9.20 Gbits/sec
[  3] 40.0-45.0 sec  5.42 GBytes  9.31 Gbits/sec
[  3] 45.0-50.0 sec  5.40 GBytes  9.28 Gbits/sec
[  3] 50.0-55.0 sec  5.42 GBytes  9.31 Gbits/sec
[  3] 55.0-60.0 sec  5.42 GBytes  9.32 Gbits/sec
[  3]  0.0-60.0 sec  64.7 GBytes  9.27 Gbits/sec

Show : Client -> NAS (TCP window size of 512k)

Code:

D:\Program Files\Applications\iPerf\iperf-2.0.9-win64>iperf.exe -c NAS10G -w 512
k -i 5 -t 60 -p 5001
------------------------------------------------------------
Client connecting to NAS10G, TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  3] local 192.168.1.1 port 63244 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  5.51 GBytes  9.47 Gbits/sec
[  3]  5.0-10.0 sec  5.45 GBytes  9.36 Gbits/sec
[  3] 10.0-15.0 sec  5.43 GBytes  9.33 Gbits/sec
[  3] 15.0-20.0 sec  5.44 GBytes  9.35 Gbits/sec
[  3] 20.0-25.0 sec  5.45 GBytes  9.37 Gbits/sec
[  3] 25.0-30.0 sec  5.49 GBytes  9.44 Gbits/sec
[  3] 30.0-35.0 sec  5.44 GBytes  9.34 Gbits/sec
[  3] 35.0-40.0 sec  5.33 GBytes  9.15 Gbits/sec
[  3] 40.0-45.0 sec  5.51 GBytes  9.46 Gbits/sec
[  3] 45.0-50.0 sec  5.43 GBytes  9.33 Gbits/sec
[  3] 50.0-55.0 sec  5.47 GBytes  9.39 Gbits/sec
[  3] 55.0-60.0 sec  5.47 GBytes  9.40 Gbits/sec
[  3]  0.0-60.0 sec  65.4 GBytes  9.37 Gbits/sec

To summerize, there is a huge performance difference between NAS -> Client vs. Client -> NAS:

NAS -> Client: 1.33 Gbit (64k) & 6.24 Gbit (512k)
Client -> NAS: 9.27 Gbit (64k) & 9.37 Gbit (512k)

How can this difference be this large? It might explain why the CIFS transfer speeds differ just as much. I don't know if my thoughts are correct but I did tune the Windows 8.1 NIC adapter settings (with thanks to gpsguy and Stryf), do I also have to do some tuning on the FreeNAS end in order to get more consistent and comparable results from and to both devices? If so, where/how can I do that?

Thanks in advance!

 

[gpsguy]

You might try the settings in this message: https://forums.freenas.org/index.ph...rdware-smb-cifs-bottleneck.42641/#post-277350

 

[JDCynical]

Since the connection I have with the NAS is seen by Windows as a public network (and I can't find where to change that) the firewall blocked all traffic.

I forget the criteria windows uses to determine if a network is 'public' vs 'private' (I do know it's not strictly based on IP range), but this may help with that particular irritation:

http://windowsitpro.com/powershell/how-force-network-type-windows-using-powershell

 

[Jetman77]

@Merlijn in case someone comes along this old thread on the interwebs
Your iperf3 -w arg saved me from replacing perfectly good cards that would only show a max 5.42 gbps

.100 is freenas --> .116 is esxi

Code:

root@freenas[~]# iperf3 -c 192.168.5.116 -B 192.168.5.100 -P 1     
Connecting to host 192.168.5.116, port 5201
[  5] local 192.168.5.100 port 39042 connected to 192.168.5.116 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   618 MBytes  5.18 Gbits/sec    0    262 KBytes     
[  5]   1.00-2.00   sec   646 MBytes  5.42 Gbits/sec    0    262 KBytes     
...
[  5]   9.00-10.00  sec   646 MBytes  5.42 Gbits/sec    0    262 KBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.28 GBytes  5.40 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  6.28 GBytes  5.40 Gbits/sec                  receiver

[root@R630ESXi6:/usr/lib/vmware/vsan/bin] ./iperf3.copy -s -B 192.168.5.116
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.5.100, port 40560
[  5] local 192.168.5.116 port 5201 connected to 192.168.5.100 port 39869
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   517 MBytes  4.34 Gbits/sec                 
iperf3: getsockopt - Function not implemented
...
[  5]   9.00-10.00  sec   646 MBytes  5.42 Gbits/sec                 
iperf3: getsockopt - Function not implemented
[  5]  10.00-10.15  sec  98.4 MBytes  5.42 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.15  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.15  sec  6.28 GBytes  5.31 Gbits/sec                  receiver

esxi .116 --> freenas .100

Code:

[root@R630ESXi6:/usr/lib/vmware/vsan/bin] ./iperf3.copy -c 192.168.5.100 -B 192.168.5.116
Connecting to host 192.168.5.100, port 5201
[  4] local 192.168.5.116 port 16586 connected to 192.168.5.100 port 5201
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.03 GBytes  8.89 Gbits/sec  8634728   0.00 Bytes     
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec    0   0.00 Bytes     
iperf3: getsockopt - Function not implemented
...
[  4]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   0.00 Bytes     
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec  4286332568   0.00 Bytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.4 GBytes  9.79 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.4 GBytes  9.79 Gbits/sec                  receiver
root@freenas[~]# iperf3 -s -B 192.168.5.100             
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.5.116, port 25170
[  5] local 192.168.5.100 port 5201 connected to 192.168.5.116 port 16586
[ ID] Interval           Transfer     Bitrate
...           
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec                 
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec                 
[  5]  10.00-10.00  sec   291 KBytes  9.81 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.4 GBytes  9.79 Gbits/sec                  receiver

@jgreco Now the question is (Like you stated @Merlijn): Is there a FreeNAS (now 11.3) tunable to insure that it can create TCP windows at least 1024K? (I have no idea if this is somehow controlled by the OS and how dynamic it is...)

My cards working fine:

 

[Jetman77]

For ref: https://netbeez.net/blog/tcp-window-size/
Will post back if I figure out how to do this in FreeBSD (FreeNAS)

 

[Jetman77]

net.inet.tcp.inflight.enable=0
loader

TURN IT OFF!

11.11.2.2. TCP Bandwidth Delay Product
TCP bandwidth delay product limiting can be enabled by setting the net.inet.tcp.inflight.enable sysctl(8) variable to 1. This instructs the system to attempt to calculate the bandwidth delay product for each connection and limit the amount of data queued to the network to just the amount required to maintain optimum throughput.