Troubleshooting megabit write speeds on gigabit link

[heater]

I have setup my first FreeNAS (FreeNAS 11) box with the components listed here.

I have configured a single RAIDz2 volume for the pool out of my 8x4TB WD reds. My current issue, I believe, is around network speeds, but if someone more knowledgeable feels otherwise, I am all ears. Currently I am experiencing proper gigabit speeds when reading from the server, but when writing to the server I am lucky to hit 120 Mbits/sec.

Here's the output from iperf

Code:

[root@freenas ~]# iperf -s													
------------------------------------------------------------				
Server listening on TCP port 5001											
TCP window size: 64.0 KByte (default)										
------------------------------------------------------------				
[  4] local 192.168.1.118 port 5001 connected with 192.168.1.180 port 57641	
[ ID] Interval	   Transfer	 Bandwidth									
[  4]  0.0-10.1 sec  83.4 MBytes  69.3 Mbits/sec							

[root@freenas ~]# iperf -c 192.168.1.180									
------------------------------------------------------------				
Client connecting to 192.168.1.180, TCP port 5001							
TCP window size: 32.8 KByte (default)										
------------------------------------------------------------				
[  3] local 192.168.1.118 port 44816 connected with 192.168.1.180 port 5001	
[ ID] Interval	   Transfer	 Bandwidth									
[  3]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

Which shows pretty sad write speeds if I have interpreted this test correctly. I have done the following to also troubleshoot my problem

• Swapped out the cable (cate5e and cat6) between the nas and my router (Linksys WRT3200ACM), with the same result as above
• Ran the same iperf test between a different computer and the nas, with the same results as above.
• Ran iperf between the two computers that are not the nas, and this resulted gigabit speeds in both directions (I can post this output if necessary, but did not feel it added any benefit)
• Added an additional gigabit switch to the router, and connected the nas and other two machines to this switch. When running the iperf test from each computer to the nas, the same problematic results occurred
• ifconfig on the nas shows the following

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500	
	   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS
UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>								
	   ether ac:1f:6b:1d:24:2a												
	   nd6 options=9<PERFORMNUD,IFDISABLED>									
	   media: Ethernet autoselect											
	   status: no carrier													
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500	
	   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS
UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>								
	   ether ac:1f:6b:1d:24:2b												
	   inet 192.168.1.119 netmask 0xffffff00 broadcast 192.168.1.255		
	   nd6 options=9<PERFORMNUD,IFDISABLED>									
	   media: Ethernet autoselect (1000baseT <full-duplex>)					
	   status: active

Note: Both non-nas systems are running Windows 7. One has an Intel gigabit NIC (built into motherboard), and the other has a Killer gigabit NIC (built into motherboard). I tried both as I've read Killer doesn't have the best reputation, but same results on both.

Unrelated to iperf tests, I did also run local speed tests against the machine to ensure it wasn't my pool. I created an uncompressed dataset and ran the following commands

Code:

[root@freenas /mnt/test_vol/uncompressed]# dd if=/dev/zero of=/mnt/test_vol/uncompressed
/test.dd bs=2048k count=50k													
51200+0 records in															
51200+0 records out															
107374182400 bytes transferred in 174.633327 secs (614855045 bytes/sec)	

[root@freenas /mnt/test_vol/uncompressed]# dd of=/dev/null if=/mnt/test_vol/uncompressed
/test.dd bs=2048k count=50k													
51200+0 records in															
51200+0 records out															
107374182400 bytes transferred in 114.284731 secs (939532176 bytes/sec)

Lastly, I ran CrystalDiskMark5 against the NAS when mounted as a drive. The results of this are attached

Other than the tests I've ran above, I am not sure where to begin to find the root cause of the issue. Any help or direction is greatly appreciated.

Update 1:
----------------------------------------------------------------------------------------------
This morning, after doing nothing to the system I am seeing faster speeds. Large transfers, ~20GB start very fast, 120MB/s then taper off around 70-88MB/s over SMB. The iperf stats still remain consistent (I am not sure how that's possible). Also I have run tcpdump and see the following

Code:

tcpdump -i igb1 -nn -vv host 192.168.1.180 > /mnt/vol_name/tcpdump.txt

11:06:23.339944 IP (tos 0x0, ttl 128, id 14642, offset 0, flags [DF], proto TCP (6), length 132)									
   192.168.1.180.49548 > 192.168.1.119.445: Flags [P.], cksum 0x6bd7 (correct), seq 882:974, ack 941, win 16079, length 92 SMB-over
-TCP packet:(raw data or continuation?)																							
																																
11:06:23.340114 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 168)										
   192.168.1.119.445 > 192.168.1.180.49548: Flags [P.], cksum 0x8516 (incorrect -> 0x6144), seq 941:1069, ack 974, win 1026, length
 128 SMB-over-TCP packet:(raw data or continuation?)																				
																																
11:06:23.340765 IP (tos 0x0, ttl 128, id 14643, offset 0, flags [DF], proto TCP (6), length 261)									
   192.168.1.180.49548 > 192.168.1.119.445: Flags [P.], cksum 0xaf41 (correct), seq 974:1195, ack 1069, win 16425, length 221 SMB-o
ver-TCP packet:(raw data or continuation?)																						
																																
11:06:23.340942 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 244)										
   192.168.1.119.445 > 192.168.1.180.49548: Flags [P.], cksum 0x8562 (incorrect -> 0x67b5), seq 1069:1273, ack 1195, win 1026, leng
th 204 SMB-over-TCP packet:(raw data or continuation?)

.119 is the NAS, and .180 is my Windows 7 machine.

From what I have researched online the SMB problem is not unique to me, but I am unsure if it could be useful info to help find the root cause.

Update 2:
----------------------------------------------------------------------------------------------
I have attempted to transfer a large file, ~10GB over FTP to the service on the local network. It seems both the read and write speeds over FTP have fully saturated the gigabit link. I have attached these screenshots to the thread, but long standing transfers of multiple large files the speeds tapers off to 60-70MB/s range again... It seems the SMB protocol is just slow (which without fine tuning is expected?), but doesn't answer why iperf is reporting such slow speeds for writes.

 

[Ericloewe]

I tried both as I've read Killer doesn't have the best reputation

It's crap. 31337 h4x0r g4m3r-grade crap, with a price tag to match and marketing that would make Monster cable go "geez, tone it down, that's unrealistic". Unfortunately, it's not the cause here, clearly...

It seems the SMB protocol is just slow

On any vaguely acceptable system, SMB can easily push GbE (90+ MB/s at least), so that's not your problem. iPerf results suggest that it is indeed a lower-level networking issue.
Do you have any antimalware solutions running on your clients? If so, try disabling them. You could also try a different OS on the same hardware to see what happens with iPerf.

 

[heater]

On any vaguely acceptable system, SMB can easily push GbE (90+ MB/s at least), so that's not your problem. iPerf results suggest that it is indeed a lower-level networking issue.
Do you have any antimalware solutions running on your clients? If so, try disabling them

I do not have any antimalware solutions running on either system.

You could also try a different OS on the same hardware to see what happens with iPerf.

Using the windows 7 system with the intel NIC, I booted into Kubuntu. Wow... the write speeds of iperf are correct.

Code:

[root@freenas ~]# iperf -s													
------------------------------------------------------------				 
Server listening on TCP port 5001											 
TCP window size: 64.0 KByte (default)										 
------------------------------------------------------------				 
[  4] local 192.168.1.119 port 5001 connected with 192.168.1.135 port 48798	
[ ID] Interval	   Transfer	 Bandwidth									
[  4]  0.0-10.0 sec  1.09 GBytes   934 Mbits/sec

Going off of the notion that maybe windows is the issue, I transferred a 10GB file to and from the NAS from Kubuntu. I fully saturate the gigabit link for the entire duration of the read and write. I never would have guessed (though in retrospect maybe I should have?) that Windows was the limiting factor. What could possibly be the reason for this to happen only with Windows clients? Also I want to ensure I am consistently getting these speeds from Kubuntu before I celebrate, and mark this as solved. Thank you Ericloewe for the nod in what seems the correct direction.

 

[Ericloewe]

I don't have a good working theory. I have noticed that my absurdly beefy workstation (Xeon E5-1650 v3, Intel I210 NIC) has slower-than-expected networking at times and a driver update didn't help last time I was investigating this.