Slow transfer rates again.

[Poppa]

So a few days ago, FreeNAS started limiting my transfers to about 36MB/s again. I just did yet another fresh install of FreeNAS but this time it did not fix the problem. My FreeNAS machine is now connected directly to my primary PC with a Mellanox ConnectX-2 10Gbe card and I should be seeing speeds well north of 200MB/s.

This is outrageous and is kinda pissing me off.

 

[dlavigne]

Build version (from System -> Information)? Which type of transfers (eg which sharing protocol or command)?

Also, post the output of ifconfig from within code tags.

 

[Poppa]

FreeNAS-11.0-U2 (e417d8aa5)

SMB Min2.10-Max3

Code:

igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500		 
		options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS
UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>								
		ether 00:25:90:91:5a:82												 
		nd6 options=9<PERFORMNUD,IFDISABLED>									
		media: Ethernet autoselect (1000baseT <full-duplex>)					
		status: active														  
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500		 
		options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS
UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>								
		ether 00:25:90:91:5a:83												 
		nd6 options=9<PERFORMNUD,IFDISABLED>									
		media: Ethernet autoselect											  
		status: no carrier													  
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384			   
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>				   
		inet6 ::1 prefixlen 128												 
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3							  
		inet 127.0.0.1 netmask 0xff000000									   
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>							   
		groups: lo															  
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500	
		options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS
UM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>	
		ether 00:02:c9:51:5d:60												 
		inet 192.168.1.50 netmask 0xffffff00 broadcast 192.168.1.255			
		nd6 options=9<PERFORMNUD,IFDISABLED>									
		media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)   
		status: active

 

[Poppa]

So any idea about what the hell is going on?

 

[Ericloewe]

You're not exactly giving us that much to go on. What does iperf say?

 

[Poppa]

You're not exactly giving us that much to go on. What does iperf say?

Not you again. You are obsessed with iperf, it is completely useless, let it go already.

I gave the information that @dlavigne asked for which should be enough to go on. I did not run iperf because @dlavigne did not ask me to and I am not going to because we have already done this song and dance and it did not help one bit and ended up with you saying to just do a clean install which did help that time but not this time and I am not going to keep reinstalling FreeNAS.

So please, what the hell is wrong with FreeNAS?

 

[Ericloewe]

You're complaining that you can't transfer at the speeds you'd like and yet you refuse to provide proof that your network is not to blame?

I gave the information that @dlavigne asked for which should be enough to go on.

No, it isn't. It's the bare minimum for anyone to have a vague idea of what you're doing.

 

[Poppa]

You're complaining that you can't transfer at the speeds you'd like and yet you refuse to provide proof that your network is not to blame.

The proof is that after having these issues yet again I decided to yet again use OMV and was able to transfer files at a sustained speed of 160MB/s, still lower than I would like for a 10Gbe connection but way faster than FreeNAS. My hardware is not the issue this time or any time, FreeNAS is clearly to blame.

No, it isn't. It's the bare minimum for anyone to have a vague idea of what you're doing.

I would be happy to provide a more detailed report if you can tell me how.


Has any of that information I provided to @dlavigne alluded to a possible problem with FreeNAS?

 

[wblock]

Assuming "OMV" is OpenMediaVault, was the share there also SMB? Samba is frequently CPU-bound.

 

[cobrakiller58]

@Poppa the people here will legitimately try to help but they are not in front of the computers that are having the issue so they need you to help them help you. If you will just go through the paces with them the problem will be resolved a lot sooner, you may be correct and it is a bug in FreeNAS but without the necessary information they will never be able to reproduce it and fix it. We know you have FreeNAS installed on
Motherboard - Supermicro X8DTH-IF
CPU - Xeon E5520
RAM - 24GB
Mellanox ConnectX-2

and windows on
Intel i7-4790K
ASRock Z97 Extreme6
16GB RAM
Mellanox ConnectX-2
connected through some 40GBe switch (if you could provide a model number that may help) and trying to transfer data over a SMB share. If you can run the Iperf test between the windows machine and FreeNAS and post results that will give an idea if it is a network issue or SMB issue. If your not sure how to run Iperf just ask, people will help. I understand it is frustrating buying new gear and it does not work properly.

 

[Poppa]

Assuming "OMV" is OpenMediaVault, was the share there also SMB? Samba is frequently CPU-bound.

Yes, I was reffering to OpenMediaVault and yes, I was also using SMB in OMV.

 

[Poppa]

@Poppa the people here will legitimately try to help but they are not in front of the computers that are having the issue so they need you to help them help you. If you will just go through the paces with them the problem will be resolved a lot sooner, you may be correct and it is a bug in FreeNAS but without the necessary information they will never be able to reproduce it and fix it. We know you have FreeNAS installed on
Motherboard - Supermicro X8DTH-IF
CPU - Xeon E5520
RAM - 24GB
Mellanox ConnectX-2

and windows on
Intel i7-4790K
ASRock Z97 Extreme6
16GB RAM
Mellanox ConnectX-2
connected through some 40GBe switch (if you could provide a model number that may help) and trying to transfer data over a SMB share. If you can run the Iperf test between the windows machine and FreeNAS and post results that will give an idea if it is a network issue or SMB issue. If your not sure how to run Iperf just ask, people will help. I understand it is frustrating buying new gear and it does not work properly.

My switch is an Arista DCS-7050QX-32-R but right now my PC and FreeNAS machine are directly connected.

I am sorry if I have been a little rude to anyone.

I did run iperf again and surprisingly, I actually got a good but conflicting result this time. For the test, I should be seeing something like 1.25GB/s if my 10Gbe connection was fully saturated and yes, I know that I am not likely to ever saturate my connection unless I am copying to/from NVMe storage. The 470MB speed I am getting on this test is the speed I am expecting to actually get but I just tried coping more files and only got speeds of about 30MB/s.

Code:

bin/iperf.exe -c 192.168.1.50 -P 1 -i 1 -p 80 -f M -t 10 -T 1
------------------------------------------------------------
Client connecting to 192.168.1.50, TCP port 80
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.1.155 port 64943 connected with 192.168.1.50 port 80
[ ID] Interval	   Transfer	 Bandwidth
[156]  0.0- 1.0 sec   463 MBytes   463 MBytes/sec
[156]  1.0- 2.0 sec   455 MBytes   455 MBytes/sec
[156]  2.0- 3.0 sec   468 MBytes   468 MBytes/sec
[156]  3.0- 4.0 sec   471 MBytes   471 MBytes/sec
[156]  4.0- 5.0 sec   467 MBytes   467 MBytes/sec
[156]  5.0- 6.0 sec   470 MBytes   470 MBytes/sec
[156]  6.0- 7.0 sec   475 MBytes   475 MBytes/sec
[156]  7.0- 8.0 sec   473 MBytes   473 MBytes/sec
[156]  8.0- 9.0 sec   476 MBytes   476 MBytes/sec
[156]  9.0-10.0 sec   457 MBytes   457 MBytes/sec
[156]  0.0-10.0 sec  4674 MBytes   467 MBytes/sec
server sent unexpected data failed
Done.

 

[Poppa]

Anyone have any idea???