SOLVED Successful new build - but I have a 50% speed loss over wifi - Only to freenas....

[Minty Trebor]

I have successfully built a very fast (well I think its fast!) 9.3 Freenas box, and I am getting about 100 mb/sec (according to windows 10) sustained write speed to a 5 disk zfs pool, over CIFS. As you can expect I am very happy with this. I also have a working vbox jail, Logitech Media Server Jail, and Plex plugin/jail.

But I have a very strange wifi transfer speed issue - Only to the Freenas server.
(I understand that windows does not give accurate write speeds, but as a comparator in this instance I think they are fine)

Using the same Windows 10 laptop as I used for the wired speed tests and the same test file; However I can only reach 20 mb/sec sustained write over wifi.

My wifi is using the AC protocol (5Ghz) and if I upload the same file to different network share on a different machine (using basic windows network sharing) I see a sustained transfer speed of around 40 mb/sec over wifi.

The network has been in place for a number of years, and the wifi has consistently delivered around a 40 mb/sec transfer speed previously. I was running a Linux Mint build (on the same H/W) with Native ZFS prior to converting it over to Freenas. With the linux mint zfs pool, I used to get around 70 mb/sec wired, and 40 mb/sec wireless.

The wireless data is not taking a different route to the server compared to the wired data (all going through the same router and cables, except for the first hop over wifi of course). I run 2 Nics, and I have tried direct writing to both IP's, instead of server name, but this made no difference.

I have read the documentation around CIFS tuning and other posts spotted around the forum about tuning NICS, but I fail to understand why this speed drop is only to the Freenas box....

I'm probably missing something obvious, but any help or ideas would be appreciated....

Specs below...

Server:
Core i5
12GB Ram
Asus P7H57D-V EVO
2 Nics (1 onboard, 1 PCI-E)
5 disk ZFS volume/pool (2tb WD reds)
2 disk stripe volume/pool (60GB SSD's) - running jails only

 

[jgreco]

That's completely normal for a board with a Realtek LAN. The little hamsters on the wheels that push bits around inside a Realtek chipset sometimes get very tired, so you may see various odd things related to them and performance. Perhaps one of your hamsters is sick or deceased.

The simple fact is that Realtek chipsets are slanted towards extremely cheap desktop connectivity for Windows PC's. As such, their ability to efficiently line up packet after packet and pump them out at full speed for billions of packets without issue is very questionable. We recommend the Intel parts.

 

[FlynnVT]

If it is only a problem on wireless, try increasing the TCP window size to 256Kb. I think the default is 64Kb (not autoscaling?) and found that it limited performance on my setup - apparently due to the effective latency of the wireless chain.

Temporarily:

sysctl net.inet.tcp.recvspace=262144
sysctl net.inet.tcp.sendspace=262144

...or you can use the -w option on iperf for experimentation.
​

To make it permanent you can add the variables and values via: http://olddoc.freenas.org/index.php/Sysctls

More information, somewhat related:

http://www.anandtech.com/show/7085/the-2013-macbook-air-review-13inch/10
http://arstechnica.com/apple/2013/0...-the-2013-macbook-airs-802-11ac-wi-fi-speeds/
I've never looked into autoscaling and don't know what the default is on FreeNAS: https://en.wikipedia.org/wiki/TCP_window_scale_option#FreeBSD.2C_OpenBSD.2C_NetBSD_and_Mac_OS_X
​

Also, make sure CIFS is configured to allow SMB3. (http://doc.freenas.org/9.3/freenas_services.html#cifs, Table 11.3b). AFAIK there are chattiness/efficiency improvements to the later protocol versions that handle this latency better too.

I see 20Mb/sec a room away from the AP when the indicated MAC rate is about ~350mbps. This rises to over 40MB/sec with the laptop closer and an indicated 866mbps MAC rate.

 

[Minty Trebor]

Thank you both for your responses.
@FlynnVT - Awesome - that worked. I set both of those sysctl values, and set the minimum SMB to SMB3, and I am now getting a sustained rate of around 35 mb/sec... Much closer to my previous speeds and definitely within tolerance... I'm sure I can find some more tweaks to tease out a few more mb/sec, and I will look into your links around auto-scaling.... Thank you so much. (as an added bonus it upped my wired transfer speeds too!!)

@jgreco - When Freenas releases V10, and I can get access to native bare metal virtualisation my plan is to migrate Freenas over to a 32gb/Core17 (extreme 6core) which has a native on=board intel nic. I will then ditch the realtek pci-e as you advise and switch over to intel exclusively. Thanks for the tip.

 

[jgreco]

Glad to see there's a mitigation of some sort, but obviously the 35 is less than the 40. Usually people are trying this on wired ethernet and with Realtek, they just never get anywhere near the native performance of a $35 desktop Intel LAN chipset.

 

[Ericloewe]

The little hamsters on the wheels that push bits around inside a Realtek chipset sometimes get very tired

It doesn't help that Realteks rely disproportionately on the little hamsters in the CPU. :D

 

[jgreco]

It doesn't help that Realteks rely disproportionately on the little hamsters in the CPU. :D

Indeed. And just like hamsters, one crummy set of silicon seems to have evolved into an entire family of crummy silicon ...

 

[jgreco]

Indeed. And just like hamsters, one crummy set of silicon seems to have evolved into an entire family of crummy silicon ...

By the way, that's an obscure reference to the fact that all modern Golden hamsters are widely considered to have descended from a single brother/sister pair.

http://www.smithsonianmag.com/ist/?...of-the-hamster-aka-mr-saddlebags-1223774/?all

 

[anodos]

By the way, that's an obscure reference to the fact that all modern Golden hamsters are widely considered to have descended from a single brother/sister pair.

http://www.smithsonianmag.com/ist/?...of-the-hamster-aka-mr-saddlebags-1223774/?all

Wow. Brother and sister. Sounds like they're from Arkansas. Those things must've bred like... umm... hamsters?

 

[jgreco]

Why you harshin' on Arkansas?