SOLVED CIFS over WiFi is very slow only to FreeNAS - Sanity Check

[msignor]

Hello,

Started a conversation over on Reddit, figured I would post over here too. I am fully aware of the limitations on Wifi, throughput, expectations, etc. Hear me out...

FreeNAS: Freshly upgraded 9.10 installation.

• Supermicro X10SLM w/ I3 4130T
• 8 Disk WD Scorpio 500GB Black in RAIDZ2
• Kingston 32GB ECC RAM
• 5 Intel NIC's (2 onboard, 1 Dual PCIe, 1 Single PCIe)
  • em0 - unused
  • em1 - LAN Access (SSH, GUI, CIFS, etc.)
  • em2 - LAN Access 2 (lagg0, since removed on 6/1/16)
  • igb0 - iSCSI vlanA
  • igb1 - iSCSI vlanB

I created a 10GB file filled with random content. Copied to 3 separate machines, rebooting in between. Source is a laptop and it has 8GB RAM connected with 802.11AC. Wired speeds are normal and included.

Timed results in copying a 10G file

• LT -> FreeNAS (Wired Gbit): 1m:47s
• LT -> FreeNAS: 10m:14s
• LT -> W10vm: 3m:42s
• LT -> Qnap TS-231 (Linux): 3m:45s

SMB.conf: http://pastebin.com/aY9akLCT
Tunables: http://i.imgur.com/qwE9aqU.png (Old - All removed on 6/1/16)
Qnap SMB.conf: http://pastebin.com/8Um1Wz2i

FreeNAS SMB Logs of Wired and Wifi: http://pastebin.com/Q89QeXGA

Screenshots:

• Qnap Copy over WiFi: http://i.imgur.com/pQ1Ls6N.png
• FreeNAS Copy over WiFi: http://i.imgur.com/TXXBvwx.png

What started all of this is that I just got a new Ubiquiti AC-Pro and expected it to be "faster" than my Cisco N AP. Doing iperf to a test box single threaded I get ~280mbit. Move it to 4 threads, and I max out the AC on my laptop (approx 450mbit, expected at ~800mbit link speed). So, while initially I suspected this to be a throughput issue at the Access Point, it does not appear to be the case. That is why I tested against a Windows 10 box as well as my Qnap and was shocked to see it go faster.

What I have tried so far...

• Set Minimum SMB to 2.0
• Set Maximum SMB to 3.0_11

I also set sysctl:

• net.inet.tcp.recvspace to 262144
• net.inet.tcp.sendspace to 262144

I had tried to force a minimum of SMB3 so that I could rule out that being the issue and when doing that, the shares are completely un-accessible. When looking at the log.smbd it clearly says no protocol available.. but when listing the "attempts" it does not even try SMB3...

When I set the Minimum back to SMB2, and the max to SMB3, then I can view the share. When looking in the logs, it looks like it negotiated at SMB3_11 (expected).

Thoughts on where to look next? I want to say it's networking too, but the fact I can copy to the W10 as well as the Qnap which is running Samba... at what I would expect an AC link to deliver, somewhat says otherwise, no?

Thanks!

 

[m0nkey_]

I suspect it has something to do with the WiFi as you report normal speeds when connected via cable.

Does SMB2 still support multi-channel or multi threading?

Samba doesn't use threads. It spawns a process for each connection made.

As per the documentation:

Samba’s write cache parameter has been reported to improve write performance in some configurations and can be added to the “Auxiliary parameters” field. Use an integer value which is a multiple of _SC_PAGESIZE (typically 4096) to avoid memory fragmentation. This will increase Samba’s memory requirements and should not be used on systems with limited RAM.

If you wish to increase network performance, read the Samba section on socket options. It indicates which options are available and recommends that you experiment to see which are supported by your clients and improve your network’s performance.

 

[msignor]

I suspect it has something to do with the WiFi as you report normal speeds when connected via cable.

Could you suggest how else to confirm this? I would agree and was convinced of the same, however copying with WiFi to Windows 10 and another Samba device yield drastically different results. My only hunch up until now is that it has to do with either SMB3 not properly negotiating or something with the fact SMB is bound to lagg0

 

[anodos]

Could you suggest how else to confirm this? I would agree and was convinced of the same, however copying with WiFi to Windows 10 and another Samba device yield drastically different results. My only hunch up until now is that it has to do with either SMB3 not properly negotiating or something with the fact SMB is bound to lagg0

In samba SMB3 = SMB3_11. Setting the minimum protocol to SMB3 effectively locks out all clients other than Windows 10. I recommend setting server min protocol no higher than SMB2. Clients automatically negotiate the highest available protocol. I recommend making sure you refresh your tree connection after changing min / max protocol. The easiest way to achieve this is to log out of your client and log back in.

Clear out all those tunables. You shouldn't use autotune unless you have a pretty large system. While you're at it post your full system specs.

One thing you might try after setting "server min protocol" to a reasonable level and clearing out the tunables is setting the following auxiliary parameters under "services" -> "CIFS"

Code:

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

 

[msignor]

In samba SMB3 = SMB3_11. Setting the minimum protocol to SMB3 effectively locks out all clients other than Windows 10. I recommend setting server min protocol no higher than SMB2. Clients automatically negotiate the highest available protocol. I recommend making sure you refresh your tree connection after changing min / max protocol. The easiest way to achieve this is to log out of your client and log back in.

I have been testing on a W10 box and that is why I expected SMB3 as a minimum to work. It did not, which I thought was suspicious. Is it?

I'll give what you suggested a try. Should I also clear out the TCP window limits?

 

[anodos]

I have been testing on a W10 box and that is why I expected SMB3 as a minimum to work. It did not, which I thought was suspicious. Is it?

Not really. It may be that the client initiates the connection using a different SMB3 dialect then upgrades to SMB3_11. If you want to force SMB3 as the minimum protocol, you should probably choose SMB3_00 (but SMB2 is an even better choice).

Should I also clear out the TCP window limits?

Yeah, I'd do that too. You have too much going on at once, which makes it harder to troubleshoot.

 

[msignor]

OK. A few things.

1) I verified that it does connect via SMB2, then negotiates to SMB3_11
2) I originally had CIFS attached to a LAGG, with no other traffic. I have now removed that and CIFS is on a dedicated NIC.
3) Removed all Tuneables, disabled AutoTune
4) Rebooted FreeNAS for giggles
5) Tried file transfer to Qnap and FreeNAS with same results.


EDIT: I put the contents of the logs when copying over wifi as well as ethernet here: http://pastebin.com/Q89QeXGA

 

[anodos]

1. OK. A few things.

1) I verified that it does connect via SMB2, then negotiates to SMB3_11
2) I originally had CIFS attached to a LAGG, with no other traffic. I have now removed that and CIFS is on a dedicated NIC.
3) Removed all Tuneables, disabled AutoTune
4) Rebooted FreeNAS for giggles
5) Tried file transfer to Qnap and FreeNAS with same results.


EDIT: I put the contents of the logs when copying over wifi as well as ethernet here: http://pastebin.com/Q89QeXGA

Post the full hardware specs for your freenas server. While you're at it, post the SMB.conf from your embedded NAS appliance.

 

[msignor]

Post the full hardware specs for your freenas server. While you're at it, post the SMB.conf from your embedded NAS appliance.

Here is the SMB.conf from the QNAP. I updated the system specs at the top of the thread.

http://pastebin.com/8Um1Wz2i

 

[cyberjock]

Honestly, I just read through this thread and all I see is "yep.. wifi is still slow.. what's new?"

I don't really see any compelling data that anything is wrong. That is besides the fact that you may feel that Wifi should be faster. But that's also why somewhere we tell people not to complain about poor wifi performance because there's no fixing that fro this forum. :P

You certainly won't be the first to be unhappy with the performance of Wifi, and I have no doubt you won't be the last. Not even the last for this week.

 

[msignor]

I don't really see any compelling data that anything is wrong. That is besides the fact that you may feel that Wifi should be faster. But that's also why somewhere we tell people not to complain about poor wifi performance because there's no fixing that fro this forum. :p

Thanks for the reply. I anticipated this response at some point so I do appreciate your candor.

However, not trying to start a debate - I am not sure how much I can "feel" when I see performance working at what all of the networking guys say it should be working at in other scenarios. I was content on saying "it is what it is" because I agree that Wifi is well.. Wifi. BUT, I can certainly disprove that WIFI in general is a bottle neck here. (at least in some situations). I also would be willing to accept that Samba isn't perfect but then I debunked that through testing with my $120 off the shelf appliance running Linux so I know it's possible with Samba.

I am not saying they are the same as FreeNAS because I know they aren't but from a "it's wifi" perspective everything is the same. Could you offer any other ideas as to why I would see this kind of drastic difference between destination platforms while keeping the source and transport controlled?

My only running theory.. is that FreeBSD and Linux implement networking differently (obviously) and because of that I am getting the same performance that I see with my TCP iperf testing with a single thread / stream. At 280mbit which is what I tested my bandwidth to be on a single iperf stream, I would assume a 50% loss due to overhead etc... puts me around 140megabits which is about the speed I am seeing when doing a copy to FreeNAS. This is only me trying to come up with some kind of reasonable explanation..

Maybe Qnap and Windows 10 understand multiple TCP streams, I don't know. I don't want to start sounding like I know all of those details - I have been feverishly reading all of the forums I can find picking up the lingo and test cases.

This morning I spun up a FeeeNAS VM on my ESX box. Made a local disk on the local SSD to see what happens.... The w10 machine I was testing with this entire time is on the same host on the same SSD, so that is consistent. When trying to copy the same 10G file to the FreeNAS VM it goes at exactly the same speeds both wired and wifi as my primary FreeNAS. Since I can copy to that same W10 VM as before within minutes and everything is OK, I don't know how else I can mitigate slight variables without a dedicated control. I suppose I could make a VM on my laptop for consistency, but I think I have more than proven with data and tests, that this is something reproducible and lies with FreeNAS somehow. I am banking on it just being configuration.. but I don't know what to look at or even google for at this point.

 

[jgreco]

net.inet.tcp.recvspace to 262144
net.inet.tcp.sendspace to 262144

Try instead:

net.inet.tcp.recvspace=2097152
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvbuf_inc=262144
net.inet.tcp.recvbuf_auto=1

and those same variables also adjusted for sendspace/sendbuf as well.

There's a whole class of issues regarding increased latency due to wifi that impact wifi performance. In some cases, other devices have been tuned or hacked around this in various ways. Increasing buffer sizes is one of those ways.

 

[msignor]

Try instead:

net.inet.tcp.recvspace=2097152

Thanks for the reply - I just got a chance to try this now, and as soon as I added the sysctl, the web gui stopped working and now I am getting "no buffer space available" at the console.

EDIT - I tested this on my VM. Every time I set the recvspace to 2097152 it blows up. Nothing works on a reboot either. I did set the MAX for them both prior to setting recvspace the second time around.

Thoughts?

 

[jgreco]

Huh. How much memory do you have?

 

[msignor]

Huh. How much memory do you have?

32GB on my production machine, and only 1gb on my VM that I was testing on. Both systems experienced the same thing. (can you say, awesome recovery options with FreeNAS?)

Sorry, perhaps I should also clarify. I tested first on the physical system itself. Nginx stopped working and then I went to the console to restart it manually and saw that it couldn't start. I opted to reboot (bad) and then I was stuck in this boot infinity where it would never fully boot to shell. I rolled back to my prior upgrade point.

I then tried on my VM, in a different order - thinking I had to set the limits before I set the option. No avail, same issue.

 

[msignor]

I tested with FTP this morning.. basic, no encryption or resume. 16MB/sec to FreeNAS - 35MB to QNAP on the same 10G file. Don't think it's SMB anymore... Would it be appropriate at this time to close this thread and open a new one under general performance issues?

 

[jgreco]

32GB on my production machine, and only 1gb on my VM that I was testing on. Both systems experienced the same thing. (can you say, awesome recovery options with FreeNAS?)

Sorry, perhaps I should also clarify. I tested first on the physical system itself. Nginx stopped working and then I went to the console to restart it manually and saw that it couldn't start. I opted to reboot (bad) and then I was stuck in this boot infinity where it would never fully boot to shell. I rolled back to my prior upgrade point.

I then tried on my VM, in a different order - thinking I had to set the limits before I set the option. No avail, same issue.

Yeah, my bad, I probably meant 131072 for the recvspace/sendspace values but sometimes I'm coffee-deprived.

Your 1GB VM isn't expected to work at all.

Basically the goal here is to increase the amount of buffering available to the networking subsystem, which requires memory. On a properly functioning ethernet with reasonable gear and good network interfaces, there isn't as much need for significant buffering to play a role in performance. Wifi can never really be a properly functioning low latency ethernet, so this translates to reduced performance.

See for example https://filipv.net/2013/06/19/effects-of-latency-and-packet-loss-on-tcp-throughput/ as a bit of a primer.

So the goal here is to buffer more stuff, ideally on each end, to give the TCP stacks more space and data to work with. This requires somewhat more memory.

I tested with FTP this morning.. basic, no encryption or resume. 16MB/sec to FreeNAS - 35MB to QNAP on the same 10G file. Don't think it's SMB anymore... Would it be appropriate at this time to close this thread and open a new one under general performance issues?

So it's slow with both FreeNAS and QNAP. Not a shock. This basically implies that QNAP's using larger settings by default, because 35 megabytes per second on a gigabit ethernet is total crap.

 

[msignor]

So it's slow with both FreeNAS and QNAP. Not a shock. This basically implies that QNAP's using larger settings by default, because 35 megabytes per second on a gigabit ethernet is total crap.

Thanks man. I'll give that a shot. I was reluctant to mess with it by myself before.

Just to clarify, the FTP test was wireless. But I agree, FTP alone was slower than CIFS :)

I also was reading up on TCP/Latency impact on throughput but didn't see the link you posted. Will try and educate myself some more as well.

Quick Edit: I took a look at the QNAP for giggles... I believe this is what they are using:

net.core.wmem_max = 8388608
net.core.rmem_max = 8388608
net.core.wmem_default = 163840
net.core.rmem_default = 163840

 

[cyberjock]

For sh*ts and giggles, try getting rid of anything you've set manually and try these sysctls:

net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.mssdflt=1460
net.inet.tcp.recvspace=613200
net.inet.tcp.sendspace=613200

And add this loader value:

cc_cubic_load="YES"


Let me know if they help or not.

The algorithm change may make a big difference.

 

[msignor]

For sh*ts and giggles, try getting rid of anything you've set manually and try these sysctls:

net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.mssdflt=1460
net.inet.tcp.recvspace=613200
net.inet.tcp.sendspace=613200

And add this loader value:

cc_cubic_load="YES"


Let me know if they help or not.

The algorithm change may make a big difference.

I removed all of the autotune before, so I just added what you noted. The system somewhat hung when adding the loader value. I tried a transfer again, getting better - up to 18-19MB/sec. Should I try the other values you noted before? I should note, I didn't reboot when testing.