(yet another?) CIFS performance Thread

[SkogNisse]

Hello There!

First to say, thanks for all the great info and knowledge that is daily shared here in this forum. I have been able, from being a normal windows guy, been able to setup my home FreeNAS 9.10-stable server, with plex/plexpy , sickrage, btsync etc in separate jails, and all is running and tugging along just nicely.

However, there is one big drawback , i haven't been able to kink out. We're talking about CIFS write performance over Ethernet.

From what it seems, the smb/cifs service, seems somewhat bottlenecked compared to the performance of FreeNAS itself.

I have a fairly powerfull server, for a home user's point of view, Consisting of :
Supermicro CSE847A-R1400 36bay 4U Chassis , Dual 1400W Ratet PSUs,
Dual Xeon X5550 (2x 4core/8thread) , and 48GB PC3-10600 ECC-REG Dimms
5x Dell Perc H310 Cross Flashed to LSI-9208 IT mode F20 firmware (Ebay from pulled servers, not china copies)
18x WD Red 3TB NAS for STORAGE pool 8x Hitachi Ultrastar 10K SAS 450GB drives for INCOMING pool.
(Two Mellanox 10GB ethernet cards,. was used for testing, they are removed now since the performance of cifs made it useless anyway)

And yes, it runs VmWare ESXi 6.5 on the bare metal. i have other windows services i need to run also, on a separate vm.
I have locked 32GB ram to the FreeNAS guest, and running all 5 perc cards as true passtrough, vt-d enabled.
Haven't had any sort of compatibility or stability issue.

My previous setup was Windows 2008 server, with Areca hardware raid controller.And that has always been stable, and blazing fast. the Areca ARC-1882-IX24 4GB cache easily gave me 650-750 MB/sec over SMB shares. (on 10GB ethernet)


My problem is this :

CIFS seems to be running a bottleneck at around 80 MB/sec write. (testing from one pc, my workstation).


When i download from the NAS , i get 112 MB/sec stable, so download seems a lot faster. (1GB Ethernet)
Upload to NAS is mostly 80 MB/s. (1GB Ethernet)

If i try to browse smb folders, delete files, and do other operations simultaneously, it feels ultra slow, like the commands are "in que" waiting for free time in between the ongoing file transfer.
so i can see delete file operations at 1file/s. if i pause the transfer, it speeds up instantly.

Is this really supposed to behave this poorly ?


To rule out if it was just the hardware, software or anythning, i did a fresh install of FreeNAS 10 nightly , on my old fileserver yesterday. Bare metal, straight to the box.
(i5 3330s, 16GB ddr3, asus z77-i desktop motherboard) but performance was similar there, around 85-90 MB/s to it, and 110 MB/s from it. behaved exactly the same.



FTP is a lot faster, i have no other way of testing speed for now. iSCSI i have no knowledge of , but it seems a bit more complicated then needed to be.

The sad part is i have already spent countless hours to setup everything running smoothly, and i really love the Resilver only existing data, snapshots, and most of all rehashing of "bit rot" data . since i do store 30+ TiB of data, that is rarely used or refreshed, the bit rot capability alone was one of the major reasons i switched to zfs from the 1300$ areca controller alone.
But now the performance is really making me bummed out.

I have tried to search and do some tweaks , but not any that have had any great changes.
I have also tried to enable the "autotune" feature, but that over time just made it go even slower. Periods my writes was down to 60-65 MB/s, and that's just rubbish.
a 250$ Qnap nas box can do that.


Hoping to hear back from people with similar experience, and how they managed to solve it. or if it's just that this setup is not what suits my application.


Basicly i would like it to max out my current 1gb Ethernet trough smb shares, with no problems, as i would expect of such powerful hardware.
I am certain that this is somewhat fixable with proper tuning and setup, i just don't know where to start.

best regards,
Tommy

 

[HoneyBadger]

Hi Tommy,

The read speed capping out at 112MB/s makes me think that there's a networking issue (or network driver issue) somehow causing things to negotiate at 1Gbps instead of 10Gbps.

I'd suggest using iperf to test links between the FreeNAS machine and your workstation.

For local disk testing, create a dataset with compression disabled entirely, and then use the dd command to shuttle from /dev/zero to your dataset.

 

[SkogNisse]

I edited the text a little, i realize it was maybe a bit confusing. I just did the 10GB ethernet cards temporarely, for testing beyond the 1Gbg speed. with 10G cards i got around 250 MB/s read, and 80-130 MB/s write.

Now with the current 1GB network , i get 109 MB/s read, and 80-82 MB/s write.
Sustained, one 25GB large test file.

I will try to do some performance tests on the disk arrays, but resilver on the storage array last time went over 1100 MB/s .

 

[SkogNisse]

Included screenshots, of a transfer of 25gb file to and from the Storage Raid.

The Storage raid is 18x3TB WDred in RaidZ2... it's running Gelid encryption.

Performance to the 8x 10K sas drives are a little bit higher write, around 92 MB/s.

How can i improve it ? I want the Write speed to match the Read speed. the Read speed is as expected.

 

[SkogNisse]

Hi Tommy,

The read speed capping out at 112MB/s makes me think that there's a networking issue (or network driver issue) somehow causing things to negotiate at 1Gbps instead of 10Gbps.

I'd suggest using iperf to test links between the FreeNAS machine and your workstation.

For local disk testing, create a dataset with compression disabled entirely, and then use the dd command to shuttle from /dev/zero to your dataset.

So i did some more research, thanks for pointing me to iperf. there is a quite different performance gap between read to nas, and write to nas. seems i have to figure out how to tweak the tcp settings maybe then ?

Got 882 Mbits from NAS , default settings. Got 623 mbit to the NAS, default settings. (just iperf -s ,and iperf -c. ) i see the difference in tcp window size varies a lot.the NAS reports 32,5 KB. the windows pc 63 KB. i have to read up more on this matter.

 

[SweetAndLow]

What nic is in your Windows machine? Is it realtek? If so those write speeds are about as fast as that nic can go.

Sent from my Nexus 5X using Tapatalk

 

[SkogNisse]

What nic is in your Windows machine? Is it realtek? If so those write speeds are about as fast as that nic can go.

Sent from my Nexus 5X using Tapatalk

IT's an Intel Gigabit onboard directly on the PCI-Ex bus :)

 

[SkogNisse]

I managed to get 931 Mbit in a single thread write to the NAS , when doing -w 256k (window size 256k).

How can i apply this improvement to "real world cifs performance" ?

This rules out my windows NIC, and ethernet switch as bottleneck . (i do get ~full wirespeed windows to windows)

 

[SweetAndLow]

Good to see you have an Intel nic. These numbers don't really make sense. Can you try removing the switch from the equation? Direct connect the server to client.

Sent from my Nexus 5X using Tapatalk

 

[tvsjr]

Start at the NAS and work your way out.

Create a dataset with compression disabled. Write a large file to it and see what the speed looks like (dd if=/dev/zero of=/mnt/tank/testdataset bs=1M count=102400). If your bottleneck is on the NAS itself, you're chasing your tail looking at the network.
Assuming that's good, create another VM on the same ESXi host. Same network segment as the NAS. Test performance from there (iperf to test the network, a large copy to test CIFS).

Keep working away from the NAS step-by-step until you find the problem.

You've also added a significant amount of complexity by running it virtualized. I'm not in the "thou shalt not" camp... but it does add challenges, especially when you start demanding high performance. You might try pulling the VMware drives and booting the system directly to FreeNAS. That would tell you whether your issue is in the virtualization layer.

Bottom line, don't assume CIFS is your problem. It could be elsewhere, and the myopia induced by that assumption may make you overlook the real cause.

 

[SkogNisse]

I will take more tests soon when i have the time. I do agree that after the latest testing and such, i have come to realize it's not the cifs thats the main problem.

I did some testing last night, and realized i didnt get more then 90 MB/s on FTP either.

when i have a window of downtime, i can just as well as you say just install freenas on a temporarely usb stick , and boot from that directly. The hba's are passtrough anyways, so they should be detected the same way directly to a bare metal install.

will post updates when around

 

[logan893]

I've had some issues with TCP window size scaling in Windows 7. Something was causing Windows 7 to not negotiate any scaling factor, which caused TCP connections to use a non-scaling window size (maximum 64kB). Having the TCP window scaling disabled and limited to a maximum 64 kB (you can verify this via packet sniffing), can be a factor in limiting saturation of even a local low-latency 1 Gbps link.

maximum throughput with a TCP window of 64 KByte and RTT of 0.5 ms <= 1048.58 Mbit/sec.
https://www.switch.ch/network/tools...6&bw=1000&rtt2=0.5&win=64&Calculate=Calculate

I'm sure you'll see fluctuations at or above 0.5 ms RTT when pushing the limits of the link, and this may limit your speed.

I believe I "fixed" it by turning off "Window Scaling Heuristics" completely.

Run this from a command prompt (as administrator):

Code:

netsh interface tcp set heuristics wsh=disabled

You could also consider setting up a dedicated (sub)net between your workstation and the server, where you run 9k jumbo frames.

 

[SkogNisse]

I've had some issues with TCP window size scaling in Windows 7. Something was causing Windows 7 to not negotiate any scaling factor, which caused TCP connections to use a non-scaling window size (maximum 64kB). Having the TCP window scaling disabled and limited to a maximum 64 kB (you can verify this via packet sniffing), can be a factor in limiting saturation of even a local low-latency 1 Gbps link.

maximum throughput with a TCP window of 64 KByte and RTT of 0.5 ms <= 1048.58 Mbit/sec.
https://www.switch.ch/network/tools...6&bw=1000&rtt2=0.5&win=64&Calculate=Calculate

I'm sure you'll see fluctuations at or above 0.5 ms RTT when pushing the limits of the link, and this may limit your speed.

I believe I "fixed" it by turning off "Window Scaling Heuristics" completely.

Run this from a command prompt (as administrator):

Code:

netsh interface tcp set heuristics wsh=disabled

You could also consider setting up a dedicated (sub)net between your workstation and the server, where you run 9k jumbo frames.

My heurestics show this info now :

netsh interface tcp>show heuristics
TCP Window Scaling heuristics Parameters
----------------------------------------------
Window Scaling heuristics : disabled
Qualifying Destination Threshold : 3
Profile type unknown : normal
Profile type public : normal
Profile type private : normal
Profile type domain : normal

My TCP global parameters show this :

netsh interface tcp>show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State : enabled
Chimney Offload State : disabled
NetDMA State : disabled
Direct Cache Access (DCA) : disabled
Receive Window Auto-Tuning Level : normal
Add-On Congestion Control Provider : none
ECN Capability : disabled
RFC 1323 Timestamps : disabled
Initial RTO : 3000
Receive Segment Coalescing State : disabled
Non Sack Rtt Resiliency : disabled
Max SYN Retransmissions : 2
TCP Fast Open : enabled

Anything ring a bell ?


Didnt change much, still the same 86-87 MB/s write to FreeNAS.

 

[logan893]

Heuristics off, looks good.

There are other tunables for TCP in FreeBSD, and for Samba, which may help. But first it's best to isolate each part, as previously suggested, and make sure there are no other obvious bottlenecks.

Some things have been suggested already.

Validate pool write speed with dd from /dev/zero (compression off). Use a RAM disk as source when sending data. Check CPU load and busy% on disks in FreeNAS during file transfers.