Poor CIFS Write Performance...

[JimPhreak]

System:

i3-4130
SuperMicro X10SLM-F-O
16GB Kingston ECC RAM
6x3TB WD Reds
M1015 HBA
FREENAS 9.1.1

So I've been having some issues with my CIFS write performance. I'm getting around 100MB/s read performance on average but my writes wind up around 50-55MB/s. I've got autotune enabled but I'm not sure how/if I need to fine tune my settings any more to improve performance. I'm planning to setup Link Aggregation shortly and want to be sure that I'm doing everything I can to saturate that 2Gb connection.

I've run the following test on my volume:

dd if=/dev/zero of=testfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.512641 secs (199749848 bytes/sec)
190MB/s write

dd if=testfile of=/dev/zero bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.202251 secs (506301682 bytes/sec)
482MB/s read

Are there any other tests I should be doing to determine if it's my volume or something with my shares maybe?

 

[JimPhreak]

Any clues why I'd only be getting 50-60MB/s write speeds over CIFS when my internal writes are closer to double that which would saturate my gig connection? Any other tests I can run to find my bottleneck?

 

[russnas]

That's normal for cifs, use ftp if you want expected speeds, I get 50-60mbs while ftp max the hdd at 100-112mbs, Im not sure why cifs is slower,

 

[JimPhreak]

That's normal for cifs, use ftp if you want expected speeds, I get 50-60mbs while ftp max the hdd at 100-112mbs, Im not sure why cifs is slower,

What are your CIFS read speeds? I mean if I'm getting over 100MB/s reads doesn't that mean that CIFS is fast enough to saturate (of come close to saturating) my gigabit connection? The read/write difference appears to be a product of the pool and not the share. At least that's how it appears on the surface to me.

 

[russnas]

well my read speeds are 66MBs and write is 77MB on a single drive,while i was looking for reasons why its slower, i see people on here able to saturate gigabit. cpu load or hardware isnt the problem,

cyberjock will be able to solve your problem

 

[JimPhreak]

I have a spare 128GB SSD laying around. Is it worth setting this up as a cache drive or am I in the 99% (as Cyberjock pointed out in his powerpoint) that will not benefit from it? If I can't get more than 50MB/s writes I'm going to have to ditch my FreeNAS setup and go back to RAID10.

 

[survive]

Hi JimPhreak,

This is what I have in the "Auxiliary Parameters" box under the CIFS servie config:

Code:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 
min receivefile size = 131072
write cache size = 131072

and this is what I just got when I copied an 8.5GB iso over to a CIFS share:

http://imgur.com/ZZ5hb5p

I checked & don't see any other sysctls or tunables I have set or any other evidence of other tuning I have done.

Maybe give those a shot & see what you get.

-Will

 

[JimPhreak]

Hi JimPhreak,

This is what I have in the "Auxiliary Parameters" box under the CIFS servie config:

Code:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
min receivefile size = 131072
write cache size = 131072

and this is what I just got when I copied an 8.5GB iso over to a CIFS share:

http://imgur.com/ZZ5hb5p

I checked & don't see any other sysctls or tunables I have set or any other evidence of other tuning I have done.

Maybe give those a shot & see what you get.

-Will

Thank you Survive, that did boost my speeds up to a little bit to 60-65MB/s write speeds across CIFS shares. What I notice is that if I write a file from one of my PC's C drives (SSD's) to a CIFS share on my FreeNAS box I get great performance (90-100MB/s writes). But if I'm moving a file between CIFS shares on my FreeNAS box I get much lower performance (60-65MB/s). Is this expected? I would have thought moving a file one share to the other internally wouldn't cause such a performance drop.

 

[survive]

Hi JimPhreak,

You mean you are moving them from one CIFS share to another through a windows box (maybe through explorer)? If so I would expect that to take (much) longer.

Try moving a big file internally on the filer & see how fast that goes (get the windows box out of the loop, if that's what you are doing). The downside to that, of course, is that might have an unpredictable effect on the permissions the file ends up with.

-Will

 

[JimPhreak]

Hi JimPhreak,

You mean you are moving them from one CIFS share to another through a windows box (maybe through explorer)? If so I would expect that to take (much) longer.

Try moving a big file internally on the filer & see how fast that goes (get the windows box out of the loop, if that's what you are doing). The downside to that, of course, is that might have an unpredictable effect on the permissions the file ends up with.

-Will

I see. Well the problem is that even if it speeds up the transfer doing it internally like that, it won't be applicable to how I implement it on my network. I have different shares (ie. Downloads and Videos) and every day I have to move files between those shares on Windows OS's. So I was looking for a way to speed up those transfers.

 

[cyberjock]

Then do 1 share for all of them. Then moving 1TB of files should go lightening fast because it's local file moves only. It's not like it really matters if you have 1 share or 10.

 

[JimPhreak]

Then do 1 share for all of them. Then moving 1TB of files should go lightening fast because it's local file moves only. It's not like it really matters if you have 1 share or 10.

Good point! That does seem to work a little better although now when I move files to different folders (datasets) within the same share it no longer shows the speed/time remaining. Now it just sits there saying "calculating time remaining" for a while and then it goes away when the move is complete. It does seem to be a bit faster but unfortunately I can't see the actual transfer speed.

 

[cyberjock]

If they are different datasets they are effectively different file systems, so instead of moving/copying over the network the server will have to do a move/copy locally. One of the reasons I try to avoid datasets. :P

 

[JimPhreak]

If they are different datasets they are effectively different file systems, so instead of moving/copying over the network the server will have to do a move/copy locally. One of the reasons I try to avoid datasets. :p

I see. In that case what are some specific scenarios where datasets are beneficial?

 

[gpsguy]

From 6.36 of the manual "Permissions, compression, deduplication, and quotas can be set on a per dataset basis"

 

[JimPhreak]

From 6.36 of the manual "Permissions, compression, deduplication, and quotas can be set on a per dataset basis"

Thank you for that reminder!

 

[JimPhreak]

Well since creating just one share to the root volume and trying to move files between folders (different datasets) within that one share I've noticed that the performance is even worse. Furthermore I can't even tell exactly what speed I'm getting because it doesn't display the speed it just tells me how much time is left for the transfer (which is not accurage because it jumps around).

Errrrrrr.