CIFS over LAN only 38 MB/s

[michaeljk]

Hello,

I have a speed problem with my new FreeNAS box. Currently I'm writing some data on it and the speed is stuck at 38 MB/s (arround 300 MBit/s). Data of the server:

Hardware:
HP Microserver Gen8, Intel Celeron G1610T CPU @ 2.30GHz, 16 GB ECC RAM, 4x4 TB WD Red (WD40EFRX) HDDs.

ZFS:
RaidZ2, Compression on (LZ4), no deduplication, atime enabled.

Sharing:
CIFS, Share Type Windows on Dataset

Stats:

diskinfo -tv ada0
ada0
512 # sectorsize
4000787030016 # mediasize in bytes (3.6T)
7814037168 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
7752021 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
WD-WCC4E5XXXXXX # Disk ident.

Seek times:
Full stroke: 250 iter in 6.595354 sec = 26.381 msec
Half stroke: 250 iter in 5.006203 sec = 20.025 msec
Quarter stroke: 500 iter in 7.808440 sec = 15.617 msec
Short forward: 400 iter in 3.389964 sec = 8.475 msec
Short backward: 400 iter in 2.771623 sec = 6.929 msec
Seq outer: 2048 iter in 0.138208 sec = 0.067 msec
Seq inner: 2048 iter in 0.139116 sec = 0.068 msec
Transfer rates:
outside: 102400 kbytes in 0.669548 sec = 152939 kbytes/sec
middle: 102400 kbytes in 0.853257 sec = 120011 kbytes/sec
inside: 102400 kbytes in 1.396211 sec = 73341 kbytes/sec

/mnt/tank/testing# dd if=/dev/zero of=temp.dat bs=1024k count=25k
25600+0 records in
25600+0 records out
26843545600 bytes transferred in 115.793239 secs (231823082 bytes/sec)
221 MB/s

iperf -c 192.168.0.4 -p 5001
------------------------------------------------------------
Client connecting to 192.168.0.4, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.51 port 53146 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.09 GBytes 934 Mbits/sec

Network and RaidZ2 speed seems ok. On a Linux Mint client I get approx. 38 MB/s write speed over CIFS, with a Macbook Air 58-59 MB/s (over Thunderbird to Ethernet-Adapter). The CIFS settings are on default, except that I disabled DOS attributes like described here:

https://forums.freenas.org/index.php?threads/cifs-directory-browsing-slow-try-this.27751/

Are there some other settings to get faster CIFS transfer speeds? The CPU is at ~35% when I transfer data, so I suppose that this is not the bottleneck?

 

[joeschmuck]

What type/size of files are you transferring? If you transfer a single large file (500MB or more), what is the speed of the transfer? Lastly, what is the connectivity between your FreeNAS system and the other computer (meaning WiFi, 1Gb CAT 5, Network Switches, etc...)?

Typically a slow network speed is related to either someone transferring a lot of small files or a connectivity issue like a 100Mbit switch being used vice a 1Gbit switch. Also if none of the above applies, hook up the FreeNAS directly to your computer, a direct CAT 5 cable and see what happens for a large file transfer. If it sucks then try a different cable.

 

[michaeljk]

I have a copy speed of 38 MB/s when I transfer large files (ISO, MKV, ...) to the server. With many smaller files it's even slower, but this is expected. Did you see my iperf stats which I posted before? It resulted in 934 MBit/s, so I don't think that the network is the problem. I'm using a HP 1810-24G Gigabit-Switch where both the clients and the FreeNAS box is connected.

 

[gpsguy]

Your MacBook Air has a SSD. Do you have a SSD on your Linux client.


Sent from my iPhone using Tapatalk

 

[michaeljk]

The Linux Mint client uses a normal SATA HDD, which should be capable of doing more than 38 MB/s read speed I'm expecting up to 90 MB/s with this setup but cannot find the bottleneck. RaidZ2 and network speeds are ok, transfers to CIFS shares are slow on different clients. So I guess that there must be a problem with Samba. Can you tell me which transfer speeds are "normal" with CIFS if you use the standard settings with FreeNAS 9.10 ? Should "Enable autotune" be activated in the advanced settings (it's currently not enabled)?

 

[UdoB]

Edit: content removed - not plausible

 

[michaeljk]

This was also my first guess. I'm currently transfering some files, smbd uses only 17-19% CPU. The "STATE" field in top switches between "select", "RUN", "CPU1". I also know that the integrated B120i controller is not the fastest, but the write test on the box shows that 221 MB/s should be enough to saturate gigabit link speeds.

 

[gpsguy]

For CIFS, your CPU is probably the limiting factor.


Sent from my iPhone using Tapatalk

 

[michaeljk]

Is there some CPU usage which is not displayed with "top" ? The above screenshot shows that the CPU is 69.2% idle while copying the data. It's no problem to change the CPU on the board, but before I want to be sure that this is really the reason for the slow speed. Does it matter if I use a DualCore or QuadCore CPU when using CIFS?

 

[joeschmuck]

Lets try a test here because I figure that your pool is likely the culprit here...
1) Create a new data store (called "no_comp") and turn off compression (you can't do the dd test with compression on or you get a false reading).
2) In the shell change directory into "no_comp" by typing "cd/mnt/tank/no_comp" (where tank is the name of your pool).
3) Lets run two tests as follows:

Code:

dd if=/dev/zero of=temp.dat bs=1024k count=25k

4) Reboot FreeNAS then do the next line (your file can be cached so rebooting purges this issue):

Code:

dd if=temp.dat of=/dev/zero bs=1024k count=25k

My results are (I used my 'backups' directory because it's already set for no compression):

Code:

[root@freenas] /mnt/farm/backups# dd if=/dev/zero of=temp.dat bs=1024k count=25k
25600+0 records in
25600+0 records out
26843545600 bytes transferred in 57.972478 secs (463039471 bytes/sec)

[root@freenas] /mnt/farm/backups# dd if=temp.dat of=/dev/zero bs=1024 count=25k
25600+0 records in
25600+0 records out
26214400 bytes transferred in 0.190318 secs (137740088 bytes/sec)

NOTE: IF I RUN THE SECOND COMMAND AGAIN, IT'S MUCH FASTER BECAUSE IT'S CACHED
[root@freenas] /mnt/farm/backups# dd if=temp.dat of=/dev/zero bs=1024 count=25k
25600+0 records in
25600+0 records out
26214400 bytes transferred in 0.052949 secs (495088177 bytes/sec)

EDIT: Forgot the cleanup step...

5) Type "rm temp.dat" to remove the file you just created with dd.

 

[joeschmuck]

Also, check your FreeNAS GUI --> Reporting --> Disk. Mine top out at 110MB/s per drive writing. What do yours look like during these tests?

 

[michaeljk]

The results:

Code:

dd if=/dev/zero of=temp.dat bs=1024k count=25k
25600+0 records in
25600+0 records out
26843545600 bytes transferred in 97.215880 secs (276123053 bytes/sec)

dd if=temp.dat of=/dev/zero bs=1024k count=25k
25600+0 records in
25600+0 records out
26843545600 bytes transferred in 109.524202 secs (245092364 bytes/sec)

Drive Reporting (Read):
http://fs5.directupload.net/images/160605/5ogkdtrp.png
http://fs5.directupload.net/images/160605/6u24slzh.png

 

[michaeljk]

Drive Reporting (Write):
http://fs5.directupload.net/images/160605/ste9ny46.png
http://fs5.directupload.net/images/160605/mf2vq5om.png

The max on read is at about 60 MB/s, but writing goes up to over 140 MB/s. The 4TB WD Reds are 5400 RPM drives.

 

[joeschmuck]

I suspect your pool is slowing down your throughput, maybe you could restructure it to be a pool which contains two pairs of two mirrored drives. You get essentially the same thing that you have now but should have better speed. If that doesn't work then I have to assume it's the CPU and/or server hardware causing the slowness.

Also the specs on your HP server is a maximum of 3TB hard drives but you have installed 4TB drives. Not sure that will make a difference at all but just wanted to point that out, and it's SATA II ports but the SATA II shouldn't make any impact in performance for your system.

 

[michaeljk]

I choosed RaidZ2 because of data security (any of the two drives might fail without data loss, instead of two specific ones with a striped mirror). The first two SATA ports on the B120i are using SATA3 (6GBit/s), the last two have only SATA2 (3 GBit/s) - but as you already mentioned, this shouldn't make any impact because the drives itself are slower. Some time ago I noticed a posting here in the forum where a user had performance problems with the B120i controller, but I cannot find the page anymore. He connected the drives to a separate controller and the speed problems were gone. So perhaps I should try to connect the harddrives to a dedicated M1015 (flashed to IT-Mode) and repeat the tests?

 

[Nick2253]

Network and RaidZ2 speed seems ok. On a Linux Mint client I get approx. 38 MB/s write speed over CIFS, with a Macbook Air 58-59 MB/s (over Thunderbird to Ethernet-Adapter). The CIFS settings are on default, except that I disabled DOS attributes like described here:

Unless I'm missing something, if you're getting ~60MB/s with your Macbook air, then you're not being slowed down by either FreeNAS or your network to 38 MB/s. That slowdown must come from your Mint client.

You may have some slowdown due to your controller as far as the Macbook Air is concerned, but the delta between your Mint server and the Macbook Air tells me that there is an issue with your Mint client. Don't overestimate the read speed from a HDD, especially if you are transferring fragmented files from the interior of the platters.

The easiest way to test for this is to create a RAM drive on your Mint client, and copy the files out of that. When you're transferring RAM -> FreeNAS and still seeing slowdowns, we can (likely) rule out any problems with your client.

 

[joeschmuck]

Unless I'm missing something

The test we performed shows a limit within the FreeNAS server.

Some time ago I noticed a posting here in the forum where a user had performance problems with the B120i controller, but I cannot find the page anymore. He connected the drives to a separate controller and the speed problems were gone. So perhaps I should try to connect the harddrives to a dedicated M1015 (flashed to IT-Mode) and repeat the tests?

If you have an M1015 card, What are you waiting for? Make sure you flash it to IT mode of course.

 

[SweetAndLow]

Lets try a test here because I figure that your pool is likely the culprit here...
1) Create a new data store (called "no_comp") and turn off compression (you can't do the dd test with compression on or you get a false reading).
2) In the shell change directory into "no_comp" by typing "cd/mnt/tank/no_comp" (where tank is the name of your pool).
3) Lets run two tests as follows:

Code:

dd if=/dev/zero of=temp.dat bs=1024k count=25k

4) Reboot FreeNAS then do the next line (your file can be cached so rebooting purges this issue):

Code:

dd if=temp.dat of=/dev/zero bs=1024k count=25k

My results are (I used my 'backups' directory because it's already set for no compression):

Code:

[root@freenas] /mnt/farm/backups# dd if=/dev/zero of=temp.dat bs=1024k count=25k
25600+0 records in
25600+0 records out
26843545600 bytes transferred in 57.972478 secs (463039471 bytes/sec)

[root@freenas] /mnt/farm/backups# dd if=temp.dat of=/dev/zero bs=1024 count=25k
25600+0 records in
25600+0 records out
26214400 bytes transferred in 0.190318 secs (137740088 bytes/sec)

NOTE: IF I RUN THE SECOND COMMAND AGAIN, IT'S MUCH FASTER BECAUSE IT'S CACHED
[root@freenas] /mnt/farm/backups# dd if=temp.dat of=/dev/zero bs=1024 count=25k
25600+0 records in
25600+0 records out
26214400 bytes transferred in 0.052949 secs (495088177 bytes/sec)

EDIT: Forgot the cleanup step...

5) Type "rm temp.dat" to remove the file you just created with dd.

When doing your read test you should set your of=/dev/null not /dev/zero. Not sure how it affects your results.

 

[joeschmuck]

When doing your read test you should set your of=/dev/null not /dev/zero. Not sure how it affects your results.

You know, when I was typing it I thought it looked off a little but I couldn't figure out what it was. But it didn't make any difference in my tests.

 

[Ericloewe]

When doing your read test you should set your of=/dev/null not /dev/zero. Not sure how it affects your results.

It's supposed to work either way.