Very slow CIFS file transfers

[knubbze]

I have a HP N40L Microserver running FreeNAS 8.3.1-RELEASE-p2-x64 from a USB stick, with 16GB of ECC RAM and 4 x 2TB HDDs in RAIDZ2 configuration, with encryption enabled.

I've set it all up, configured shares etc, but I am getting extremely slow file transfer speeds (~7MB) when copying files from a Windows machine via a wired gigabit connection (Windows machine -> cat6 -> gigabit switch -> cat6 -> Microserver). When copying files FROM the NAS, I get ~40MB/s sustained transfers. It's just copying files TO the NAS that is very slow.

I also tried enabling 'Large RW support', 'Send files with sendfile(2)', and 'Enable AIO' in the CIFS service settings (as per the suggestion in the FreeNAS user manual), but this didn't make any discernable difference to the file transfer speed. I also enabled the 'autotune' checkbox in the 'Advanced Settings' page; again, this didn't really make any discernable difference.

I ran 'dd' several times with different block sizes and counts to benchmark the disk I/O; here are the results:

Code:

[knubbze@freenas] /> dd if=/dev/zero of=/mnt/storage/Public/test1.dd bs=2M count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 739.207902 secs (28370259 bytes/sec)
 
[knubbze@freenas] /> dd if=/dev/zero of=/mnt/storage/Public/test2.dd bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 14.258437 secs (73540740 bytes/sec)
 
[knubbze@freenas] /> dd if=/dev/zero of=/mnt/storage/Public/test3.dd bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 35.538892 secs (29505028 bytes/sec)
 
[knubbze@freenas] /> dd if=/dev/zero of=/mnt/storage/Public/test4.dd bs=2M count=1000
1000+0 records in
1000+0 records out
2097152000 bytes transferred in 58.855795 secs (35632039 bytes/sec)
 
[knubbze@freenas] /> dd if=/dev/zero of=/mnt/storage/Public/test5.dd bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 346.735037 secs (30241420 bytes/sec)

I also ran 'iperf' several times to test the network bandwidth/performance:

Code:

[knubbze@freenas] /> iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49345
[ ID] Interval      Transfer    Bandwidth
[  4]  0.0-30.0 sec  1.61 GBytes  461 Mbits/sec
[  5] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49348
[  5]  0.0-10.0 sec  633 MBytes  530 Mbits/sec
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49349
[  4]  0.0-10.0 sec  513 MBytes  429 Mbits/sec
[  5] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49350
[  5]  0.0-10.0 sec  656 MBytes  549 Mbits/sec
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49351
[  4]  0.0-60.0 sec  3.51 GBytes  502 Mbits/sec

The results of the tests (all run one after another, while the box was otherwise idle) are pretty inconsistent and vary wildly - what would be the reason for this? And what other reasons could there be for such slow file transfers? I really need to figure this out, as I have several TBs of data which I need to transfer over to the FreeNAS box ASAP.

 

[cyberjock]

First, your dd commands need higher counts. 2GB test file is too small when you have 16GB of RAM.

Nonetheless, your speeds fluctuate pretty wildly between tests. My first guess is you have a hard drive that is having problems.

dd command is pretty much a best case scenario for disk writes. The fact that you are only doing about 30MB/sec means that your shares have no chance of getting that speed. I wouldn't even expect 1/2 of that speed to be honest. You need to start looking at your hardware more closely and see if something is wrong, not working correctly, etc.

Maybe your CPU is overheating causing it to underclock a lot, failing hard drive, etc.

I'm not sure what speeds you were hoping to get, but that's not a high performance CPU, and I'd be impressed if you could get 50MB/sec to the server during normal conditions.

 

[knubbze]

First, your dd commands need higher counts. 2GB test file is too small when you have 16GB of RAM.

I did several dd runs, with 10Gb and 10GB test files.

Nonetheless, your speeds fluctuate pretty wildly between tests. My first guess is you have a hard drive that is having problems.

I also checked for this by running SMART tests, all of which showed no apparent problems,

dd command is pretty much a best case scenario for disk writes. The fact that you are only doing about 30MB/sec means that your shares have no chance of getting that speed. I wouldn't even expect 1/2 of that speed to be honest. You need to start looking at your hardware more closely and see if something is wrong, not working correctly, etc.

Maybe your CPU is overheating causing it to underclock a lot, failing hard drive, etc.

I'm not sure what speeds you were hoping to get, but that's not a high performance CPU, and I'd be impressed if you could get 50MB/sec to the server during normal conditions.

I suspected that having encryption enabled may be the cause of the bottleneck, so I added a 5th HDD to the Microserver and set it up as a single-drive RAIDZ volume, thinking/hoping that I would get much better CIFS transfer rates when writing and reading data to it. Unfortunately, the CIFS write speed (again, across gigabit network) increased just by ~4MB to ~12MB/s, and reads (copying files from the NAS to my desktop machine) stayed the same, ~40MB/s.

Has anyone else suffered lower-than-expected CIFS transfer rates with HP Microservers? I've read a few posts saying that the FreeBSD driver for the unit's inbuilt Broadcom NIC has problems, and suggesting that I drop an Intel CT NIC into the box. But these posts were from 2011, so I'm not sure if the driver has been improved/fixed since then?

 

[cyberjock]

Encryption is a performance killer, bigtime. Even high end desktop CPUs with AES-NI(your CPU doesn't have this instruction set) can do a max of 200-300MB/sec assuming no other system load. ZFS + your other services can constitute a significant load.

As for the dd command, you really want to do a test that is 2-3 times your total system RAM to ensure that the results aren't being cached. I can do dd commands that can show you that I can do more than 50TB/sec, but we both know that's impossible. The dd results don't matter at this point though since you've admitted to using encryption.

I don't recommend encryption if performance matters unless you are using a CPU that has AES-NI support. Check out this link for people that have kindly done testing and shown you the results for some CPUs. Feel free to add your CPU to the post... the more the merrier. When I build FreeNAS servers if I think there's even a remote chance they'll want AES-NI I recommend the chip proactively. If you get an Intel Xeon(which supports ECC) then they all support AES-NI anyway. AMD is sketchy and as I'm not well informed with AMDs so I'll leave that to you to research if you want to go with AMD.

 

[gpsguy]

As cyberjock said, I believe your biggest problem is due to encryption on a mediocre CPU.

I'm running FN 8.3.0, on a stock N40L upgraded to 8Gb ECC RAM, with a pair of 7200 RPM Seagates in a ZFS mirror.

While I don't have CIFS statistics handy, on Sunday I think it took about 30 minutes to transfer a 65Gb file from my desktop to my server.

 

[knubbze]

Thanks for the info. The thing is though, suspecting that encryption was the source of the bottleneck, I added a 5th HDD to the Microserver and set it up as an unencrypted single-drive RAIDZ volume, thus eliminating encryption and RAIDZ2 striping from the equation. Unfortunately, the CIFS write speed (again, across gigabit network) increased just by ~4MB to ~12MB/s, and reads (copying files from the NAS to my desktop machine) stayed the same, ~40MB/s. Surely,with 16GB ECC RAM, I should be getting better speeds than this? I've seen posts from people using the same N40L Microserver with less RAM who are getting speeds far in excess of this.

 

[knubbze]

As cyberjock said, I believe your biggest problem is due to encryption on a mediocre CPU.

I'm running FN 8.3.0, on a stock N40L upgraded to 8Gb ECC RAM, with a pair of 7200 RPM Seagates in a ZFS mirror.

While I don't have CIFS statistics handy, on Sunday I think it took about 30 minutes to transfer a 65Gb file from my desktop to my server.

That works out at 36.11 MB/s, much faster than the 12MB/s that I get when transferring files to my N40L running a single-disk ZFS volume with no encryption :/

Are you using the built-in NIC on your N40L or an add-in one?

 

[gpsguy]

Just the built-in one.

Are you using the built-in NIC on your N40L or an add-in one?

How did you connect your 5th hard disk? To use the CD-Rom connecter, you need to run a hacked BIOS version.

 

[cyberjock]

Knubbze,

When you have an underpowered system like yours, small things that differentiate your system from another can cause major performance issues. It's inherent with an overworked system. Just like an overworked employee results can vary widely, even from one job to another.

You may see some alleviation if you use an Intel NIC. I'm fairly sure that the onboard is a Realtek, which loads the CPU even more at a time when you aren't looking for more load.

You mentioned above that you tried enabling 'Large RW support', 'Send files with sendfile(2)', and 'Enable AIO' in the CIFS service settings.

Try my settings:

In the CIFS settings page here's my values after the "Directory mask".

Only the following is checked: Send Files with sendfile(2), and zeroconf share discovery. All other settings fields are empty except AIO which is set to 4096(but doesn't really matter since AIO is disabled)

Do a reboot after trying those settings and see what happens.

I know that there's some good Samba settings around the forums that people use, but the defaults in Samba have changed over the last 2 years and many of those are the defaults now. If you choose to set your settings to the defaults it may seem like a zero-sum game, but there is a loss. If the Samba developers change their settings to help performance you will still be stuck with the settings you have forced. As a rule I tell people(especially people that aren't thoroughly familiar with Samba, FreeBSD, etc.) not to mess with tweaking because you're far more like to get it wrong than right. There's only 2 people on the forum I'd trust to tweak a system if I needed it, and I know we'd probably spend weeks trying to get the right settings dialed in. The defaults are the defaults for a reason, they work for most people. I know everyone wants to think they're special and they know better, but the truth from observing people tweaking their systems on the forum is that you aren't as special as you think(sorry).

Keep in mind that if your dd commands are still not doing at leats 50-75MB/sec, I wouldn't expect to ever hit 40MB/sec over a share, even in best case. I've seen that sharing over CIFS results in a best case scenario of about 1/2 of what a dd can do. There are exceptions, but that seems to work out as a pretty good guesstimate. So focus on the dd commands and worry about the file sharing later. Maybe gpsguy can post the output of a dd command test on his system for comparison.

You might want to run a long SMART test on all of your hard drives just to verify one isn't failing. A bad disk can kill your system performance and result in very unpredictable performance issues. I know you said there were no problems, but if you could post the SMART info on your drives one of us more knowledgable guys can see if something looks amiss. Quite often people think its good when its not.

 

[jgreco]

No, the NxxL's come with Broadcom bge, technically the HP NC107i / BCM5723. They're good server-grade Ethernets. I used to prefer Broadcom to the Intels, except some of them failed here in some spectacular ways that I'm not convinced were entirely Broadcom's fault (problematic Tyan boards). However, Intel's enthusiastic driver development on FreeBSD absolutely rocks, and Broadcom is often disappointingly silent.

If it seems to be working, I would expect it to be performing within ~10% of the efficiency of an Intel, and not an obvious target for being a problem.

 

[gpsguy]

I can run a dd test tonight, what block size and count should I use?

 

[gpsguy]

I wasn't sure what to set my blocksize and count to, but decided on the values below. This test yielded a write speed of ~77.5MB/s.

Code:

[root@freenas] /mnt/vol1/home/gpsguy/FreeNAS# dd if=/dev/zero of=zerofilenew.000 bs=1m count=20000
20000+0 records in
20000+0 records out
20971520000 bytes transferred in 258.195024 secs (81223564 bytes/sec)

Maybe gpsguy can post the output of a dd command test on his system for comparison.