HP N54L RaidZ2 Performance

[shimian5]

Finally got the new NAS built!

I'm running an N54L w/8GB RAM, 5x4TB Seagate 5900 RPM, FreeNAS9.3 latest stable and the stock NIC. LZ4 compression is turned on, but does not seem to impact my overall write speeds. The drives are all in 1 vdev. SMBD uses anywhere from 25% to 70% of CPU.

My CIFS transfer speeds are about 52MB/s write and about 65MB/s read. ZPOOL iostat looks as follows

Code:

pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
VOL1        98.4G  18.0T      7    662  67.9K  64.6M
VOL1        98.4G  18.0T      3    598  24.0K  63.4M

A DD test had the following results:

Write

Code:

[root@NAS1] /mnt/VOL1/TEST# dd if=/dev/zero of=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 101.836529 secs (205933177 bytes/sec)

Read

Code:

[root@NAS1] /mnt/VOL1/TEST# dd of=/dev/zero if=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 47.284432 secs (443518491 bytes/sec)

Are those within the normal range for this server, or are they a bit low?

 

[gpsguy]

Those are excellent speeds for a N54L. CIFS is a single threaded protocol and you only have a 2.2GHz CPU.

Are you aware that you can put 16Gb of RAM in the box? I put 16 in my N54L.

Since you are running 5 drives in it, are you running a hacked BIOS? The hacked BIOS enables AHCI on the ODD and eSATA connections.

 

[shimian5]

I do have the hacked BIOS, so all 5 drives run at full speed.

My CPU hovers at about 25% when writing a single file over CIFS. The peak I can get with a Windows file copy is ~48MB/s

Are there any CIFS tweaks to be made? Is there benefit in swapping in an Intel PRO/1000 NIC?

I also want to add that FTP has the same maximum speed as CIFS. From what I've been reading, FTP should be able to push it faster. Could this be NIC related?

I will also try another gig switch.

 

[shimian5]

UPDATE:

Tried a spare 1gbit switch (was using my router with 1gbit switch), and voila, 85MB/s write and 90MB/s read!

I still might swap out the NIC in the N54L for a faster one if anyone can let me know that it made any difference.

 

[gpsguy]

The onboard Broadcom NIC isn't bad. If it was a Realtek, I'd upgrade in a heartbeat. I doubt you'll see any significant difference by adding an Intel NIC.

 

[ovizii]

Can I ask a few questions in here since I have very similar hardware (4x 1TB 7200rmp) and speed related questions? If not, please feel free to move to a new thread.

Initially I had no idea how to measure my speed, I just perceived that copying files from my MacBook to the NAS via a LinksysWRT54G which acts as a repeater was painfully slow so I have now tried all the methods the initial poster used:

Code:

[root@freenas] ~# zpool iostat
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
freenas-boot   534M  6.92G      0      0  2.05K    652
zfsmain      680G  2.96T      1     44   137K  1.31M
----------  -----  -----  -----  -----  -----  -----
[root@freenas] ~#

write

Code:

[root@freenas] ~# dd if=/dev/zero of=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 134.705635 secs (155684059 bytes/sec)

read

Code:

[root@freenas] ~# dd of=/dev/zero if=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 22.963114 secs (913269863 bytes/sec)

Seems very close to the first person's values except for the write speed of 1.31M in the first test vs. the other person's 64.6M

I've just tried the first test again after rebooting:

Code:

[root@freenas] ~# zpool iostat
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
freenas-boot   534M  6.92G      6      0  79.4K  8.01K
zfsmain      595G  3.04T      2     30  11.6K   186K
----------  -----  -----  -----  -----  -----  -----

Any suggestions? I'm not looking to tweak and tune until I drop but if my Macbook takes 3h to copy 5GB that's definitely too slow. How would I determine the bottle neck? I'm sure I had faster speeds with the same setup earlier or possibly before upgrading to 9.3
###edit###
I had fiddled with Acoustic Level and Advanced Power Management settings for my HDs could that have influenced my performance this badly?

 

[gpsguy]

One big bottleneck is your 10/100 Linksys router. Buy a gigabit Ethernet switch, plug FreeNAS and your MacBook into the new switch and then run a cable from the switch to your router.

Start there and report back.


Sent from my phone

 

[ovizii]

I'M very well aware of that but cannot do anything about that right this moment.
The point is I am 100% sure I had better rates with the same combo.
Any other ideas?

 

[gpsguy]

Do a direct connection between your server and your MacBook, by passing the router.

BTW, hopefully you have at least 8GB RAM in your Microserver.


Sent from my phone

 

[ovizii]

Jepp, I've got 8GB and I figured out I don't even need a cross over cable with my MacBook. The speed is indeed adequate now, about 5mins for 10GB.
So its definitely the router, gotta make a plan and replace it.
THANKS!

 

[gpsguy]

As I said earlier, you don't have to replace the router. You could buy a gigabit ethernet switch to plug your devices into and then run one connection from it, out to your router.

 

[ovizii]

Its a bit more complicated, there's a glasfiber router on the first floor and the wrt54gl is on the groundfloor connected to the upstairs one via wireless to provide ethernet for the NAS as I have heard wifi isn't recommended for freeNAS so I'll have ot move the NAS upstairs or figure something out. Cabling is out of the question unfortunately...

 

[rogerh]

If the WiFi is only connecting the FreeNAS server to the Internet, and all your client computers are connected to the WRT54GL (or a switch next to it, with a patch cable to the WRT54GL, which doesn't really affect the quality of the Internet connection) then this should be fine. Unless you want to serve files over the Internet, in which case WiFi would be a bad idea.

 

[ovizii]

glas fiber => fritzbox router => wifi => wrt54g => ethernet => freeNAS
glas fiber => fritzbox router => wifi => macbook

 

[HoneyBadger]

glas fiber => fritzbox router => wifi => wrt54g => ethernet => freeNAS
glas fiber => fritzbox router => wifi => macbook

You have two wireless hops in the FreeNAS->Macbook path, I'm surprised your speeds aren't worse.

IMO as soon as "wireless" is in the picture you should just assume that your speeds will be terrible.

Take the FreeNAS machine and plug it directly into the Fritzbox router if it has a port, otherwise do this:

Code:

glas fiber => fritzbox router => switch => freenas
                              => wifi => macbook

 

[ovizii]

Yes, I know and will do. I was just trying to figure out if my problems did indeed come from the wifi part and that is now confirmed by plugging it directly into my macbook. Next step is to convince my parents that my NAS HAS to be upstairs, there is no noticeable noise and yes, it has to run the whole time ;-)