RAIDZ1 with four 3TB drives - getting around 12 MB/s writes?

[victorhooi]

Hi,

Ok, so I've created a ZFS RAID-Z1 pool with four drives on a HP N54L.

The drives are 3.5" mechanical drives - two Seagates (7200 RPM), two Hitachi Deskstar 5K3000 (5940 RPM).

I'm doing an RSync from a MacBook Pro connecting to an external drive via USB 3.0, over AFP via Gigabit Ethernet to the NAS. The files I'm copying are larg-ish - around 2-4 GB.

The write performance I'm getting seems a bit on the low side - averaging around 12 MB/s for the pool (if I'm reading zpool output correctly):

Code:

[victorhooi@freenas] /% zpool iostat -v 5
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                245G  10.6T      0    136  2.18K  12.9M
  raidz1                                 245G  10.6T      0    136  2.18K  12.9M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0     74    583  4.47M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0     69    573  4.42M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0     72    545  4.47M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0     68    539  4.43M
--------------------------------------  -----  -----  -----  -----  -----  -----

Can I just double-check if this seems reasonable to you guys? Are there any obvious issues with my setup? (Apart from the fact that I'm running four drives in RAIDZ1 - although my understanding is that this wouldn't have this large an impact on performance).

Cheers,
Victor

 

[gpsguy]

Victor,

Please provide more information about your hardware.

How much RAM do you have in your N54L? 2Gb? ZFS on FreeNAS needs at least 8Gb of RAM. You can put 16Gb in your machine (I have 16Gb ECC RAM in my N54L).

How big are your drives?

How is the MacBook Pro connected to your N54L? Do you have a 10/100 switch/hub/router between them? If you copy files from your MacBook Pro's hard disk (not USB) to your FreeNAS server, what kind of performance do you get?

 

[victorhooi]

Hi,

Sorry, apologies, that was a mistake - it's not a N54L - it's actually a HP MicroServer Gen8 (the model after the N54L) - I also have a N54L...lol.

And this was a Mac Mini with a SSD, not a Macbook Pro.

The Gen8 has 10 Gb of RAM in it (I know ZFS doesn't deal well with constrained memory situations).

Code:

[victorhooi@freenas] /mnt/datastore% dmesg | grep memory
real memory  = 10737418240 (10240 MB)
avail memory = 10126843904 (9657 MB)
pci2: <memory> at device 0.0 (no driver attached)

The drives are all 3TB each.

The switch is a HP PS1810-8G switch (the one they sell with the Gen8 server).

@gpsguy - I think you found the issue - silly, I should have checked this myself.

I think the issue may be the USB 3.0 external drive - I did a copy from the Mac's SSD to the NAS - and I'm getting between 70-90 MB/s:

Code:

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                405G  10.5T      3    761  13.6K  71.2M
  raidz1                                 405G  10.5T      3    761  13.6K  71.2M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    292  4.00K  20.7M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      1    291  4.80K  20.6M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    373  1.60K  24.3M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    336  3.20K  24.3M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                405G  10.5T      4    952  20.0K  90.5M
  raidz1                                 405G  10.5T      4    952  20.0K  90.5M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    423  4.00K  31.3M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      1    392  5.60K  31.2M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    420  3.20K  31.3M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      1    380  7.20K  31.3M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                406G  10.5T      4    924  16.8K  90.4M
  raidz1                                 406G  10.5T      4    924  16.8K  90.4M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    416  4.00K  31.3M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0    370  3.20K  31.2M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      1    413  6.40K  31.3M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    389  3.20K  31.3M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                406G  10.5T      5    981  21.6K  90.7M
  raidz1                                 406G  10.5T      5    981  21.6K  90.7M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    434  4.00K  31.4M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      1    397  7.20K  31.3M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    431  3.20K  31.4M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      1    396  7.20K  31.4M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                407G  10.5T      3    604  13.6K  57.6M
  raidz1                                 407G  10.5T      3    604  13.6K  57.6M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      1    310  7.18K  19.9M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0    295      0  19.6M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      1    306  6.39K  19.9M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    282      0  19.9M
--------------------------------------  -----  -----  -----  -----  -----  -----

But copying from the USB 3.0 external drive to the Mac's SSD - I get between 10-20 MB/s:

Code:

datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.0.shadowIndexGroups
     19,580,706   2%   13.50MB/s    0:00:01 (xfr#37, ir-chk=1175/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.0.shadowIndexHead
     19,584,802   2%   13.38MB/s    0:00:01 (xfr#38, ir-chk=1174/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.directoryStoreFile
     21,681,954   2%   13.70MB/s    0:00:01 (xfr#39, ir-chk=1173/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.directoryStoreFile.shadow
     22,861,602   3%   13.56MB/s    0:00:01 (xfr#40, ir-chk=1172/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexArrays
     60,521,954   7%   15.52MB/s    0:00:03 (xfr#41, ir-chk=1171/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexCompactDirectory
     67,832,458   8%   15.66MB/s    0:00:04 (xfr#42, ir-chk=1170/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexDirectory
     68,797,750   9%   15.68MB/s    0:00:04 (xfr#43, ir-chk=1169/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexGroups
     68,998,442   9%   15.67MB/s    0:00:04 (xfr#44, ir-chk=1168/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexHead
     69,002,538   9%   15.65MB/s    0:00:04 (xfr#45, ir-chk=1167/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexIds
     71,009,458   9%   15.19MB/s    0:00:04 (xfr#46, ir-chk=1166/1220)
datastore_backup/.Spotlight-V100/Store-V2/903BCB91-0446-436A-8B71-3B866206B68D/live.1.indexPositions
 
    245,091,051  32%   18.02MB/s    0:00:12 (xfr#47, ir-chk=1165/1220)

But I'm guessing the 70-90 MB/s seems more in line with what we should be seeing on the NAS, with this sort of hardware? Or is there hope of pushing through more?

So the theoretical limit of Gigabit is 125 MB/s - how close could I be aiming to get close to that?

Cheers,
Victor

 

[eraser]

[[ removed incorrect comment about USB 2.0 speeds ]]

Maybe confirm that your external drive is actually connecting at USB 3 speeds?

12 MB/sec is also close to the max transfer rate for 100 Mbit/sec ethernet. Check to make sure both your FreeNAS and Mac are successfully connecting at gigabit ( 1000 Mbit/sec ) speeds.

*Edit - remove my incorrect comment about USB 2.0 transfer speeds.

 

[eraser]

Hi,

But I'm guessing the 70-90 MB/s seems more in line with what we should be seeing on the NAS, with this sort of hardware? Or is there hope of pushing through more?

So the theoretical limit of Gigabit is 125 MB/s - how close could I be aiming to get close to that?

Cheers,
Victor

70-90 MB/sec is around what I get reading/writing data to my FreeNAS when actually hitting the disks (instead of accessing the ARC). I use CIFS instead of AFP. I also have 4 disks in my server (but mine are set up as a stripe of 2 mirrors instead of 1 raidz). I am running FreeNAS 9.2.

Once your ARC cache warms up then some of your common read requests end up being served from ARC (RAM) instead of from disk with a corresponding increase in read throughput.

I plan to increase my raw IO performance by adding more spindles into my ZFS pool at a future date.

 

[victorhooi]

Hi,

For USB 2.0 - I thought the theoretical limit was 480 Mbps? So that should be around 60 MB/s, right?

I've confirmed on my Macbook Pro, the external drive is definitely using USB 3.0.

And yeah, if it was 100 Mbps - would be around 8.3 MB/s, right? OSX is reporting it as 1000baseT, so that should be ok, right?

I know the Ethernet cable isn't the newest cable - however, if it wasn't able to provide Gigabit - it wouldn't show that, right? Or would it just report that, but transmit at a lower speed?

Anyhow, I just tried something - previously, I had the NAS mounted via AFP, and I was doing rsync to the directory under /Volumes.

Code:

rsync -avz --info=progress2 /Volumes/datastore /Volumes/datastore_backup/

However, I'm now doing it via SSH:

Code:

rsync -avz --info=progress2 /Volumes/datastore root@<ip_address>:/mnt/datastore

and I'm seeing speeds of between 30 - 50 MB/s:

Still not as fast as it should be - so I'm guessing there's something else going on here:

Code:

                                          capacity    operations    bandwidth
pool                                    alloc  free  read  write  read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                717G  10.2T      1    352  7.00K  32.6M
  raidz1                                717G  10.2T      1    352  7.00K  32.6M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    157  4.00K  11.3M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0    142      0  11.2M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    152  3.00K  11.3M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    141      0  11.3M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                          capacity    operations    bandwidth
pool                                    alloc  free  read  write  read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                717G  10.2T      1    476  7.00K  50.8M
  raidz1                                717G  10.2T      1    476  7.00K  50.8M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    291  1023  17.5M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0    277  2.00K  17.5M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    290  2.00K  17.5M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    278  2.00K  17.0M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                          capacity    operations    bandwidth
pool                                    alloc  free  read  write  read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                717G  10.2T      1    334  8.00K  33.0M
  raidz1                                717G  10.2T      1    334  8.00K  33.0M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    164  1023  11.4M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      0    142  1023  11.4M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      0    163  3.00K  11.5M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    163  3.00K  11.5M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
                                          capacity    operations    bandwidth
pool                                    alloc  free  read  write  read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
datastore                                718G  10.2T      3    488  15.0K  47.8M
  raidz1                                718G  10.2T      3    488  15.0K  47.8M
    gptid/36d5bb2e-86cf-11e3-8a5c-10604b92dc14      -      -      0    260  4.00K  16.5M
    gptid/37e9e6af-86cf-11e3-8a5c-10604b92dc14      -      -      1    256  5.00K  16.3M
    gptid/388b045d-86cf-11e3-8a5c-10604b92dc14      -      -      1    257  5.00K  16.4M
    gptid/39059520-86cf-11e3-8a5c-10604b92dc14      -      -      0    230  1023  16.2M
--------------------------------------  -----  -----  -----  -----  -----  -----

But it's definitely faster via SSH than over AFP - can anybody suggest why?

Cheers,
Victor

 

[eraser]

Hi,

For USB 2.0 - I thought the theoretical limit was 480 Mbps? So that should be around 60 MB/s, right?

...

And yeah, if it was 100 Mbps - would be around 8.3 MB/s, right? OSX is reporting it as 1000baseT, so that should be ok, right?

You are absolutely correct on the theoretical USB 2.0 speeds. I don't know what I was thinking when I wrote that. Actually I do remember -- Many years ago I had a 1st Gen NAS storage appliance connected via USB 2.0 and ~12.5 MB/sec was the rate that I could write to it. My brain must have associated the bottleneck with the USB 2.0 connection instead of my particular NAS. I will go and edit my previous reply above.

100 Mbps ethernet should have a theoretical max transfer rate of 12.5 MB/s. ( and 1 Gbps has a theoretical max of 125 MB/sec).

 

[krikboh]

I assume your external USB drive contains a single 5400 RPM drive. Not unreasonable to only hit 30-50 MB/s with a single slow drive over USB.

 

[Durandal]

Hi,

For USB 2.0 - I thought the theoretical limit was 480 Mbps? So that should be around 60 MB/s, right?

No, since USB 2.0 can't send 480 Mbit/s in both directions full duplex, the teoretical maximum transfer rate of USB 2.0 is around 30-32 MB/s. With some small hacks you can achieve around 35-38 MB/s but that with some non-standard software.

(USB 3.0 can send all data in full duplex which is a big improvement)