Extremely slow ST4000DM000 drives...

[helloha]

Hi,

This is my current setup:

Supermicro X7DWN+ - Dual Xeon 5130 - 64GB Ram - 8x 4TB Seagate ST4000DM000 5900 rpm - 2x 4TB HGST HDN724040ALE640 7200 rpm - 1x 256GB Crucial_CT256MX100SSD1 - 6x 3TB HGST 7200 rpm -TDK LoR TF30 USB 3.0 PMAP (boot) - 2x Dell H310 SAS/SATA Controller in H16 IT Mode - 1x HP360T Gb NIC - Supermicro SC825 2U Chassis 920Watt Platinum PSU.

When I put 6 3TB 7200 rpm disks into stripe as a test I get 550 MB/s

When I put the 8x 4TB ST4000DM000 and 2x 4TB HDN724040ALE640 in 2 Vdevs of 5 disks, each vdev in RaidZ1 I get a measly 230 MB/s.

I also put all of them in RaidZ2 but it was even worse. Like 180 MB/s

They are all aligned properly but I can't seem to figure out why they are so slow.

Sure they are 5900 rpm drives, but if you put a bunch of them together they must be faster then this...

When I do my testing each drive has a R/W of about 20 MB/s.

Anyone any suggestions of what to try to get better performance?

Cheers,
K.

 

[L]

This performance is exactly as expected, actually you are getting much better than I would expect. Your test must be laying the data out sequentially or pulling the read data out of cache.

With the 2 vdev's you typically can only expect the performance of a single disk in the vdev. If you had 3 vdev with 3 disks each you would get better, 4 vdevs better.. and so on.. More vdevs, better performance, less vdev worse performance. On 5600(5900?) rpm drive I usually will spec at apx 50MB/sec for random. You are getting 115MB/sec per vdev, which is very good for the layout you have choosen and the test you are doing.

http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

 

[helloha]

Hi, thanks for the reply and the clarification. Yes these were sequential writes. So the only way to increase throughput is to split the drives up into more vdevs like 3x3 for a 3x increase or 4x3 for a 4x increase. I'll do some experimenting. But I do lose a lot of capacity...

Is this only inherent to ZFS? I am wondering if it's worth it in my scenario. The server is an onsite backup so data integrity is perhaps not super important. It's for video editing, so maybe UFS would be a better choice if it would provide more speed...

Thanks!
K.

 

[L]

If you don't need integrity or redundancy then the stripe might work.

I would be interested to see how you would fair graid and ufs. Although there was a day and time when I was a UFS expert, those parts of my brain have been overwritten. UFS won't cache as much file data as you will see with ZFS

Is that a good testing for your workload? ZFS usually does pretty well with video editting because you will end up caching a lot of frequently used data into ram. Use a 1 Meg block size too.. If you have one user and one file though, caching will not be as helpful.

 

[Bidule0hm]

You list a Gbe NIC so even with you worst test you can saturate it, why you want more pool speed if the network can't follow?

 

[Bidule0hm]

Please note that there's only ZFS in FreeNAS 9.3, no UFS.

 

[helloha]

I need the IOPs, not so much the throughput. The editing part works pretty well since proxy file formats are about 30 Mbit/s. But if you deal with large projects of 100+ MB opening them takes a long time. It's essentially a big database that needs to be verified and linked to the media that lives on the server. That can take 3-5 minutes on large projects and is a drag...

Where can you set the block size you mentioned?

 

[Bidule0hm]

Ok. If you want IOPS only then use a stripe. If you want IOPS + some redundancy then use striped mirrors ;)

 

[helloha]

Just out of curiosity,

How is it that ZFS actually bottlenecks the drives? If I check my graphs in freenas I can see that my ST4000DM000 disks read and write at about 20 MB/s in this 2 vdev configuration. Like Linda Kateley mentioned, you get the performance of the amount of vdevs.

Why is this exactly, since the disks can average 80-120 MB/s individually?

Cheers!
K.

 

[helloha]

Think I already found it or something similar:

The RAID penalty for each write operation.

• RAID5 & RAID6 require 4+ disk operations for every write. The controller must read the block and then read the parity data (two operations), calculating the new parity data and then update the parity block and update the data block (Two more operations). RAID6 has two parity blocks and therefore requires three reads and three writes. RAID5 & RAID6 arrays are therefore capable of fewer IOPS then RAID1.

http://serverfault.com/questions/53...-different-then-calculating-iops-for-raid5-ra

 

[cyberjock]

Think I already found it or something similar:

The RAID penalty for each write operation.

• RAID5 & RAID6 require 4+ disk operations for every write. The controller must read the block and then read the parity data (two operations), calculating the new parity data and then update the parity block and update the data block (Two more operations). RAID6 has two parity blocks and therefore requires three reads and three writes. RAID5 & RAID6 arrays are therefore capable of fewer IOPS then RAID1.

http://serverfault.com/questions/53...-different-then-calculating-iops-for-raid5-ra

That is for RAID5 and RAID6. That is NOT For ZFS' RAIDZ1 and RAIDZ2. ;)

 

[helloha]

I know but I'm sure some of the same magic happens in ZFS no? a lot of writing, updating etc of parity data without writing the actual data, thus taking up the iops and bandwidth of the disk.

 

[cyberjock]

Some, but not all. So how much difference is there? (The question is rhetorical. If you can't answer the differences, then you can't truly answer the question you originally asked, "Why is this exactly, since the disks can average 80-120 MB/s individually?")

So it would appear that you haven't found an answer at all. :P