ZFS raidz low read speed

[Matt]

So I have just build a new system and I am wondering why my read speed is so much lower than my write?

I have advanced format hard drives (4k sectors) and I have read as much as possible on the subject on how to properly set up the array and I am wondering if i missed something. From looking at benchmarks from other similar setups my write speed appears to be on par, about 500 MB/s while my read speed is around 150 MB/s.

I have used gnop create -S 4096 for all of my drives and the created the pool from them. That increased my write speed but not my read.

If more information is necessary I will happily provide it. Thanks for any effort put into solving my problem.

 

[matthewowen01]

how are you benchmarking?
how is your pool setup (disks vdev type)

 

[Matt]

To benchmark I am using

Code:

for write
dd if=/dev/zero of=/mnt/pool1/zerofile bs=10m count=50k
51200+0 records in
51200+0 records out
536870912000 bytes transferred in 1030.379909 secs (521041712 bytes/sec)

for read
dd of=/dev/null if=/mnt/pool1/zerofile bs=10m count=50k
51200+0 records in
51200+0 records out
536870912000 bytes transferred in 3308.693639 secs (162260690 bytes/sec)

I have 8 Samsung F4EG drives in a single raidz pool

 

[matthewowen01]

that does seem low.

how are the drives connected to your computer, what controllers are the connected too.

oh and everything else about your system would help, cpu, memory, MB...

 

[Matt]

Ok here is the full setup

8x SAMSUNG EcoGreen F4
HighPoint RocketRAID 2320 PCI Express x4
Drives configured individually as JOBD in card config

Intel Pentium G840 Sandy Bridge 2.8GHz Dual-Core CPU
Intel BOXDH67CLB3 LGA 1155 Intel H67 Motherboard

4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1333

CORSAIR 650W 80 PLUS PSU

FreeNAS Build: FreeNAS-8.0-RELEASE-amd64
and as far as I know everything is set to default except for the creation of the pool and I am using whatever drivers are included with freenas.

 

[matthewowen01]

your setup looks nice enough.

from the commad line, can you type 'zpool status' and post the results.

 

[Matt]

Code:

freenas# zpool status
  pool: pool1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0

errors: No known data errors

 

[matthewowen01]

well i'm struggling at this point.

can you poke around the RocketRAID's bios and see if there is a way to disable it's cache and possibly it's read ahead settings?

 

[Matt]

Well I poked around the card's config and could not find anything like. Also I flashed it with the most recent BIOS which did not seem to make any difference. It is still fast enough to saturate gigabit so it is not terribly important but it is still bugging me.

 

[SoftDux-Rudi]

What happens if you connect the drives directly to the motherboards's SATA ports? You didn't say which mobo you have, so I don't know if it has 6 or 8 ports. But, what performance do you get by bypassing the RAID card? If possible?

 

[Matt]

It has 4 internal and 2 eSATA so I will test what I can. I still can return the card if it is the problem, any recommendations for a replacement.

 

[matthewowen01]

i can recommend this one with high marks

LSI Internal SATA/SAS SAS3081E-R 3Gb/s

you can get it from newegg for $200 not including cables.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118100

you'll need 2 SFF-8087 MiniSAS to 4 SATA Breakout Cable

here are 3 options, different lengths and connector angles ($20 - $30 each):
http://www.newegg.com/Product/Produ...&cm_re=miniSAS_to_sata-_-16-151-080-_-Product
http://www.newegg.com/Product/Produ...&cm_re=miniSAS_to_sata-_-16-151-082-_-Product
http://www.newegg.com/Product/Produ...&cm_re=miniSAS_to_sata-_-16-151-081-_-Product

 

[SoftDux-Rudi]

Matt it may not necessarely be the card itself, but rather the PCI bus on your motherboard that's rather slow. So even if you do replace the card you may still have the same results.

 

[Matt]

I a pci express 4x on quality motherboard I doubt that is the problem. Also it was the only card in the system. Even if so why would the write be 4 times faster than the read on a 500GB file, to effectively remove caching from the equation.

 

[Matt]

An update. I have now LSI Internal SATA/SAS SAS3081E-R 3Gb/s and
http://www.newegg.com/Product/Product.aspx?Item=N82E16816151080&cm_re=miniSAS_to_sata-_-16-151-080-_-Product cables but now the card will not detect any of my drives. Is my build just cursed?

 

[matthewowen01]

when your system is booting, does the card list the drives?

while your system is booting there should be some key combo to enter the bios for the card. take a look in there.

next time i reboot, I'll take a look and see if i remember tweaking something.

 

[Matt]

No it only detects one drive and it is listed as disk 7 in the card config.

 

[matthewowen01]

No it only detects one drive and it is listed as disk 7 in the card config.

finally got around to rebooting. i took screen shoots of all the pages of my controller.