RAIDZ reads are slower than writes

Status
Not open for further replies.

fcueto

Dabbler
Joined
Aug 10, 2014
Messages
26
Hi there,

I installed freenas a few days ago.

My hardware is as follow:
- HP Microserver gen 8
- upgraded CPU Intel Xeon E3-1265L V2
- 16 GB of ECC UDIMM
- 4 x 4TB seagate NAS HDD ST4000VN000
- Single RAIDZ1 pool

I have a major issue with ZFS speeds. Reads are slower than writes. I was hoping to have at least read speed = write speed....

Writes speed is ~100MB/sec:
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1024 count=300k
307200+0 records in
307200+0 records out
314572800 bytes transferred in 2.949346 secs (106658491 bytes/sec)
2.98s real 0.08s user 2.86s sys



Read speed is really slower (~67MB/sec):
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=sometestfile of=/dev/zero bs=1024 count=300k
307200+0 records in
307200+0 records out
314572800 bytes transferred in 4.446092 secs (70752654 bytes/sec)
4.44s real 0.01s user 4.43s sys


I tried with higher counts to minimize the "write cache" effect but I doesn't change anything.
I checked and prefetch is enabled. I can't see where I should dig...

Can you help me please ?

Thanks
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
Not sure if this will make a difference, but I'm pretty sure you want your read destination to be /dev/null not /dev/zero.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
And you want block size = 1M, not 1024.

Doing tests that actually use proper parameters would be an excellent start. ;)
 

fcueto

Dabbler
Joined
Aug 10, 2014
Messages
26
I'll give a try with both your suggestions tonight but:
  • as fas as I know, writing to /dev/zero is the same than writing to /dev/null. Extract from Wikipedia : Unlike /dev/null, /dev/zero may be used as a source, not only as a sink for data. All write operations to /dev/zero succeed with no other effects. However, /dev/null is more commonly used for this purpose.
  • I'll try with bs=1M instead of bs=1k. It probably will enhance performances, but for both read and write and I will still have write speed > read speed
I'll let you know.

One other thing I forgot to mention : compression is active on the volume (lz4)
 
Last edited:

fcueto

Dabbler
Joined
Aug 10, 2014
Messages
26
OK, I disabled compression on the dataset and made some new tests.

cyberjock, you were right once again :)

Write speed with bs=1M is ~247MB/sec
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 41.454704 secs (259015679 bytes/sec)
41.45s real 0.00s user 2.80s sys



Read speed with bs=1M is ~500MB/sec !!!
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=sometestfile of=/dev/null bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 20.348966 secs (527664074 bytes/sec)
20.38s real 0.00s user 1.39s sys


I have to say I'm pretty impressed !

Thank you all
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
Those are some killer speeds. You're running SSD array?
 

fcueto

Dabbler
Joined
Aug 10, 2014
Messages
26
No. See my config on the first post.

I think the cache is involved in these results. I have to do it again with more data.
 

fcueto

Dabbler
Joined
Aug 10, 2014
Messages
26
New tests, compression still disabled:

Write : ~249MB/sec
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1M count=100k

102400+0 records in
102400+0 records out
107374182400 bytes transferred in 410.730264 secs (261422622 bytes/sec)
6m50.73s real 0.11s user 28.12s sys


Read : ~279MB/sec
[root@freenas] /mnt/volume1/documents# /usr/bin/time -h dd if=sometestfile of=/dev/null bs=1M count=100k
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 366.350479 secs (293091421 bytes/sec)
6m6.35s real 0.03s user 18.02s sys



It seems more realistic !
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Well, the last test is the only test that is really worth looking at. A 4 disk RAIDZ1 with your disks would typically give 200-300MB/sec and you are pretty much right in the middle. So I'd say 'nothing is broken'.
 
Status
Not open for further replies.
Top