RAIDZ slow local read

[sampledi]

Hello everybody,
I've seen one similar thread but I wasn't able to fix this issue. Read speed on my 5x1TB RAIDZ is much slower than it should be (and two times slower than write):

[root@freenas] ~# dd if=/dev/zero of=/mnt/RAID/tmp.000 bs=1M count=50k
51200+0 records in
51200+0 records out
53687091200 bytes transferred in 169.153484 secs (317386849 bytes/sec)
[root@freenas] ~# dd if=/mnt/RAID/tmp.000 of=/dev/null bs=1M count=50k
51200+0 records in
51200+0 records out
53687091200 bytes transferred in 370.134838 secs (145047387 bytes/sec)

RAID is 65% full

System specs:
FreeNAS-9.3-STABLE-201506292332
Supermicro X8SAX
Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
24GB ECC RAM
5x1TB WD Red RAIDZ
Intel 4 port server ethernet adapter

Compression Off
atime Off

When system was built (and empty) last year, read speed was over 400MB/s and I'm not sure what's causing such huge drop in performance.

Any help is much appreciated :)

 

[SweetAndLow]

You can test the read speed of each drive individually if you want. You can also check smart data to see if the disks are having problems.

 

[Robert Trevellyan]

When system was built (and empty) last year, read speed was over 400MB/s

At 65% full it could be pretty well fragmented.

Compression Off

This could actually be slowing it down, though not dramatically.

 

[sampledi]

I wan't aware that fragmentation can affect read speed so significantly already @ 65%.
Smart hasn't reported anything bad, so I guess I'll have to destroy and recreate this pool.

 

[Robert Trevellyan]

I wan't aware that fragmentation can affect read speed so significantly already @ 65%.

I didn't mean to suggest that this was necessarily the complete explanation, just that it could easily be a factor, perhaps even a major factor. There may be other issues, e.g. a lower ARC hit ratio for a larger working set. It was also suggested that a bad drive can cause poor performance, which is definitely worth checking, although with WD Reds and TLER this would not have as much impact as with consumer drives.

 

[Robert Trevellyan]

I guess I'll have to destroy and recreate this pool.

This would be a good opportunity to switch to RAIDZ2.

 

[sampledi]

This would be a good opportunity to switch to RAIDZ2.

There is no need for RAIDZ2 because on that NAS I need speed and space. Everything is rsynced to another NAS in another room, with separate UPS + I have third (manual) backup with important things on external location.
And online backup too :)
Actually even RAID0 wouldn't be so bad :)

 

[Robert Trevellyan]

even RAID0 wouldn't be so bad

Well, a stripe would also be the fastest configuration, with the maximum storage space too.