Slow 4K+ performance, speedy benchmarks

Status
Not open for further replies.

Crustacean

Dabbler
Joined
Jan 31, 2014
Messages
22
Mobo: Gigabyte GA-F2A88XM
CPU: AMD A6-5400k
Ram: GSkill ripjaws (8gb DDR3 1600mhz)
Drives: 6x1TB assorted 7200rpm sata3 drives (mostly WD blacks)
Configuration: raidz2

Benchmarks in linux show insane reads/writes. I believe it was 700 read and 350 write.
However I noticed writing to a CIFS share was very slow (12MB/s).
So i looked a little deeper. Ran CrystalDiskMark from a Windows machine and got the following results:
Sequential reads: 100MB/s.... Sequential writes: 95MB/s
512k random reads: 90MB/s... 512k random writes: 80MB/s
4k random reads: 10MB/s... 4k random writes: 9MB/s

Why does my performance decrease like this? I should get more than 12MB/s.. CPU usage is below 50% during benchmarks. Ram usage is maxed out all the time... But still, 12MB/s?

Sorry if im one of a million asking a question like this. I searched around, i couldn't find anyone experiencing performance this poor.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Your being the only one writing to them has nothing to do with how busy they are. Kind of like my asking "how fast did you drive" and you saying "I was the only one in the car" as though that had something meaningful to do with it.

Try logging in to the CLI and using something like gstat to see how busy the disks actually are during the time you think it is being slow.
 

Crustacean

Dabbler
Joined
Jan 31, 2014
Messages
22
Your being the only one writing to them has nothing to do with how busy they are. Kind of like my asking "how fast did you drive" and you saying "I was the only one in the car" as though that had something meaningful to do with it.

Try logging in to the CLI and using something like gstat to see how busy the disks actually are during the time you think it is being slow.


That's why i looked for a disk usage graph!
I had forgotten about gstat. Thanks.

During sequential 4k benchmarking, gstat shows all 6 disks between 30%-60% busy, benchmark reports 100mb/s.
during 512k random benchmarking, gstat shows all 6 disks between 50%-80% busy, benchmark reports 12mb/s

What gives? how could 12mb/s occupy the disks as much as 100mb/s?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
A benchmark doing 4K random reads and writes is deliberately trying to get your system to seek around, that's the purpose of a random read/write test. ZFS allows you to short circuit such tests by throwing more resources at the problem, but basically you have to throw sufficient resources at the problem that the tests all end up mostly cached or something. ZFS cannot completely fix a workload that exceeds the available resources and ways to optimize.

Things to try include:

1) Make sure there's sufficient RAM. RAM is cache. Cache is speed.
2) Once you've maxxed out on RAM (32GB+) you can also add L2ARC to boost read speeds. L2ARC requires a little while to warm up, but after it is warm, it is great.
3) Write speeds are affected primarily by sync requests and by the amount of free space in the pool. Sync requests are mitigated by the use of SLOG for ZIL. Keeping sufficient free space in the pool (like at least half your pool) results in significantly reduced fragmentation and usually much faster writes.

Faster drives will only moderately improve your speeds at significantly increased cost. I am not super convinced that's the right solution for most workloads.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Hint: 12MB/sec is exactly what 100Mb LAN connections transfer data at...
 

eraser

Contributor
Joined
Jan 4, 2013
Messages
147
...
However I noticed writing to a CIFS share was very slow (12MB/s).

Are you copying many small files to your CIFS share? If so, test again by copying a very large file (like an .ISO image or something) and let us know your results.
 
Status
Not open for further replies.
Top