Best option for one single transfer (QD1) performance?

[logan893]

I'm currently running a single vdev mirror using two 4TB drives. My use case is mainly media serving and moving files around, which is a workload with queue depth 1. Mainly over SMB, but also one or two iSCSI targets.

Previously I was maxing out my 1 Gbps network, and has since upgraded to point-to-point 10 Gbps between the server and my workstation. This works great when reading files cached in ARC, but my use case doesn't see much benefit from this most of the time. Perhaps it's my imagination, however I still seem to be limited to the read performance of a single drive. I reach no greater speeds than 100-130 MB/s while reading large (hopefully sequentially written) files. This pool is less than 50% full. Perhaps I should be glad I have this level of performance still. :)

I now have the option to move four 640 GB drives to the server, and would like to set these up in the best way possible for my situation.

Would a single four-drive RAIDZ1 vdev, or two striped two-drive mirrors, give me the best performance both short and long term? Expected space requirement for this pool is less than 800 GB, which eventually may translate into ~50% used space for the RAIDZ1 option, and perhaps too close for comfort to 80% used space for the striped mirrors option.

I may also consider skipping ZFS completely for this specific pool, if I can find a decent alternative.

Are there other options for tuning I should look at?
I've tried modest tuning of "vfs.zfs.vdev.cache.bshift", and SMB's "aio read size", with little noticeable effect.

 

[dlavigne]

Did you decide upon a solution?

 

[logan893]

@dlavigne, no, not quite yet. I plan to briefly test both striped mirrors and RAIDZ1 before deciding which way to go. Mainly I'll run sequential read and write tests on empty pools.

Unless I see a huge difference in performance I'm leaning towards simply going with striped mirrors to make any future upgrade and expansion of the pool easier. I also suspect that an array with striped mirrors will handle future data fragmentation better due to its inherent higher IOPS capacity.

Monitoring my current 2-drive mirror array with gstat shows that both drives work at over 95% busy to achieve 100-130 MB/s. This leads me to believe that a striped mirror will be the best option as long as I can manage to limit the pool to at most 75% space utilization.

 

[logan893]

Just finished some quick write and read tests. Results for both RAIDZ and striped mirror are basically what I was expecting and hoping for.

4-way mirror is a bit disappointing for a single stream read but is likely a very fringe case.

I'll go with the striped mirror vdev setup and see how that works out with my real workload. I'll opt for the higher read performance over write performance as compared with RAIDZ, as well as the improved redundancy and lower CPU load.

ESXi 6.0 U2 host
FreeNAS 9.10.1-U2 guest
Xeon E3-1226v3
2 virtual cores, 12GB RAM
LSI 9212-4i4e (VT-d passthrough)
Four 640 GB WD Black (made in 2010; ~16500 hours on them)

All tests executed with compression off.

Tunables:
vfs.zfs.txg.timeout=1 (Flush transaction group every 1 second; default 5)
vfs.zfs.vdev.cache.bshift=17 (17 means disk reads are padded to 128 kB; default 13 == 8 kB)

Test 1:
4-way stripe (2.2 TB volume; no redundancy!)
%busy write: ~100
%busy read: ~100
%CPU: ~5-10
[logan893@freenas] /mnt/testvol/testset# dd if=/dev/zero of=test64gb bs=1m count=65536 ; dd if=test64gb of=/dev/null bs=1m count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 157.522574 secs (436251612 bytes/sec) [416 MB/s]
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 153.005635 secs (449130365 bytes/sec) [428 MB/s]


Test 2:
4-way mirror (555 GB volume)
%busy write: ~100
%busy read: ~60-65
%CPU: ~5-10
[logan893@freenas] /mnt/testvol/testset# dd if=/dev/zero of=test64gb bs=1m count=65536 ; dd if=test64gb of=/dev/null bs=1m count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 608.633086 secs (112907889 bytes/sec) [107 MB/s]
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 319.446804 secs (215120251 bytes/sec) [205 MB/s]


Test 3:
2-by-2 striped mirror (1.1 TB volume)
%busy write: ~100
%busy read: ~85-90
%CPU: ~5-10
[logan893@freenas] /mnt/testvol/testset# dd if=/dev/zero of=test64gb bs=1m count=65536 ; dd if=test64gb of=/dev/null bs=1m count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 306.125705 secs (224481236 bytes/sec) [214 MB/s]
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 192.311990 secs (357333293 bytes/sec) [340 MB/s]


Test 4:
4 drive RAIDZ (1.7 TB volume)
%busy write: ~70-100 (100 for 2 seconds, drops to 70-80 for one second, repeat)
%busy read: ~50-100 (100 for 2-3 seconds, 50-70 for 2 seconds, rotating between drives)
%CPU: ~15-25 (spikes when write speed drops, didn't look too closely at this one)
[logan893@freenas] /mnt/testvol/testset# dd if=/dev/zero of=test64gb bs=1m count=65536 ; dd if=test64gb of=/dev/null bs=1m count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 215.715633 secs (318565121 bytes/sec) [303 MB/s]
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 208.237363 secs (330005508 bytes/sec) [314 MB/s]


Test 5:
4 drive RAIDZ2 (1.1 TB volume)
%busy write: ~70-100 (100 for 2-4 seconds, drops to 60-90 for 1-2 second, repeat)
%busy read: 2@~100, 2@~30-50 (same two drives stay at 100% busy, with the other two hovering around 30-50%)
%CPU: ~15-25 (spikes to 30-40% briefly when write speed drops, looks like it's close to maxing out one core; ~15 during read)
[logan893@freenas] /mnt/testvol/testset# dd if=/dev/zero of=test64gb bs=1m count=65536 ; dd if=test64gb of=/dev/null bs=1m count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 315.935727 secs (217510939 bytes/sec) [207 MB/s]
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 307.130955 secs (223746502 bytes/sec) [213 MB/s]