2 drive mirror READ performance (no ARC) beats 2x2 mirror?

Status
Not open for further replies.

xyzzy

Explorer
Joined
Jan 26, 2016
Messages
76
I've got two volumes on the same FreeNAS 11 box that use HDDs for data storage:
-- #1 is a 2 drive mirror using 5 year old WD Se 4 TB HDDs (WD Se was the predecessor to WD Red Pro)
-- #2 is a 2x2 mirror (like a RAID10) using brand new WD Gold 10 TB HDDs

Both are set up the same way with full disk zvols that are used as ESXi datastores via iSCSI.

#1 is 83% full (usually I keep this under 50%) and #2 is 8% full. #1 is significantly fragmented, #2 has almost no fragmentation as I just set it up a few days ago.

When reading large files (> 1 GB each) off both volumes, I'm finding that the 2 drive mirror is at least 25% faster than the 2x2 mirror when the reads are *not* coming from the ARC. IE, the actual disk read performance is being tested here because I'm reading seldomly accessed files. I've tested this two ways:
1) From the ESXi console running "dd" commands that output to /dev/null
2) From a Win 2016 VM that has a VMDK on each volume, copying files from each disk to a RAM disk

In both cases, I'm monitoring the disk performance on the FreeNAS box using "zpool iostat -v 1".

FWIW, the performance on the 2x2 mirror seems to go up and down quite a bit while the performance on the 2 drive mirror is more consistent / less jumpy. I've also noticed with the zpool iostat output that each drive in the 2x2 mirror is doing 70% more read ops per second than each drive in the 2 drive mirror.

Any idea why the 2x2 mirror is so slow?

(System: FreeNAS 11.0-U3 on Supermicro X9SRE-F with E5-1650v2, 128GB RAM, mirrored USB flash for boot disk, 3 pools, each pool has 1-2 SLOG SSDs)
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
I'm finding that the 2 drive mirror is at least 25% faster than the 2x2 mirror when the reads are *not* coming from the ARC.
Yup, I can see that if I understand how mirrors work and striping works. You have one set of mirrors which reads fast, by rule of thumb it is ~ twice as fast as a single drive. Now you have a striped set of mirrors in the second pool and reading is limited to the speed of the stripe now. I suspect that if you put just two sets of mirrors against each other that maybe your 10TB drives would win out however without some benchmark testing to gauge random access speeds of each drive then you are just making a guess.

Of course I could be wrong about the mirrors, I don't run them myself.
 

xyzzy

Explorer
Joined
Jan 26, 2016
Messages
76
Yup, I can see that if I understand how mirrors work and striping works. You have one set of mirrors which reads fast, by rule of thumb it is ~ twice as fast as a single drive. Now you have a striped set of mirrors in the second pool and reading is limited to the speed of the stripe now. I suspect that if you put just two sets of mirrors against each other that maybe your 10TB drives would win out however without some benchmark testing to gauge random access speeds of each drive then you are just making a guess.

Of course I could be wrong about the mirrors, I don't run them myself.
Well, from what I understand (and have experienced using HW RAID controllers):
1) A mirror *can* read (in theory) 2X faster than a single disk because the controller can read from both drives in parallel.
2) A stripe of 2 mirrors should (in theory) read even faster assuming the data was distributed fairly well across both sets of mirrors.

So, what I'm seeing is *not* what I would expect....unless I'm misunderstanding something, perhaps something specific to ZFS or FreeNAS.

FWIW, the drives were all individually tested and benchmarked before being placed in the FreeNAS box and the 10TB drives are generally faster than the 4TB drives in random and sequential benchmarks.
 

c32767a

Patron
Joined
Dec 13, 2012
Messages
371
Well, from what I understand (and have experienced using HW RAID controllers):
1) A mirror *can* read (in theory) 2X faster than a single disk because the controller can read from both drives in parallel.
2) A stripe of 2 mirrors should (in theory) read even faster assuming the data was distributed fairly well across both sets of mirrors.

So, what I'm seeing is *not* what I would expect....unless I'm misunderstanding something, perhaps something specific to ZFS or FreeNAS.

FWIW, the drives were all individually tested and benchmarked before being placed in the FreeNAS box and the 10TB drives are generally faster than the 4TB drives in random and sequential benchmarks.

Do you have atime enabled on the volumes? that might cause some mixed reads and writes, which might disrupt the streaming nature of your reads on the stripe of mirrors.

Have you benchmarked the volumes locally on the NAS to rule out network/ISCSI issues?

You might also look at what the ZFS prefetch is up to and how busy your drives are. The new 10G drives might have deeper queues and/or be able to empty them faster. Something small like a queue depth difference might be enough to skew the results. There's lots of stuff out there written about ZFS tuning, particularly the SPA and DMU. http://cuddletech.com/?p=204 etc.
 

xyzzy

Explorer
Joined
Jan 26, 2016
Messages
76
Do you have atime enabled on the volumes? that might cause some mixed reads and writes, which might disrupt the streaming nature of your reads on the stripe of mirrors.
Good question. Yes - I always disable atime.

Have you benchmarked the volumes locally on the NAS to rule out network/ISCSI issues?
Not the specific drives I ended up using. I was assuming these 7200 RPM spinners would work like my older ones (which I had benchmarked locally in the past) and I'm beginning to think that was a bad assumption.

I did do some early end-to-end tests using a Windows 8.1 test VM that was using thin provisioned drives on the 2x2 mirror volume and they performed reasonably well so I didn't dig deeper. However, the volumes got re-created a few times since then and the final setup uses a different OS (Windows 2016) and much larger thin provisioned drives, so I've put a lot of variables in play that I need to examine individually. Sigh. :(

You might also look at what the ZFS prefetch is up to and how busy your drives are. The new 10G drives might have deeper queues and/or be able to empty them faster. Something small like a queue depth difference might be enough to skew the results. There's lots of stuff out there written about ZFS tuning, particularly the SPA and DMU. http://cuddletech.com/?p=204 etc.
Thanks for pointing this out....I was wondering if ZFS was doing any sort of read-ahead. This could be really helpful although that article went out of its way to say that these settings shouldn't be tweaked in most cases.

I think I'm going to end up dropping the zvol and trying to do some local benchmarks on the 4 drives individually, then try them in 2 drive mirrors and compare, then 2x2 and compare the results.

I guess I was hoping there was something in my results that someone might have seen in the past that would point in a specific direction.
 

xyzzy

Explorer
Joined
Jan 26, 2016
Messages
76
Also, any suggestions on the best way to benchmark the true storage performance? I haven't found a way to temporarily disable the ARC so I typically set the vfs.zfs.arc_max "Loader" tunable to 1 GB and reboot.
 
Status
Not open for further replies.
Top