Slow mirrored read performance, but not a network or single disk issue

[garm]

Okey, just to get the terminology agreed opon; a pool consists of vdevs striped together. Each vdev is either a single disk, N mirror pair or raidz with N parity.

In your first post you had 400+ MBps read with a single 4 wide mirror vdev. You then said you had slightly less with 2x 2 wide mirror vdevs. In your last post you said you had 600+ MBps with a striped pool, eg 4 single disk vdevs? I would expect your throughput to increase with the number of vdevs as I stated earlier. And this seams to prove it. However I wonder what throughput you get with two single drive vdevs and four mirror vdevs as well. I would expect the numbers to be consistent with the number of vdevs. Of course the relationship is not 1:1 due to metadata and complexity of the tree, but correlated.

 

[bubba173]

I guess the following is what you are alluding to?

"But from the point of view of a single application that issues a single write or read IO, then waits until it is complete, the time it will take will be the same for a mirrored vdev as for a single disk: Eventually it has to travel down one disk's path, there's no shortcut for that."

Well, as we have seen from the actual performance of read throughput then FreeNas obviously implements the mirror reads differently. As you could see in the screenshots above all 4 disks were read from at the same time through a single read request. Also the throughput was a multiple that of a single disk. And hence the time it took was also less than half. I am a very strong believer in numbers and statistics and numbers to me do not lie. I do not exactly know what FreeNas defines as a mirror and how it specifically implements operations on a mirror, especially on >2 way mirrors.

I have decided that I cannot spend more time profiling and I will most likely settle on the following solution:

As my primary operating environment is a local network with Windows machines, I decided to use the 4 disks in Windows Storage Spaces on my Windows Server and to set up a virtual disk that stripes across 4 columns. I also set up another 2 way mirror with 2 additional disks for resiliency purposes. I then use the application "GoodSync" to run a one-way backup in real-time between the striped virtual disk and the mirror virtual disk. I also use GoodSync to run a one-way backup job to keep a backup on the cloud in sync with the content of the mirror.

That gives me the following advantages:
* First and foremost, I keep things simple, things are running directly on my Windows Server, I do not need a virtualized environment anymore
* Then I get the foll advantage in terms of read performance from the striped volume (I can read at 640-650 megabytes per second from 4 mechanical disks)
* The mirror provides resiliency and is decoupled from the stripled volume , making future upgrades way easier than having it all in one
* And finally, not only do I get resiliency but also have I backed up my data in the cloud
* Last but not least, have I forgotten to mention that GoodSync offers full file versioning as well.

I am impressed though with FreeNas among all the other Linux based solutions I have tried (Nas4Free, OMV, Rockstor). Unfortunately this is not a hobby project but something that simply has to work and is supposed to protect some very important data, including work related data. I cannot spend more time on this side project.

Thanks all for your efforts to chime in and try to help with suggestions.

I think it might be useful for you to read this article:
https://constantin.glez.de/2010/06/04/a-closer-look-zfs-vdevs-and-performance/

Particularly the section titled:
Mirrored Performance Considerations

Basically it outlines why you are seeing more than 1 disk worth of performance, but less than 4.

 

[bubba173]

correct re my tests I ran. And I probably agree with your conclusion. As described I have decided to implement a different solution as described in the above post. Thank you again for your help and ideas. In the end I was not satisfied with what I had and resorted to something more familiar and something I know is extremely portable and stable. (Storage Spaces is also the main product on all of Windows Server's product line).

Okey, just to get the terminology agreed opon; a pool consists of vdevs striped together. Each vdev is either a single disk, N mirror pair or raidz with N parity.

In your first post you had 400+ MBps read with a single 4 wide mirror vdev. You then said you had slightly less with 2x 2 wide mirror vdevs. In your last post you said you had 600+ MBps with a striped pool, eg 4 single disk vdevs? I would expect your throughput to increase with the number of vdevs as I stated earlier. And this seams to prove it. However I wonder what throughput you get with two single drive vdevs and four mirror vdevs as well. I would expect the numbers to be consistent with the number of vdevs. Of course the relationship is not 1:1 due to metadata and complexity of the tree, but correlated.

 

[garm]

Well I disagree with your conclusion, but that is no business of mine. Best of luck.

 

[bubba173]

What in particular do you disagree with? I do believe that exchange like this is very productive, helps both of us learn and it also offers others the chance to learn and to chime in.
As long as we can disagree on issues that respect the validated factual numbers that I have displayed then I am happy to engage in exchange. As you have seen there were a lot of insinuations and guesses thrown around by some users that were disproven by the actual performance metrics. Is it that you do not believe Storage Spaces is a solid solution or you think I could have had more performance improvements with a different setup with FreeNas? if then which ones would have given me resiliency and close to 4 times a single disk read throughput?

Well I disagree with your conclusion, but that is no business of mine. Best of luck.

 

[toadman]

Interesting thread. Glad you finally figured out a solution. Re: performance, it's interesting because there are a lot of elements to what the numbers are showing.

For example, I know your HDDs are spec'd at 170 MB/s sustained. But I've seen reviews where they only hit like 150 MB/s under windows server 2008. I'm not sure what the storage stack in FreeBSD might do in a low level test (one without the additional potential overhead of the freenas middleware running).

I also don't know much about hyper-V. For example, you said you passed through the drives but not the controller. I don't even know how that is possible. But assuming it is possible, does that imply the hyper-V stack is then involved with the I/O to the disks? How can it not be if freenas can't see the controller (and I assume is seeing a virtual controller)?

And I'm sure you did your tests on basically an "empty" pool, but I don't know if there is or is not something that might have introduced some gaps (in sectors) in the sequential data. That might slow it down a bit too.

Looks like the ~650 MB/s reads you are getting on your Windows rig matches the other reviews I've seen for the drives. Are you also running Windows Server under hyper-V with only the drives passed through? Or is your Windows Server a bare metal setup?