SOLVED Help understanding Mirrored vdevs in relation to pools

[jblack]

I have six 1TB SSDs that I'd like to use for remote VMs, NFS shares, and other such fast, random heavy operations. As I understand it, these applications are grouped together into "block" storage, as opposed to "archival" storage (like storing pictures, music, etc). Reading through these articles (links at bottom, the 'insert link' button doesn't seem to be working for me right now) it seems that for such block storage, mirrored vdevs are the way to go. What I'm still confused on, however, is whether I could put multiple mirrored vdevs in one pool or if I'd have to have each in their own pool. With my six disks, I could do three two-way mirrored vdevs in one pool for a total size of 3tb, but as I understand it, that would be striping the three mirrors together, which is exactly what is being avoided by not using RAIDZ. I could of course have three individual pools with one two-way mirrored vdev in each pool, but then managing the storage becomes a pain, as I'd have to manually balance which share/VM goes on which pool.
Do I need to have three pools of one two-way mirrored vdev? Could I get away with one pool of three two-way mirrored vdevs? Am I over-thinking this whole thing and it doesn't matter because I'm not using spinning hard drives? I'm basically looking for advice on how to best set up these six drives for this use-case.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

 

[Arwen]

Yes, all 6 x 1TB SSD can be in the same pool.

Basically, a ZFS pool consists of 1 or more vDevs, of various types, with the minimum being 1 data vDev.

So, in your case, you could have 3 vDevs consisting of 2 x 1TB SSD.

Not sure why you think striping is bad;

I could do three two-way mirrored vdevs in one pool for a total size of 3tb, but as I understand it, that would be striping the three mirrors together, which is exactly what is being avoided by not using RAIDZ.

Striping across vDevs gains IOPS.


Now you should be clear that some SSDs are not meant for high writes and will wear out faster. Do some research on yours.


Last, VM storage has different desired behavior than NFS shares. Make sure you read that path to success for block storage.

 

[jblack]

Not sure why you think striping is bad;

Striping across vDevs gains IOPS.

From the "Some differences between..." article:

Further, from an IOPS perspective, a RAIDZ vdev tends to exhibit the IOPS behaviour of a single component disk (and the slowest one, at that).

So we see a lot of people coming into the forums trying to store VM data on their 12 disk wide RAIDZ2 and wonder why their 12 disk 30 TB array sucks for performance. It's exhibiting the speed of a single disk.

The solution to this is mirrors. Mirrors aren't as good at making good use of the raw disk space (because you only end up with 1/2 or 1/3 the space), but in return for the greater resource commitment, you get much better performance.

I had assumed that the IOPS being limited to the slowest disk was a result of striping in RAIDZ, but maybe I'm understanding that wrong? Would a striped set of mirrored vdevs be limited to the IOPS of the slowest vdev?
As a side question, for the redundancy of a pool of three two-way mirrored vdevs, would that mean that I could lose one disk for certain without losing data, and up to three disks if I got lucky and the right disks failed?

Now you should be clear that some SSDs are not meant for high writes and will wear out faster. Do some research on yours.

That is a good point, mine are consumer drives, so perhaps I would drop it back to four active disks and two hotspares (or cold spares maybe).

Last, VM storage has different desired behavior than NFS shares. Make sure you read that path to success for block storage.

Another good point, a lot of my NFS shares would be for things like datasets within docker containers, or other things that would involve a lot of random activity, but VMs is my main concern here.

 

[Patrick M. Hausen]

A very blunt rule of thumb: each vdev gives you the IOPS of a single disk. So a RAIDZ2 is quite resilient to failure, very economic in terms of "wasted" space, but not suitable for transaction heavy workloads.

12 disks in 6 mirror vdevs give you the IOPS of 6 disks. At the expense of less resiliency - one disk is easy peasy, lose two disks and you might still be ok, but as soon as you lose an entire vdev, the pool is toast. So your assumption about the right three disks failing is correct.
In mission critical transaction heavy applications one might chose lots of 3-way mirrors. If resilience to failure and availability is more important than the cost of disks ... which is frequently the case in enterprise applications.

Fast, reliable, cheap - pick any two.

RAIDZ - cheapest, very reliable, not fast in terms of IOPS
2-way mirror - more expensive, less reliable, considerably more IOPS
3-way mirror - even more expensive, but more reliable than 2-way, same performance as 2-way

HTH,
Patrick

 

[jblack]

A very blunt rule of thumb: each vdev gives you the IOPS of a single disk. So a RAIDZ2 is quite resilient to failure, very economic in terms of "wasted" space, but not suitable for transaction heavy workloads.

Ok, I misunderstood the relationships between RAIDZ, IOPS, and striping then. Some more reading is in order I think.

12 disks in 6 mirror vdevs give you the IOPS of 6 disks. At the expense of less resiliency - one disk is easy peasy, lose two disks and you might still be ok, but as soon as you lose an entire vdev, the pool is toast. So your assumption about the right three disks failing is correct.
In mission critical transaction heavy applications one might chose lots of 3-way mirrors. If resilience to failure and availability is more important than the cost of disks ... which is frequently the case in enterprise applications.

Got it, I think I may drop back to a pool of two two-way mirrored vdevs with two spares in case one fails. I don't much like the idea of having to order a new drive and just hope a second disk doesn't fail until I get the new one, I'd rather have a smaller pool and spares on hand to swap in immediately.

 

[Patrick M. Hausen]

I always keep one extra drive on the shelf for all my productive installations. That's a cost I am willing to pay regardless of the pool topology.

If you are that concerned about availability, why not simply 2x 3-way mirrors? Having the disks in the NAS doing nothing seems like a waste to me. Plus you still have the redundancy after a single disk failure without even resilvering.

 

[jblack]

I always keep one extra drive on the shelf for all my productive installations. That's a cost I am willing to pay regardless of the pool topology.

If you are that concerned about availability, why not simply 2x 3-way mirrors? Having the disks in the NAS doing nothing seems like a waste to me. Plus you still have the redundancy after a single disk failure without even resilvering.

I would also like to have an extra drive sitting on the shelf as a cold spare, with a 2x 2-way mirrored pool, the other two drives would be that. If I used all six in the pool, I would just have to buy another one for a cold spare. With a 2x 2-way mirror and a cold spare the only risk would be if a second disk failed before I could replace the first one, which is a risk I'm willing to live with. Plus I think 2tb is enough for now, and I can always rebuild the pool later with more mirrored vdevs if I need more space. Mostly just with 2x 2-way I would need 5 drives, which I have, but with 2x 3-way I would need 7, which I do not have.

 

[Etorix]

Ok, I misunderstood the relationships between RAIDZ, IOPS, and striping then. Some more reading is in order I think.

Roughly, a vdev (any type) has the IOPS of a single drive; pools are stripes of vdevs, and the IOPS of a pool is the sum of the IOPS of its vdevs.

With a 2x 2-way mirror and a cold spare the only risk would be if a second disk failed before I could replace the first one, which is a risk I'm willing to live with.

You can also put one drive in the pool as hot spare and keep the other in a drawer as cold spare.

Another good point, a lot of my NFS shares would be for things like datasets within docker containers, or other things that would involve a lot of random activity, but VMs is my main concern here.

Block storage (zvol, iSCSI) demands IOPS, which means mirrors.
File storage (NFS, SMB) typically demands less IOPS than block storage, and thus does better on raidz# than block storage. That does NOT mean that file storage requires raidz#. File storage performs very well on mirrors, it's just not very space efficient.
If it fits, put everything on mirrors and be happy.

 

[Patrick M. Hausen]

2x 2-way mirror:

- a single drive fails
- either a hot spare kicks in or you replace from the shelf
- the pool resilvers
- while that is happening you have no redundancy in that vdev while putting additional stress on the remaining disk


2x 3-way mirror:

- a single drive fails
- you don't have a hot or shelf spare
- you are still in as good a situation as with the 2-way mirror pool and all drives intact!
- go get a new drive, take your time, all is still well


That's what I don't understand. Spinning drives are more likely to not come up after 5 years in storage than to randomly fail if they are just powered on and spinning and nice and evenly tempered ... these things are made to spin. That's why I also do not recommend spin down.

P.S. In the second setup (3-way) you could even detach the failed disk and the system would never know it was there. You would have turned the vdev into a perfectly happy 2-way mirror. Detach one disk from the second still 3-way vdev and add that as a hot spare. Now you have 2x 2-way with one hot spare. When the new disk arrives remove the hot spare, attach 1 disk to each of the mirrors - back to 2x 3-way.

In mirrored vdevs disk can come and go as you see fit as long as there is at least one left. And there is no extra burden or risk because the system thinks "oh now, I am missing one of three disks now!" 3-way with a failed disk is exactly like 2-way with both disks intact.

 

[jblack]

You can also put one drive in the pool as hot spare and keep the other in a drawer as cold spare.

Yeah that might make more sense than two cold spares.

Block storage (zvol, iSCSI) demands IOPS, which means mirrors.
File storage (NFS, SMB) typically demands less IOPS than block storage, and thus does better on raidz# than block storage.

I need to look into iSCSI, I think it may be more appropriate for some of the things I'm using NFS for at the moment.

That's what I don't understand. Spinning drives are more likely to not come up after 5 years in storage than to randomly fail if they are just powered on and spinning and nice and evenly tempered ... these things are made to spin. That's why I also do not recommend spin down.

The drives in this pool are all SSDs, my main storage pool is spinning rust and it never spins down. But I do still see your point, I'll just have to think about it a bit, but either way (2x 2-way + 2 cold or 2x 3-way + 0 cold) my main question was answered.

Thanks to everyone for your input.

 

[Patrick M. Hausen]

I added some thoughts to my last post. But I was not aware you are using SSDs. In that case of course shelf or hot spare means no writes, which makes perfect sense given that SSDs are write limited.

 

[jblack]

P.S. In the second setup (3-way) you could even detach the failed disk and the system would never know it was there. You would have turned the vdev into a perfectly happy 2-way mirror. Detach one disk from the second still 3-way vdev and add that as a hot spare. Now you have 2x 2-way with one hot spare. When the new disk arrives remove the hot spare, attach 1 disk to each of the mirrors - back to 2x 3-way.

In mirrored vdevs disk can come and go as you see fit as long as there is at least one left. And there is no extra burden or risk because the system thinks "oh now, I am missing one of three disks now!" 3-way with a failed disk is exactly like 2-way with both disks intact.

That hadn't occurred to me, but makes sense.