The case for mirrors

[johnnychicago]

Hello

After having played for a bit, I am currently mapping out a 'proper' setup for a long term use Freenas box. Power consumption and noise are factors for me, so I am leaning towards the minimal number of drives possible to fill my storage needs. Maximum 4, but less is more.

Currently I could just go and have a pair off 8TB drives ,mirror them into a vdev and be happy. It would be more expensive than 3 4TB drives for the same space, but I see following advantages:

* Resilvering a replaced drive should go considerably faster on a mirror than on a raidz
* If I were to grow it, I would 'just' have to buy two new drives and replace one after the other - not three or four drives. Again resilvering goes faster
* I could in a pinch add two more drives, build a second mirror vdev and extend the pool quickly
* obviously, once I am at four drives, it will be hard to reduce to two again. I'll have to wait for 'really big' drives to be available and do trickery with reducing redundancy and building a temporary second pool to move over

I can see this approach not working well if one wants bigger double digit TB's in ones NAS. But for a smaller setup, I currently fail to see the advantage of going with more, but smaller drives (unless you have them, of course).

Am I missing something, or is the above reasonable?

 

[Dice]

long term use Freenas box. Power consumption and noise are factors for me, so I am leaning towards the minimal number of drives

This is definitely reasonable. Raidz2 does not seem to be for you, then mirrors it is.

One thing to consider for long term is RAM capability. Depending on how much you want to do with your system, and if you'd like to have it ready for massive expansion beyond any initial planning.
Just putting it out there.

 

[johnnychicago]

Raidz2 on a four drive setup really seems to have only disadvantages versus two pairs of mirrors, doesn't it? I can see it on a bigger number of drives, but the complexity starts getting daunting then. :)

Thanks for the headsup on RAM - I'm starting with 8GB, but there's room on the board. Since storage is what I am after I feel rather relaxed there, especially since 16 or 32 are feasible.

 

[Dice]

What board are you using?
Starting at 8GB is ...ehrm... ..outside my realm of 'reasonable'.

 

[danb35]

Raidz2 on a four drive setup really seems to have only disadvantages versus two pairs of mirrors, doesn't it?

It has one big advantage, and that's redundancy. In a four-disk RAIDZ2 vdev, any two disks can fail without loss of data. In two pairs of striped mirrors, if the wrong two disks fail, you lose your pool. I'm not sure how comfortable I'd be with 8 TB disks in a single-parity arrangement, for the same reason that many people are saying "RAID 5/RAIDZ1 is dead"--the chance of a data error on the remaining disk(s) is getting pretty high.

 

[scwst]

I'm not sure how comfortable I'd be with 8 TB disks in a single-parity arrangement, for the same reason that many people are saying "RAID 5/RAIDZ1 is dead"--the chance of a data error on the remaining disk(s) is getting pretty high.

For those like me who are new here, this seems to be the background on the "RAID 5 is dead" discussion: http://www.zdnet.com/article/has-raid5-stopped-working/ . Basically, AFAIK, it boils down to the observation that the reliability of the drives has not increased with their size. The tl;dr of the article is (emphasis added):

... consumer RAID5 arrays can't be trusted to store your data reliably. Quickly, yes. In large chunks, yes. More simply than individual USB drives, yes. But not more reliably than a single drive.​

Do I understand this correctly that using a single 8 TB drives gives you the failure rate of two 4 TB drives combined? Or is that too simplistic?

 

[johnnychicago]

@Dice, the board is one of the Supermicro Mini ITX boards. I currently have 8GB in my test box, and it seems to handle just fine, saturating its link.

This is after all purely serving files. No jails, no transcoding, nothing else. Moving bits over NFS to two kodi boxes, rsync'ing backups from a number of sources and keeping somewhere around 500 snapshots right now. Should I realize I won't get along with 8GB, it's straight forward to double or even quadruple it.

I can see the point about losing any two drives in a Raidz2 setup vs two mirrors. But I am not buying the assumption that just because one drive is dead without a warning, the other suddenly won't be able to keep up for another two days or so and then spit back its contents in order to resilver a replacement drive.

And if it does, I'll have to go and grab the off site backup. I will remember you guys then and set up a three drive mirror next time :)

 

[danb35]

But I am not buying the assumption that just because one drive is dead without a warning, the other suddenly won't be able to keep up for another two days or so and then spit back its contents in order to resilver a replacement drive.

You asserted that a four-disk RAIDZ2 had "only disadvantages" compared to two sets of striped mirrors, and I explained how that's not the case. Your assessment of the risk is of course up to you, and the odds are pretty good you'd be fine with the mirrors, but that doesn't change the fact that you'd be safer with the RAIDZ2 array. Disks do occasionally fail suddenly and without warning, and multiple near-simultaneous disk failures are not unheard of. And, for that matter, FreeNAS's email alerting being broken isn't unheard of either.

 

[danb35]

Do I understand this correctly that using a single 8 TB drives gives you the failure rate of two 4 TB drives combined?

That probably is a little too simplistic. On average, the chance of an unrecoverable read error on a single 8 TB disk is going to be about the same as the chance on two 4 TB disks combined, but then you have the independent chance of overall mechanical failure of the disk.

 

[nojohnny101]

I can see the point about losing any two drives in a Raidz2 setup vs two mirrors. But I am not buying the assumption that just because one drive is dead without a warning, the other suddenly won't be able to keep up for another two days or so and then spit back its contents in order to resilver a replacement drive.

That is where you are not completely understanding. The re-silvering process is exactly where the drive would fail, especially since you are using such large drives (8TB). Re-silvering is faster on mirrors than a raidzx setup so you've got an advantage there, but the fact still remains that the second drive to die in an array often does during a re-silver.

 

[Mlovelace]

the chance of a data error on the remaining disk(s) is getting pretty high.

That is mathematically incorrect, your overall probability is no greater than the individual probabilities. Each individual event has no effect on the other events. If you want to determine the probability of a combined event, you have to calculate the chances of having two drives fail with URE at the same time. This is done by multiplying the events together. So 1/(1×10^14) times 1/(1×10^14) equals 1/(1×10^28) probability of simultaneous failure, that is a URE of 1×10^-28. However, at the end of the day no amount of redundancy replaces having a backup.

And as for drives, I always opt for the most amount of spindles to reach the desired storage capacity, generally speaking it always gives the best performance.

 

[depasseg]

Each individual event has no effect on the other events.

I think he was factoring in the additional stresses a resilver places on a drive which does change the probability, since the variables that make up the probabiltiy (heat, head movement, extended use) are different than in a steady state example.

 

[Stux]

That is mathematically incorrect, your overall probability is no greater than the individual probabilities. Each individual event has no effect on the other events. If you want to determine the probability of a combined event, you have to calculate the chances of having two drives fail with URE at the same time. This is done by multiplying the events together. So 1/(1×10^14) times 1/(1×10^14) equals 1/(1×10^28) probability of simultaneous failure, that is a URE of 1×10^-28. However, at the end of the day no amount of redundancy replaces having a backup.

And as for drives, I always opt for the most amount of spindles to reach the desired storage capacity, generally speaking it always gives the best performance.

Also, with double redundancy, assuming one failed drive, you would need a URE to occur not only twice, but to be collocated in the same block on the array.

 

[johnnychicago]

No, I still don't buy it. I can understand that two drives affected by the same event (e.g. heat, vibration, case toppled over) would be somewhat linked in their failures. And I think it makes total sense to buy two drives from different suppliers with a few days in between, to hedge against manufacturing and storage/transport boo-boos.

But, that apart, reliability is independent. And if a drive doesn't hold up to a resilvering, it would've died at the last scrub, wouldn't it? The load seems similar in both cases.

I am not saying a drive may not die during a resilver. I just think it's not more or less probable than it dieing anyway, and it's not related to another drive death.

Is there data around for these events? I can look back at my experience, but that's obviously anecdotal.

 

[Ericloewe]

But, that apart, reliability is independent. And if a drive doesn't hold up to a resilvering, it would've died at the last scrub, wouldn't it? The load seems similar in both cases.

I am not saying a drive may not die during a resilver. I just think it's not more or less probable than it dieing anyway, and it's not related to another drive death.

In principle, yes. In practice, the following factors come into play:

• The resilvering drive will be writing, causing a worse thermal and vibration environment
• Scrubs are generally scheduled for quiet hours. A resilver might not have that luxury.

 

[johnnychicago]

@Ericloewe, now I am confused (and I like to be, since it often leads to learning)

When I am resilvering a mirror, the existing drive is only read from, is it not? It was my understanding that in a mirror environment both drives are bit-by-bit copies of each other. The new drive would get the beating of the writes, while the old one is being read of.

I can see the point about resilvering during busy hours, although not sure how much of a difference it would make to the survival rates. Currently wondering if the load of the resilver could be dampened a bit. If I remember my playing with mdadm on Linux, one could set a maximum bit rate for RAID rebuilds. Something similar available here?

I do understand the theoretical reasoning for arguing against big RAID5/Raidz1 - the often quoted URE rate of 10^14 and the fact how you're statistically running into an URE on every rebuild. It' just that that doesn't seem to be working out for me. If that URE is relevant, the pure size of an 8TB device would make it an ROO (read only once) proposition, since it contains 6.4*10^13 bits. It would fail a read once every 12 TB.

Assuming a mirror of those filled to 50% (I think scrubbing only reads used blocks) I should expect each individual drive to fail one scrub out of three. Is that what people see? Converted to 4TB drives, one in six? If I had a 6 drive setup with 4TB drives, statistically every scrub should fail?

 

[Ericloewe]

When I am resilvering a mirror, the existing drive is only read from, is it not? It was my understanding that in a mirror environment both drives are bit-by-bit copies of each other. The new drive would get the beating of the writes, while the old one is being read of.

Yes (except for a little bit of metadata about the pool status and layout), but the writing drive worsens the environment (slightly) for the other drives.
I wouldn't say bit-by-bit, though, but it's "close enough" for the current discussion.

I do understand the theoretical reasoning for arguing against big RAID5/Raidz1 - the often quoted URE rate of 10^14 and the fact how you're statistically running into an URE on every rebuild. It' just that that doesn't seem to be working out for me. If that URE is relevant, the pure size of an 8TB device would make it an ROO (read only once) proposition, since it contains 6.4*10^13 bits. It would fail a read once every 12 TB.

Assuming a mirror of those filled to 50% (I think scrubbing only reads used blocks) I should expect each individual drive to fail one scrub out of three. Is that what people see? Converted to 4TB drives, one in six? If I had a 6 drive setup with 4TB drives, statistically every scrub should fail?

No, those specs are pessimistic in most environments. They're also vague.

 

[johnnychicago]

@Ericloewe, thanks for the explanation. I think I'll consider the added risk to the reading drive by a neighbouring writing drive more as an inconvenience. ;)

The URE bit got me thinking - I've been running my test/play machine here with a pair of well broken in 3TB drives that got scrubbed twice a month since March. They were scraping along the 80% usage mark, so they've clearly given back way more than the URE assumption of bits read during their scrubs alone - without a hitch as far as I can see. This lets me assume that a pair of 8TB ones should not keep me from sleeping at night.

 

[fta]

Keep in mind if you have a failing, but not failed drive, you can add the replacement drive to the mirror before removing the failing drive. That way if a URE were to occur on the non-failing drive, the system will probably be able to get the block from the failing drive. That along with a spare and frequent scrubs give me plenty of peace of mind with mirros. In the very unlikely event of losing two drives in a mirror, I've got backups.

 

[Arwen]

Keep in mind if you have a failing, but not failed drive, you can add the replacement drive to the mirror before removing the failing drive. That way if a URE were to occur on the non-failing drive, the system will probably be able to get the block from the failing drive. That along with a spare and frequent scrubs give me plenty of peace of mind with mirros. In the very unlikely event of losing two drives in a mirror, I've got backups.

This is one very good reason to use ZFS, options. The "zpool replace" option also works for RAID-Zx
and Stripes! But, only works if you have a spare disk slot, (or can make one).