Lower RAM requirement for RAID1?

[John Nas]

Hi, I have been considering an unconventional arrangement of three mirrored vdevs, each containing a 6TB hard drive. Would such a configuration have lower ram requirements than some of the typical RaidZ, RaidZ2, RaidZ3 configurations, or are the memory requirements dictated only by drive size and the ZFS file system, and thus have less to do with the complexity of parity calculations?

 

[kdragon75]

The minimum is 8GB and anything more than that has nothing to do with the size of your disks. Its all about how much data you need to use on a regular basis. More RAM more cache. There are lots of other cases where you will want more ram but purely storage and nothing funny (dedupe, L2ARC, etc..) You should tell us EXACTLY what your hardware is and EXACTLY what you goals are.

EDIT: Ok so there is more to it than that but it depends on your application. Most of home NAS units don't need 32GB of RAM.

 

[gpsguy]

BTW, I wouldn't consider 3 mirrored vdev's to be unconventional. Whether one uses mirrors or some form of RAIDZ is up to the user and/or their use case.

 

[John Nas]

Thanks for the prompt reply!

Hardware is negotiable since I'm trying to decide if it's feasible to repurpose what I have or better to shop around. Initial plan for system is a recently inherited, rack mounted, intel i5-2500k, gigabyte GA-Z68MA-D2H-B3 mobo with two 4GB sticks of Kingston DDR3 HyperX Blu.

I have two 1TB WD Green RE_GP drives from an old nas4free home server build, and two new 6TB Ironwolf Pro drives I snapped up in a recent sale. Sadly there was a limit per customer, so I wasn't able to get the four I initially tried to buy.

My aim is to store the data for my new small business, which primarily is graphics and video work; so relatively few large files, rather than many smaller ones. Files will be mostly be being archived, so I don't expect them to be heavily accessed since copies used for editing will be on the local drive of the work station. Data integrity will be the priority since things like video footage is high value data that cannot be recreated if lost. File size and terrible internet service in my area pretty much negates any serious cloud solution.

My original plan had thus been 4x6TB in RaidZ2 (and perhaps a separate pool backed by Raid1 vdev with the WD drives, for lower priority data). However I've become increasingly focused on the risks I've read about regarding the disk stress caused by long rebuilds. In addition since Raid is not a backup, I've also had to try and budget for the disk capacities of potential secondary and even tertiary copies.

So I swapped over in my thinking to a 3 drive Raid1, providing the same redundancy, with one less drive than RaidZ2 and making rebuilds after a failure a simple cloning operation. I also get to avoid the perfromance impact of dual parity calculations in typical use. With this setup I'd have one out of my original four drive setup spare to be an actual backup that I could move to a secondary location.

I've been exploring this general thinking in a SuperUser question: https://superuser.com/questions/134...on-to-multi-terrabyte-setups-in-small-busines

 

[danb35]

I've become increasingly focused on the risks I've read about regarding the disk stress caused by long rebuilds.

Ah, so you've been reading FUD.

 

[kdragon75]

and perhaps a separate pool backed by Raid1 vdev with the WD drives, for lower priority data

NEVER use hardware RAID to back a pool. Ill be back with more in a bit.

 

[danb35]

NEVER use hardware RAID to back a pool.

...or never use hardware RAID (with FreeNAS) at all.

 

[kdragon75]

Ah, so you've been reading FUD.

I think he's referring to the strong recommendation for RAIDz2. I'm not sure what is FUD about this. Enlighten us!

 

[John Nas]

The original article that made me investigate the idea: zdnet.com/article/why-raid-6-stops-working-in-2019 is undoubtedly clickbait, nevertheless isn't some of the reasoning behind RaidZ2 exactly because there's a risk of drive failure during (because of) a RaidZ1 rebuild? That second parity doesn't just mitigate two drives failing simultaneously, but also the more likely scenario of a second drive failing during the week long rebuild caused by the first drive failing.

I also see, though fiercely contested, that growing drive capacity is putting the chances of encountering a URE at a higher risk, especially during a rebuild. In that regard I thought a raid scheme based on mirrors vs striping provides greater secuirty albeit at the cost of capacity.

You might have to excuse my newbie terminology, between vdevs, zpools and the like. All I meant is I would have a vdev comprising two drives, allocated in a mirror, not that I would create a single Raid1 device and assign that to the vdev.

 

[danb35]

I think he's referring to the strong recommendation for RAIDz2.

Since the context is that he was considering a RAIDZ arrangement, but now is looking at mirrors, I'd disagree with your interpretation. I've seen a number of articles and blog posts recommending mirrors over RAIDZn arrangements citing similar concerns: "when you resilver RAIDZn pools, you're putting so much wear and tear on all the disks in the pool!" Well, yeah, exactly as much wear and tear as you'd add by doing a scrub. That argument is FUD. There are other valid arguments favoring mirrors in certain situations, but that isn't one of them.

But the promotion of RAIDZ2 over RAIDZ1 often reaches FUD-like levels, too, when people say things like "resilvering a RAIDZ1 pool with > 1 TB disks will probably result in pool failure." No, no it won't (of course it can, but the probability is nowhere near 50%, and certainly not over 50%, so "probably" is false--and I've seen it stated even more strongly than "probably"). Yes, I'm (roughly) familiar with the math in question, but (1) that doesn't account for the fact that a single read error in ZFS will not result in catastrophic failure; and (2) a great many RAIDZ1 pools have, in fact, been successfully resilvered. Yes, RAIDZ2 is a better arrangement, and it's a rare case indeed where I'd think that RAIDZ1 would be a good recommendation--but the odds are still good that you'd be able to safely and successfully resilver a 3 x 2 TB RAIDZ1 pool.

Edit: I'll add that many of the pro-ECC arguments, especially the ones about the "scrub of death", also border on FUD.

 

[danb35]

Isn't some of the reasoning of RaidZ2 not because it mitigates two drives simultaneously failing, but because of the risk that one drive might fail, and then a second drive fails during the rebuild caused by the first drive failing?

Yes, and it's the same sort of logic you cited to favor mirrors--that resilvering puts so much stress on the other disks in the array that there's a significantly-increased likelihood of another disk failing. IMO, it's bunk. Why would only one more disk fail? If it's so stressful, why wouldn't they all fail?

The stress on the disks in a pool from the resilver is the same as the stress from a scrub. Scrubs are routine maintenance operations that are generally recommended every few weeks. If that amount of stress caused disks to fail, we'd see pools failing left and right.

I've also seen, albeit fiercly contested, that growing drive capacity is putting the chances of encountering a URE at a higher risk, especially during a rebuild.

This is the core of the "RAID5/RAIDZ1 is dead" argument. I think there are a couple of counters: (1) apparently, modern hard disks do much better than their specs require in this regard (otherwise you'd be pretty well guaranteed a URE on anything bigger than about 1 TB), and (2) ZFS stores at least two (and as many as six) copies of all metadata, even before accounting for redundancy. So even if a URE happened in the middle of an important piece of metadata, there's another clean copy of it somewhere else--and the system knows which is the clean copy because everything has a checksum.

 

[Arwen]

@John Nas
Note that ZFS has a feature to split off a mirror from a pool made of Mirrored vDevs.
So if you simply attach a backup disk(s) as a mirror, (2nd or 3rd Mirror), you wait
for the re-silver to complete. Then split the disk(s) off, and store safely.

Can't do that with RAID-Zx pools.

That said, for my archivial storage, I use a 4 disk RAID-Z2. Just don't want the risk.

 

[John Nas]

@danb35
Debunking the bunk on the internet, especially in fields I'm not versed in, is not easy. Dave the system admin with 10 years experience says one thing, Dan the network technician with 11 years cries 'rubbish!' The problem comes down to the fact that the best advice you can get is from those with actual experience, but then anecdotes (rightly) become evidence for a position more than any math or reasoning. So for every guru that's "never had a Raid5 fail a resilver", there's another who's "lost more than one pool that way."

Mirrors, most likely appealed to me because as a complete amateur, where I'm not sure what the real risks are they offer a simplified less 'magic-box' solution than the higher Raid schemes. Drives can fail but so long as one works the data is there, easily readable, migratable to another machine etc, and should it happen I can order a new drive and be back to the same level of security via a clone in what I assume would be hours over the days a RaidZ rebuild would take.

 

[John Nas]

That said, for my archivial storage, I use a 4 disk RAID-Z2. Just don't want the risk.

I thought a Raid-Z2 is no less risk than a mirror over the same amount of disks? In fact over four disks a mirror provides 3 drive redundancy while the Z2 provides only two. It's the capacity of that extra drive in the first configuration that you lose that led us to develop parity systems in the first place no?

 

[danb35]

via a clone in what I assume would be hours over the days a RaidZ rebuild would take.

Yes, resilvers of mirrors are faster than resilvers of RAIDZ pools. Mirrors are also easier to (safely) expand, at least until vdev expansion is released (hopefully next year). I'm not saying mirrors are a bad thing; there are use cases where they make lots of sense. But some of the arguments used in their favor are, to be charitable, questionable.

By the same token, resilvers of RAIDZ1 arrays do fail. So do resilvers of RAIDZ2 and RAIDZ3 arrays. Less frequently with increasing redundancy, but nothing is impervious to failure.

In fact over four disks a mirror provides 3 drive redundancy while the Z2 provides only two.

Yes, a four-way mirror provides more redundancy than a four-disk RAIDZ2 pool, but half the capacity. Four disks in striped mirrors, though (comparable to RAID 10), have worse redundancy than RAIDZ2. On its face, they both have two disks' worth, but in RAIDZ2, any two disks can fail. In striped mirrors, if the wrong two disks fails, you lose the pool.

 

[garm]

I’m not sure on the math here.. four 6 TB drives is less then the six drives it would take to run 3 mirror vdevs. Or did you plan to run a 3 way mirror?

 

[John Nas]

@danb35
Significantly faster resilvers or only a marginal improvement? And I assume the 'stress' people have been referring to is dictated by the amount of data read and written not the prolonged time over which it happens? So a parity rebuild is no worse than a clone in terms of drive wear? Else is it the case parity calculations require more reads per drive than a simple mirror?

Would the modest PC specs I provided lean your recommendation one way or another? I think the obvious weak point is that I only have the minimum required RAM bringing me back to the subject line of the thread. Other than that the processor would seem to be somewhat equivalent in compute power to the intel core i3-43330 from the mid-range section of the recommended hardware list (though without ECC memory support).

In fact speaking of error correction, you mentioned "many pro-ECC arguments.. border on FUD", while the hardware guide describes not using error correcting memory as "a bit counterproductive"and suggests other file systems with lower resource requirements exist if you're not going to embrace ECC ram. Is it the case that the complexities of ZFS (scrubbing, checksums etc) make ZFS more dependent on RAM and thus more sensitive to errors or just that the on-disk data integrity ZFS has over other filesystems is undermined if you're not going to take the same precautions in RAM? I assume from your earlier statement you still advocate for ZFS even in an non-ECC system?

 

[John Nas]

@garm
A three way mirror. In fact I started out musing if this would provide some of the facilities provided by ZFS on a non-ZFS system. For example, the ability to fix bit-rot via an odd-man-out check. I was exploring this idea on the SuperUser forum when the ability to have multi-disk mirrors in ZFS was raised and I read the following on ZFSBuild.com:

Mirrored Vdev’s (RAID1)
This is akin to RAID1. If you mirror a pair of Vdev’s (each Vdev is usually a single hard drive) it is just like RAID1, except you get the added bonus of automatic checksumming. This prevents silent data corruption that is usually undetectable by most hardware RAID cards. Another bonus of mirrored Vdev’s in ZFS is that you can use multiple mirrors. If we wanted to mirror all 20 drives on our ZFS system, we could. We would waste an inordinate amount of space, but we could sustain 19 drive failures with no loss of data.

 

[danb35]

I assume from your earlier statement you still advocate for ZFS even in an non-ECC system?

Whether I advocate for it or not (I do), if you use FreeNAS, you're going to use ZFS. ECC RAM should be used, IMO, in any server, under any OS, with any filesystem--if you care about your data, you should be using ECC RAM as much as possible. But there's nothing about ZFS that makes it more risky than any other filesystem with non-ECC RAM.

I think the obvious weak point is that I only have the minimum required RAM bringing me back to the subject line of the thread.

The flavor of RAID you're using has no significant effect on the RAM requirements. The pool capacity does, to a degree--the bigger your pool, the more cache you're going to want, and that means more RAM.

 

[garm]

@garm
A three way mirror. In fact I started out musing if this would provide some of the facilities provided by ZFS on a non-ZFS system. For example, the ability to fix bit-rot via an odd-man-out check. I was exploring this idea on the SuperUser forum when the ability to have multi-disk mirrors in ZFS was raised and I read the following on ZFSBuild.com:

Mirrored Vdev’s (RAID1)
This is akin to RAID1. If you mirror a pair of Vdev’s (each Vdev is usually a single hard drive) it is just like RAID1, except you get the added bonus of automatic checksumming. This prevents silent data corruption that is usually undetectable by most hardware RAID cards. Another bonus of mirrored Vdev’s in ZFS is that you can use multiple mirrors. If we wanted to mirror all 20 drives on our ZFS system, we could. We would waste an inordinate amount of space, but we could sustain 19 drive failures with no loss of data.

There is no way of mirroring vdevs.

Or to put it another way, a vdev can be built as N-way mirrors or RAIDZN, but vdevs are striped together as a single block of storage (aka pool)