1x VDEV, RAID-Z3 vs 2x VDEV, RAID-Z2 (+ other questions)

[dzog]

Hey FreeNAS folks!

I've had storage arrays for many years for personal use, generally always md linux software RAIDs.

My old RAID-6 (8x 1TB drives) is getting long in the tooth, so I'm planning to build a new array (16TB+ usable space) and am looking towards FreeNAS as it seems like the modern thing.

For hardware, I'm thinking 32 GB of ECC RAM, either a dual-core (i3-6100) or a quad-core (some Xeon) thing, and a nice PSU. I have a large server case that can hold lots of drives. Not sure yet on motherboard - I was thinking just a Gigabyte thing (I've always used those in the past), but now I'm looking at SuperMicro based on recommendations here.

This is a single-user array, mostly for backups / bulk media storage / art asset storage (mostly large files!). I likely want to do some plex transcoding and perhaps try to set up crashplan but that would be it for fancy stuff.

1. It seems that "11 drives per vdev" is a recommended max. Is that still accurate in 2017?

What would be better?

One VDEV, RAID-Z3, 11x 2TB disks
-or-
Two VDEVs, RAID-Z2, 6x 2TB disks each
-or-
One VDEV, RAID-Z3, 12x 2TB disks (this was my original plan)

2. In the past I've seen recommendations for using different model drives in arrays, reason being: drives from the same line are more likely to fail at the same time. Thoughts on this? I have a bunch of WD Caviar Black 2TBs and a bunch of Toshiba DT01ACA200 2TB drives (these have CCTL - Hitachi's TLER - enabled) and I'm thinking of mixing and matching.

3. Perhaps outside the scope of this subforum, but would the dual-core Skylake be sufficient for my usage (one user, but a fair number of disks)? Or do I really need to spring for a quad core here?

4. I've had really good luck in the past with ext3/4 on md RAID-6 (haven't lost data in 12 years of using RAID) and am experienced managing those systems. Is jumping to something like FreeNAS even worthwhile? The lack of recovery tools makes me nervous - though I have external backups of my most critical and irreplaceable stuff, an array loss would still be a big headache and timesink as I can't afford to truly back up everything. I realize this is a tradeoff I am making, but perhaps md+ext4 is better suited for me in this case?

Thanks!

 

[MatthewSteinhoff]

1. RAID-Z3 is overkill. Two Z2 vdevs will be more responsive (twice the IOPS) than a single large vdev.

(As an aside, 2TB drives are really tiny by today's standards. If you already have them, fine. Otherwise, the pricing sweet spot seems to be 4TB if not 6TB. This becomes even more true when you factor in slot cost, port cost, power cost, etc.)

2. Meh. I mix and match whatever is handy and cheap. I can do this because our systems are well-monitored and replicate regularly. If you want to set-in-and-forget-it, spend the extra $10-15 a drive and get a known-good brand designed for always-on NAS use (WD RED and similar).

3. Single user file serving? Pretty much any processor will work just fine. You mentioned Plex transcoding: it requires 2,000 worth of PassMark for every HD video stream. Toss in a few cycles for Crashplan. Still, not a lot of power required.

4. You may not have lost data but are you sure your data hasn't rotted, suffered silent degradation? How would you know with ext3/4? We use FreeNAS because we want to prevent not just catastrophic data loss but also bit-level loss.

FreeNAS is pretty much hands-off once you get it up and running. I typically log into our primary FreeNAS server about once a week. I can't remember the last time I logged into our FreeNAS replication target, probably months ago. If it has a problem it'll email me.

Cheers,
Matt

 

[StephenFry]

1. RAID-Z3 is overkill. Two Z2 vdevs will be more responsive (twice the IOPS) than a single large vdev.

I disagree with the first statement; redundancy is why we RAID, so is there such a thing as overkill...?
Also, OP says it's a single-user system, for mostly static storage. The IOPS aren't that relevant.

Having said that, 11-drive Z3 pools are what *I* have found to be a FreeNAS sweet-spot, just like 6-drive Z2 pools, but YMMV.

 

[Ericloewe]

so is there such a thing as overkill...?

Yeah, otherwise 24-way mirrors would be popular.

 

[rs225]

I disagree with the first statement; redundancy is why we RAID, so is there such a thing as overkill...?
Also, OP says it's a single-user system, for mostly static storage. The IOPS aren't that relevant.

I look at it much differently.
Raidz1 is for redundancy.
Raidz2/3 are for people who think they don't need to backup.

Having said that, 11-drive Z3 pools are what *I* have found to be a FreeNAS sweet-spot, just like 6-drive Z2 pools, but YMMV.

Agreed.

I can't remember the last time I logged into our FreeNAS replication target, probably months ago. If it has a problem it'll email me.

And if it doesn't?

 

[Ericloewe]

And if it doesn't?

There's little reason to doubt that it will, after you validate that it's working. Just two days ago, I got an email warning me about some SMART values that had moved literally within minutes of the disk reaching this conclusion.

If you don't want to trust it, check it yourself at whatever interval you deem reasonable.

 

[rs225]

The problem with set it and forget it, is that you forget it. There should be an inspection period. If the email server decides to start rejecting emails from the FreeNAS box or junk them as spam, how will you ever know?

No news isn't always good news.

 

[Ericloewe]

There's the weekly "I'm still here! Nothing to report!" email and you can set up your own email tasks with whatever you want. The Resources section has a few cool scripts that email you the drives' status, among other things.

 

[Ericloewe]

https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[rs225]

I don't know... Seems to me this is how those stories start about servers lost in walls and closets that only get discovered when they finally fail.

 

[Ericloewe]

What good is a server if you need someone staring at a screen 24/7?

 

[farmerpling2]

A few above...

If you care about your data, make sure your NAS uses TLER drives. You are playing with fire cause "Murphy" is real and will get you...

I will get the write up why non-TLER drives should NOT be used in a NAS system.

I have seen systems loose their RAID set(s) when badblocks start popping up like crazy and the system becomes non responsive, so people reboot because something must be wrong after 2-3 hours...

Just say NO to non-TLER drives when using RAID... You will use them until you get burnt. Just like RAID-0 is playing with fire on magnetic media drives (not so much for SSD).

 

[Vito Reiter]

More VDevs will naturally increase your speeds and redundancy given that they'll be striped together and total disk failure tolerance would be 4 (2 in each VDev, keep in mind). Now as for the 12 drive RaidZ3, for some reason it is highly unrecommended to put more than 11 Drives in an array and people have strangely lost data this way. RaidZ3 is a little overkill at the time being but as drives grow and resilver times take longer it might be something to consider for the future (Especially if your data is MISSION CRITICAL).

Personally, I've had great luck with a 2x RaidZ2 arrays even though you'll lose a good amount of usable space you'll have more speed, redundancy, and quicker rebuild times. If you're looking for more space a <= 11 Drive RaidZ2 would even work, although resilver times would be quite long, you'd have 9 drives of usable space (Probably the only way you'll manage a solid 16TB of real usable storage).

Good luck,
Vito Reiter

 

[MatthewSteinhoff]

There's the weekly "I'm still here! Nothing to report!"

Exactly! It looks something like this...

Code:

-------- Message --------
Date: Sat, 08 Apr 2017 08:15:00 -0000
Subject: FreeNAS weekly run output
From: support@
To: freenas@

Security check:
	(output mailed separately)

-- End of weekly output --

Or the 'Update Available' email that comes every day because we are slow to pull the trigger on upgrades.

If everything is working fine, I get the weekly security report showing absolutely nothing. If replication fails, I get notified. If there are drive problems, I get notified. If nothing is working, my clients call me. FreeNAS 9.x has been rock-solid reliable and does a really good job of notifying me when things are questionable.

When we first brought FreeNAS online, I had the dashboard up a lot. Now it really is something we more or less ignore.

If the email server decides to start rejecting emails from the FreeNAS box or junk them as spam, how will you ever know?

The same email server that processes the FreeNAS email processes email for a dozen other internal hosts. If I get to work in the morning and I have zero email instead of 30 messages, I know something is wrong.

The key to being a good SysAdmin is knowing how things fail and what stuff looks like when it's working.

For example, I get a report on system backups every morning. It is so regular and consistent that I can look at the email size and time stamp to know if there are problems. Most days, I don't even if have to read the email to know if everything worked or if investigation is warranted. Bigger or smaller than 36k +/- 2k? Investigate. Arrives more than five minutes away from 4:27 am? Investigate.

Cheers,
Matt

 

[StephenFry]

I look at it much differently.
Raidz1 is for redundancy.
Raidz2/3 are for people who think they don't need to backup.

That's a truth bomb right there, people.

I'm guilty of this as well, although I suspect in much the same way many of you -yes you, I'm looking at you- use (some of) your personal FreeNAS systems.

If you have a "media" NAS, it's not such an issue to backup a few TB of family photos.
But once you start with large systems for purely, say, dvd/bluray content, material that is not at all mission critical, it would not have a big impact if it were to be a total loss.
(... the data is obviously all backups of the original media that you own... right?...)

HOWEVER, it is a lot of work to rebuild/rerip that library and on top of that, it's a hobby/sport/challenge to keep all these bits and bytes 'alive'. For certain data, I'm willing to gamble on Z2/Z3 keeping it going, maybe throw in a UPS, put it in a high place, a bunker, or whatever.

For all other stuff: backup backup backup

 

[rs225]

For all other stuff: backup backup backup

Yes! Create a special dataset, everything important goes in, and back it up regularly. You'll be surprised how fast it grows once you think this way, because it turns out it is easier to backup something important than to keep track of it for a potential lengthy re-do.

 

[dzog]

Thanks for all the info, everyone!

Avoiding silent bit rot is totally a great reason to use zfs - I think I'm sold on it. It also makes rebuilds less dicy - it doesn't do free space and a single URE isn't going to kill it, if I understand things correctly, and that's a big deal.

Another bonus of having 2x VDEVs is upgradability - I can piecemeal upgrade individual VDEVs when drives get cheaper instead of having to do it all at once (makes it more likely for me to even bother upgrading vs doing a new build). I could also add additional (smaller) VDEVs with external drive chassises.

In terms of comparing redundancy between various models, this seems complicated to figure out since there's so many variables (disk size, vdev width, raid-z layer) and types of probability involved. How does 11-drive RAID-Z3 compare to 2x 6-drive RAID-Z2 in terms of chance of data loss?

Re: disk size, I already had a bunch of 2 TB disks from a previous array (was a hardware RAID for a dual-booting hackintosh/Windows machine, now old and defunct) and bought a bunch more on clearance. I could sell them and buy 4 TB but I'm not sure the math works out at this point.

Thinking towards one of these configurations:

- Two RAID-Z2 of 6x 2TB drives (16TB)
- Two RAID-Z3 of 8x 2TB drives (20TB, gives me closer to 16TB usable, high % parity)

I know 8x isn't a recommended size for -Z3, but my guess is that it will still be totally fine for maxing 1 Gbps Ethernet? How much does this actually make a difference (I may run 10 Gbps sometime soon)? I can run some tests when my hardware arrives.

My case can fit 17 drives into hot-swap bays (+ two screwed-in), so 16x drives + one hot spare comfortably maxes out my case. It does mean buying 4x more 2 TB drives though.

 

[dzog]

As far as usage and "RAID instead of backup, even though RAID is not a backup", my data falls into the following categories:

- irreplaceable things (original content, archives, rarities, etc) that I generally want high-availability of but are also backed up

- very replaceable things (live computer backups). I store backups of my local machines on RAID, in addition to offsite (crashplan). There's no need to back this stuff up a third time.

- replaceable things (media backups, mostly) that would cost a lot of time to replace. This is a lot of my stuff. I don't back these up. I've managed not to lose any of it so far in 12+ years. This is the riskiest stuff.

"Always back up" is definitely the safest answer, but unlimited $$$ isn't a thing. There are many ways to lose stuff (user error, fire, water, thievery, etc) but I've found disk failure to be by far the most common for me, so I try to protect against that with RAID. I may try to do some offsite backups of my "replaceable" data this year, but at the upload speeds I get, this may take a very long time.

 

[Scharbag]

Z1 with large drives is a gamble if you need highly available data.

2 @ Z2 vDevs will be faster than 1 Z3 vDev. BUT, you have to look at your data efficiency and usage requirements. 12 drives at Z3 will offer more space than 12 drives with 2 6 drive Z2 vDevs. And there is NO reason to not use 12 drives in a Z3. ZFS "can" do far more in a vDev (at some point, things will go south, but ZFS allows for really large vDevs). I run a 12 drive Z2 vDev for backups and it is just fine. Scrubs at close to 1GBps and has been reliable since its inception.

Bottom line, RAID is NEVER EVER EVER a replacement for backups.

RAID allows for data AVAILABILITY, nothing more (this has ALWAYS been the case). Backups ensure that WHEN things fail, you do not loose your data. Anyone who uses RAID as a replacement to a good backup strategy for important data is asking for trouble (there are cases where you don't care, and that is fine, but not typical). Anyone who uses Z1 with todays large drives (2TB is still fairly large) is asking for their data to be unavailable until restored from backup (or lost if backups are not available).

This is my typical I/O demand for my production array in home use, running 5 VMs and media players (2 users) off of this volume:

Does not even tax the system.

Cheers,

 

[StephenFry]

How does 11-drive RAID-Z3 compare to 2x 6-drive RAID-Z2 in terms of chance of data loss?

I'm a big fan of 6-drive Z2 and of 11-drive Z3 systems. Given a choice, I'll always pick an 11 or 12 drive wide z3 over a 'double-wide' z2. The latter will give you a bit better IOPS performance, but ...meh.

You can use this calculator to see some numbers:
https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

servethehome has another one.

Maybe my interpretation of the math is off, but I've always concluded that a 12 drive z3 is in a different league than a 2*6 drive z2 system when it comes to MTTDL.

Having said that, 12 drives in one vdev is considered too wide by some. I don't subscribe to that view, but as with anything I say, it's based on my preferences, experiences, etc, so YMMV.

And don't worry too much about 'optimal' drive setups, but do your research. Look a bit into the performance and overhead of 11 and 12 drive systems and see what works best for you.