Should I RAID-Z1 or RAID-Z2?

[dougoftheabaci]

I'm going to be building myself an NAS in a month or two and I'm trying to decide whether I want RAID-Z1 or RAID-Z2. Now, if this was an enterprise-level application I'd just go with Z2 and be done with it but this is for my own, personal use.

The zpool is going to be used for file storage and local-web-development. It'll be rather low traffic. All the drives are going to be the same make and model and bought at the same time.

Now, when I say low traffic, it'll sit on my network and every once and a while I'll do some light dev work with it and maybe every night stream a movie. For the most part, that's how it's going to be used.

Do I really need to worry about a two-drive failure? Since we're talking about low-use where I can easily turn off my NAS if a drive dies and the only one to suffer is me, is this something I really need to worry about?

Now, space isn't something I'm going to be wanting for drastically. I'm going to be using five 2 TB drives on my NAS for my RAIDZ zpool. If I'm correct, under Z1 that leaves me with 8 TB of usable, redundant storage and under Z2 it's 6 TB. I'm only using 2 TB in my current system and it took me a little while to fill that. The extra 2 TB would be nice for the future but it's not my primary concern.

What do you guys think? What would you choose?

 

[matthewowen01]

you have 5 disk that can fail, the more disk you have the higher the chance that a second will fail while your working to replace the first that failed. i would highly recommend raidz2. it just gives you another disk of redundancy. after i lost both drives in a mirror within 4 hours, i stopped using single disk redundancy.

 

[jafin]

Comes down to how much risk.

As you'll probably guess there are many stories of RAID5/RAIDZ failing with one drive, and then when the rebuild/resilver occurs another drive goes, or in a case you haven't acquired a replacement (in my case Seagate can take 2 weeks to ship a replacement drive) drive theres a window of chance for additional failure. So RAIDZ2 gives you an additional level of protection by having another safety net of 2 drive failure. With such large capacity drives, there is a larger chance of more bits going bad quicker and the pain in restoring large amounts of data and the relative cheap price of 2tb storage I would strongly urge RAIDZ2.

But if you are happy to have just a certain level of risk, and are happy to rebuild/restore should the unfortunate happen, then go RAIDZ1

And as always, RAID/ZFS is not a backup strategy, even with RAID I urge you to consider offline backups or a mirrored machine offsite or cloud backups for true recovery.

Heres an intersting post about theoretical raid failures. http://blog.kj.stillabower.net/?p=93

 

[dougoftheabaci]

Oh, I'm well aware it's not a backup strategy in of itself. To some degree, anyway. It kind of depends on how you mean it and how you're using it. I'm also familiar with that article but the problem is a number of documents, including that one, look at things from a very enterprise-centric point of view. Which is great if that's the world you're living and operating in. The world of the consumer is drastically different.

That being said, I think RAID-Z2 is where I'll be going. The possibility of a failure during recovery is one I'd rather not have to consider. If going with 6 TB of usable storage instead of 8 TB makes it extremely unlikely that anything will destroy my data? Sounds good to me.

 

[joeschmuck]

snip snip.... All the drives are going to be the same make and model and bought at the same time. Snip Snip...

Do I really need to worry about a two-drive failure?

What do you guys think? What would you choose?

First I HIGHLY recommend you buy your hard drives from different batches. You are increasing the risk of catastrophic failure from all the drives at the same time. Think of it as if there was a defect in the build process where some circuit will fail prematurely, not saying that will happen but you never know. When one drive fails, the others are sure to follow given they should have the same characteristics having been build at the same time on the same assembly line. One recommendation I heard was to use different brands but I like Samsung and will always buy them. WD are okay and Hitachi are very good. I unfortunately found out this advice after I bought my four drives and yes, all manufactured the same month even though I purchased one 2 months after the first three but they were all from NewEgg.

Second question related directly to how you are using the data on the NAS, if you are backing up the data to another location, or if you can afford to lose it. I use my NAS for backup images of my computer so you would think it was very important to me to not have a drive failure however it doesn't bother me so long as the computers I build the backups from are still working because I can create another backup. The photos I have on the NAS are also located on my wives computer so they are in a second location as well. The program and other data that is not located on another device are either saved on a DVD-R or I don't care about them and could just download them from the internet if needed. And my last thing is for DVD Ripped movies, they reside on a single UDF drive at this time, I plan to move them to a single ZFS once FreeNAS 8.x is more complete and stable. If I lose the movies, no big deal because I can rip them again whenever I need to.

As for RAIDZ or RAIDZ2, that depends on how valuable your data is and if it needs to be online all the time. If you have a good backup strategy then RAIDZ is very good. Also keep in mind speed, RAIDZ2 generally is slower due to calculations required. Let me make a suggestion for you to think about... For items you have backed up like photos should be placed on DVD periodically, make that a single ZFS drive.

Last thought... Keep in mind that if you want to upgrade to a larger size drive ever (doubtful based on what you are expecting to purchase) that the pool will not expand until all drives are of the larger size. I would recommend a 3 drive RAIDZ and a ZFS Mirror pair. What this gets you is the option to shut down one of those drive set when you find out you have too much space.

 

[dougoftheabaci]

My data does not need to be online all the time. As I said, I'm the only one who's really going to be accessing it so if a drive dies and I need to leave the server for a day or two while it rebuilds? Fine.

As for how much I care about my data? I care but not enough to spend an extra $1,000 to build a second server. Not yet, anyway. It will come to that eventually, I'm sure, but not in the next year. Mostly, I'm going to ZFS because it offers the same large-volume benefits of RAID without all those pesky data corruption issues. That and I can build an NAS for a fraction of what an equivalent RAID box would cost, NAS or DAS.

In fact, the only data I truly care about is just over 20 GB. That's the back-log of my projects, various documents and save-game files. That's the only stuff I'd actually be pained to lose. Which is why I'm looking at a way of backing up just that data to the cloud. There are a number of options in the sub-50 GB range that would fit the bill there.

Straight RAIDZ would act similar to RAID0, correct? So what you're suggesting is the equivalent of a RAID0+1? That would work but one thing I'm unsure of is if I create a two RAIDZ zpools and then put files on one, if I mirror the other to it does that destroy my data or does it just start the copy?

As you say, based on what I'm considering, larger hard drives are a long way off. I would almost look at smaller hard drives (since they have a slightly better track-record for reliability) but a 1 TB drive costs around $10-$20 less than a 2 TB. If I'm already spending $59 what's $79 for twice the capacity per drive?

 

[jpaetzel]

RAIDZ is a better choice for performance, RAIDZ2 will offer better more redundancy in the case of drive failures. RAIDZ is similar to RAID 3/5 not RAID 0. RAIDZ2 is similar to RAID 6.

In FreeBSD, RAIDZ seems to perform better than RAIDZ2

In ZFS a 5 drive RAIDZ performs better than a 5 drive RAIDZ2.

In something like a decade of doing RAID arrays, I've managed to lose filesystems to multiple drive failures twice where there wasn't an obvious hardware failure (eg black scorch marks on all the disks). Of course odds don't matter when it's your data that has vanished. I'd recommend RAIDZ and up to date off site backups.

 

[matthewowen01]

My data does not need to be online all the time. As I said, I'm the only one who's really going to be accessing it so if a drive dies and I need to leave the server for a day or two while it rebuilds? Fine.

that's a 2 day window that loosing another disk will hose everything.

Straight RAIDZ would act similar to RAID0, correct? So what you're suggesting is the equivalent of a RAID0+1?

raidz is basically a raid5, not a raid0. it spreads the data across all but one drive and uses the last to store some information that can be used to reconstruct any drive that fails (it's actually more complex than this but this should give you the general concept). With raidz, you can loose one drive and not loose data.

a raidz2 is basically a raid6, not a raid 0+1. Instead of one disk being able to reconstruct any lost disk like in raidz, there are 2 disks that can be used. with raidz2, you can loose 2 drives and not loose data.

i would recommend the raidz2, it wastes more space than a raidz but it gives more protection. when i used to run a raidz, i would sweat bullets when i lost a drive. yes there is a speed difference between raidz and raidz2, but i doubt it will be a bottle neck for you. with my 8 disk raidz2, i get about 2.5Gb/s read and write speeds. My 6 disk raidz2 is about 2 Gb/s i am assuming you have a single gigabit NIC which is only capable of 1 Gb/s. so it will still be your bottleneck.

 

[esamett]

it has been posted that 6 or 10 disks is optimal vs. 8.

 

[dougoftheabaci]

The case I'm using holds six drives. Did I say eight? Sorry, I was considering an eight-bay enclosure back when I was looking into doing a DAS box instead of an NAS box but I walked away from that.

I'm actually going to be using the Fractal Designs NAS box that is becoming rather popular on the forum. I found it before I even came here (chose hardware before I chose OS) and then saw almost the exact same configuration is used by a number of people.

However, I was planning on only using five of the drives for my general data pool and then saving the sixth as a backup drive. It's the drive that'll be used and accessed by far the most (Time Machine so every hour). If a drive is going to fail it makes sense for that to be the one that fails first.

Thanks Matthewowen01, that's what I had originally suspected but what had been said here was confusing me a bit.

I can understand the push for RAIDZ2 over RAIDZ due to that extra level of protection, and I might do that anyway, but I have to think about it.

One question I have that I haven't been able to find anywhere is whether or not RAIDZ can be increased? Either by replacing each drive with a larger one or by adding new drives to it. Not as a separate RAIDZ but actually just adding one more disk to the pool.

 

[matthewowen01]

One question I have that I haven't been able to find anywhere is whether or not RAIDZ can be increased? Either by replacing each drive with a larger one or by adding new drives to it. Not as a separate RAIDZ but actually just adding one more disk to the pool.

It is possible to replace each disk, one by one with larger ones to increase your pool size.
It is also possible to attach a second pool to the first, for example, you could add a second raidz2 to your first, ie, a raidz2 of 6 disks, and add a raidz2 pool of 8 disks for a total of 10 data disks, 4 are protected by 2 and 6 are protected by a different 2.
It is Not possible to take a raidz2 of 6 disks and add 2 to make it a raidz2 pool of 8 disks.

 

[jafin]

One question I have that I haven't been able to find anywhere is whether or not RAIDZ can be increased? Either by replacing each drive with a larger one or by adding new drives to it. Not as a separate RAIDZ but actually just adding one more disk to the pool.

If you replace all the drives in the vdev with ones of a larger size then the capacity of the vdev will increase.

See this thread. http://forums.overclockers.com.au/showthread.php?t=961125
See the note about the autoexpand property on the pool

Code:

zpool set autoexpand=on poolname

 

[dougoftheabaci]

I kind of had a feeling that's how it was going to work. Still, works fine. By the time I fill up five 2 TB drives 3 TB drives are going to be dirt cheap. I'm still going back and forth on whether I want RAIDZ or RAIDZ2, but at least now I know why to choose one over the other.

Thanks for the advice, guys! Now I just have to start saving so I can build this bad-boy.

 

[Milhouse]

@dougoftheabaci, can you stretch to 6 disks for storage? If so then I would go with two vdevs in a single zpool, each vdev with 3xRAIDZ1 disks (assuming 2TB disks, this gives 8TB usable). The advantage of this setup is that it would give you double the Write IOPS* compared with a single 5 or 6-disk vdev/zpool.

Alternatively, if 5 disks is your limit for storage then what about again two vdevs in a single zpool, but this time 3x RAIDZ1 and 2xRAID1 (mirror), 6TB usable. Again, double the Write IOPS of a single 5-disk vdev/zpool.

Each of these options would provide a similar level of redundancy to RAIDZ2, in that you could suffer up to 2 disk failures (one from each vdev), but you would be totally stuffed if you lost two drives from a single vdev.

* The Write-IOPS that a single vdev can produce is limited to the IOPS of the single slowest drive in the vdev. This is because each member of a vdev must sign off on any changes made to any files on that vdev, and the operation is performed individually. If you want more performance, add additional vdevs to the zpool, rather than making a single larger vdev.

 

[esamett]

Six drive bay is ideal for RaidZ2. There is no "need" for an additional backup drive. That's what the ZFS is for. If RaidZ1 then five disks is ideal. Having experienced a Raid drive failure I am firmly in the RaidZ2 camp. You have the room for it and if you watch the sales its only $60 bucks more than a 5 disk RaidZ1 array with the same amount of storage.

 

[Derision]

Wouldnt it be easier to go with RaidZ1 and just buy a spare drive ready if one fails?

 

[matthewowen01]

Wouldnt it be easier to go with RaidZ1 and just buy a spare drive ready if one fails?

easier yes, better no. it's the same thing except you have an hour or so that you are unprotected for when a drive fails. just do a raidz2.

as for the 2 vdev pair of raidz's you won't see a bit of difference since you're caped at your gigabit Ethernet connection. go for safety.

optimal settings are great when you're not capped by your network connection. you'll still exceed it with non optimal numbers of disks.

lastly ZFS, and raid in general is not a substitute for a backup.