11 disks in RAIDZ3 or 2 vDev's of RAIDZ2?

[darkconz]

Hi,

I apologize if this doesn't belong to Hardware section, but this didn't look like a configuration issue either... I did a quick search and I didn't find the answer to my question. I know the recommended practice is to keep it below 12 drives as I had read it on some of the posts here. Here I have a case which can host 12 drives and now I have a dilemma.

I have the option to configure my drives in my case:

1. Setup 11 drives as one big RAIDZ3 and have 1 drive as hot spare
2. Setup 2 vDevs, each with 6 drives in RAIDZ2 (no hot spare)

If my math is right, both will give me same amount of raw storage given all the drives are same capacity.

Which one is a better practice? Performance wise? Reliability wise?

Thank you for taking your time to answer my question.

 

[cyberjock]

I don't subscribe to the 12 drive limit myself. I have an 18 drive RAIDZ3 and scrubs do over 1GB/sec. There's performance and reliability tradeoffs. Small reads suffer in very wide setups, but large reads are much faster. Since I use the RAIDZ3 for streaming my DVD rips I don't care about small reads. In my case, I also monitor my SMART info for my drives, so seeing 4 drives fail simultaneously is, in my opinion, very unlikely for me.

Reliability-wise, there's a trade-off. Statistically you are better off with 2xRAIDZ2s. But if you happen to have 3 drives fail in 1 RAIDZ2 you are screwed, whereas in the RAIDZ3 you'd be fine.

 

[titan_rw]

I'd rather have an 11 wide z3 than a 12 wide 'double' z2.

The double z2 might give you slight advantages in random performance, as cyberjock pointed out. But depending on the expected workload, that probably doesn't matter.

I don't know if I'd bother with a hot-spare. Especially until zfs can fully and automatically utilize it. If money weren't an issue, I'd probably go straight to a 12 drive z3. Setup scrubs, occasional smart long tests, and keep an eye on the smart attributes, and I'd imagine you'd be ok. Having a 'cold' spare around might not be a bad idea. Especially if sourcing drives later takes longer than just popping by the local store.

You don't say how big these drives are going to be, but I'd make sure you have ample ram. If running socket 1155, I'd max out the MB to 32gig. Also keep in mind z3 combined with the large amounts of drives is going to use quite a bit of cpu. I've got an i5-3570 (quad core at 3.4ghz), and I've been able to max the cpu out on more than one occasion. (I've got 10 drives in z3).

 

[cyberjock]

You don't say how big these drives are going to be, but I'd make sure you have ample ram. If running socket 1155, I'd max out the MB to 32gig. Also keep in mind z3 combined with the large amounts of drives is going to use quite a bit of cpu. I've got an i5-3570 (quad core at 3.4ghz), and I've been able to max the cpu out on more than one occasion. (I've got 10 drives in z3).

Your i5 should be smoking my e5606 CPU and I haven't ever hit 60% CPU utilization. What else do you use it for besides CIFS? CIFS is all I use mine for. No jail, NFS, or AFP.

 

[titan_rw]

Your i5 should be smoking my e5606 CPU and I haven't ever hit 60% CPU utilization. What else do you use it for besides CIFS? CIFS is all I use mine for. No jail, NFS, or AFP.

For purely nas work, scrubs are the only time I really tax the cpu. I see 80% utilization when scrubbing. I also scrub about 1GB/sec. Even this was far more than I would have imagined. I saw a pretty big difference between RAIDz2 and RAIDz3 cpu usage.

Other features like compression, deduplication, snapshots, replication, encryption, multiple protocols, multiple nics, jail software, etc, are all going to add to cpu usage. I don't do all of these, but I have managed to load the cpu up pretty good.

If / when I upgrade to 10gbe, I wonder if this cpu is going to be able to keep up.

All I really meant by my post was to not underestimate the cpu requirements of software raid.

Btw, I'm still quite in awe of your 18 drive z3 pool. I'd like to ditch my consumer pc case, move into some kind of 4u chassis and do something similar. 24x 3tb drives, setup 2 vdevs of 12 drives in z3 sounds pretty awesome.

 

[darkconz]

I'd rather have an 11 wide z3 than a 12 wide 'double' z2.

The double z2 might give you slight advantages in random performance, as cyberjock pointed out. But depending on the expected workload, that probably doesn't matter.

I don't know if I'd bother with a hot-spare. Especially until zfs can fully and automatically utilize it. If money weren't an issue, I'd probably go straight to a 12 drive z3. Setup scrubs, occasional smart long tests, and keep an eye on the smart attributes, and I'd imagine you'd be ok. Having a 'cold' spare around might not be a bad idea. Especially if sourcing drives later takes longer than just popping by the local store.

You don't say how big these drives are going to be, but I'd make sure you have ample ram. If running socket 1155, I'd max out the MB to 32gig. Also keep in mind z3 combined with the large amounts of drives is going to use quite a bit of cpu. I've got an i5-3570 (quad core at 3.4ghz), and I've been able to max the cpu out on more than one occasion. (I've got 10 drives in z3).

I have the following parts coming in the mail right now...
E3-1230v2
Supermicro X9SCL-F
32 GB RAM

I haven't purchased the drives yet. I am debating whether to go with the WD Red 3TB or Seagate 4TBs. I am leaning towards the Seagate because if the array is in RAIDZ3, the very pessimistic calculation is still less than 0.00000003 failure rate in 10 years.

You had mentioned 12 drives in RAIDZ3, I thought 12 isn't the magical number =P. This is why if I were to go with RAIDZ3, then I should leave a slot open and have a back up drive laying around.

I agree that I won't benefit too much from 2xRAIDZ2 and I originally was leaning towards the RAIDZ3. However, I saw some potential performance gain with 2x vDevs so I couldn't resist and had to ask for other people's opinion =)

 

[titan_rw]

I don't think the 'magic number' of disks in a vdev matters that much unless you have really intense random workloads. Like heavy duty databases, or high io vm's running off of iscsi or something.

If it's just being used to share files via cifs, I wouldn't worry to much about it.

My 'non optimal' 10 drive z3 pool works wonderful.

11 drives in z3 would be good too. I'm not trying to convince you to do 12. Either way, that's a decent size pool.

 

[paleoN]

I know the recommended practice is to keep it below 12 drives as I had read it on some of the posts here.

FYI, the recommended practice is to keep the drives less than 10 in a single vdev, being 9 drives or less.

I don't subscribe to the 12 drive limit myself. I have an 18 drive RAIDZ3 and scrubs do over 1GB/sec.

Personally, I think that's foolishly wide, but I know you are aware of the issues, it may work well with your data anyway and I believe you left yourself other options. I shudder to think about trying to resilver one of those drives when the pool is mostly full. Still ZFS, ZFS on FreeBSD in particular, may very well have seen improvements in this particular area.

 

[maiitax]

What is considering good enough a raidz2 or raidz3 configuration?

I have 10 drives in a single vdev that were all bought at the same time. Within a month period I had a triple drive failure. I believe you could get away with raidz2 provided you diversify your drives by manufacture, model and hour count.

I am in the process of rebuilding my server and I am just wondering what is the best answer to OP's question? (8 + 2) or (8 +3) or (4 + 2 and 4 + 2). From what experience has taught me if you don't diversify there is a very good chance you will lose another drive or two during the resilvering process. I know raid is not a backup solution.

I have another pool 3 x 4TB raidz1 which I am not comfortable with either. I'm at the point where I read posts like this and I don't know what to do anymore. So many different options with no definitive answer.

 

[Starpulkka]

Not sure about device number limits, its tweakable, but i look safety and integrity from my zfs setups (dont everyone?). Have been there that intel driver screwed important data and very next tryed backups and same happened an instant i put power on on hddrack. Have been using zfs almost 1.5 years now and started with raidz1 pool 5 months later looked this forum and decided change pool raidz2 and very next month 1 of 6 hdds started to get errors. Moved data to safety and started resilvering on first hdd and 2 days later another hdd did broke. So google statictics is correct that theres high chance that you get 2 hdd fails on same week. What more i have learned that if your hdd temperature goes over 35-40c hdd lifetime drops, ofcourse theres variation on differend brands and models. As for your setups i would go option 2vdevs other options on even 11 raidz3 would be a nono for me. (just calculate hdds is 2 of 6 worst than 3 of 12) or (is 2 of 6 worst than 3 of 11) (if you calculate 2hdd of 18hdd)... its everyones freewill of course how many to use and by that zfs is awesome.

 

[therealmarv]

I only want to answer this (although late) because this is wrong risk calculation. It's more risky to run a 6 disk setup with parity 2 than a 12 disk setup with parity 3 (it's really playing in another league when looking at MTTDL). See yourself the calculations here http://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/ and here https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

 

[AVB]

Being that I run two 8 drive Z2 volumes in my pool in my main box (3TB drives) and two 6 drive Z2 volume in my pool in my backup (4TB drives) I looked at growth in that it is easier and less initial cost to grow one smaller volume than a larger one. Of course the end result is the same cost. The odds of having 3 drives all go out at once in one volume were sufficiently large that I chose not to worry about it.