15 bay drive configuration

[Spazzy]

Howdy,

First post please play nice.
Freenas is going to boot off USB
How would you configure this drive array in a 15 bay case ?

The blank drives I have on hand for this project.
5x WD30ezrx (3TB)
5x Deskstar 7K30000 F12460 (3TB)
5X Deskstar 7K2000 0F10311 (2TB)
The deskstars will gradually be replaced with more WD30ezrx.

Primary usage will be storing/serving large ISO files of DVD's to serve to 3 KODI setups in a windows share.
What my feeble mind came up with is 1 pool, 2 vdevs, 7 drives each using raid-z2 + one hot spare.
seems nice and clean.
I can not help but think there could a better way to do this though.

7 drives feels wrong in a z2 configuration, not sure I fully understand where I need to be worried.
Usage of a hotspare looks good on paper but is it really? I can not find any reason NOT to use one, if the slot is available. Is it really just a case of I am admin/god so I control what happens and when, or is there a level of caution that needs to be considered that I am overlooking?

rest of the hardware cause someone will ask,
Mobo: SuperMicro H8DAE
CPU: 2x Opteron 285 cpu's. (2600mhz)
RAM: 8x 4GB DDR3 ECC RAM (32GB total)
I actually waited to upgrade from 16 before I posted just to be safe
SATA Controller: 2x SuperMicro AOC-SATA-MV8
no drool fuel here.

 

[depasseg]

2x7 looks good, and your hot spare is a good plan.

The only thing to think about is that the 2x3TB drives will be limited to 2TB when mixed with the 5x2TB's. Once you replace the 2TB's with 3TB's then you will gain all that space.

 

[Spearfoot]

Howdy,

First post please play nice.
Freenas is going to boot off USB
How would you configure this drive array in a 15 bay case ?

The blank drives I have on hand for this project.
5x WD30ezrx (3TB)
5x Deskstar 7K30000 F12460 (3TB)
5X Deskstar 7K2000 0F10311 (2TB)
The deskstars will gradually be replaced with more WD30ezrx.

Primary usage will be storing/serving large ISO files of DVD's to serve to 3 KODI setups in a windows share.
What my feeble mind came up with is 1 pool, 2 vdevs, 7 drives each using raid-z2 + one hot spare.
seems nice and clean.
I can not help but think there could a better way to do this though.

7 drives feels wrong in a z2 configuration, not sure I fully understand where I need to be worried.
Usage of a hotspare looks good on paper but is it really? I can not find any reason NOT to use one, if the slot is available. Is it really just a case of I am admin/god so I control what happens and when, or is there a level of caution that needs to be considered that I am overlooking?

rest of the hardware cause someone will ask,
Mobo: SuperMicro H8DAE
CPU: 2x Opteron 285 cpu's. (2600mhz)
RAM: 8x 4GB DDR3 ECC RAM (32GB total)
I actually waited to upgrade from 16 before I posted just to be safe :)
SATA Controller: 2x SuperMicro AOC-SATA-MV8
no drool fuel here.

Welcome to the forum!

There's nothing wrong with 7-disk RAIDZ2 vdevs - I use one (see my signature for details).

You'll hear differing opinions about hot spares: some think they're a waste of resources and should only be used in systems that you must administer from a physically remote location that takes several hours transit time to reach. I don't have a horse in this race... :)

Did you mean AOC-SAT2-MV8? I can't find any info on an AOC-SATA-MV8 controller. The AOC-SAT2-MV8 has several things going against it: it's based on a Marvell chipset, which may or may not work with FreeNAS; it's limited to 3Gb/s; and it may not support drives > 2TB in size. Instead of this out-dated card, you might want to consider an LSI-based PCIe HBA such as the LSI 9211/IBM M1015/Dell H200.

Good luck!

 

[Jailer]

Make sure all 2TB drives are in the same vdev. I say unless the server is generally inaccessible skip the hot spare and run two 7 drive RAIDZ2 vdevs. If you're going the have the drive in the server and spinning it might as well contribute to your available pool space.

 

[Mirfster]

I vote for the 2 x 7 vDevs. Use the extra port for something like a SSD. Can use it for the OS Drive instead of USB (since I am partial to SSD over USB), as a jail, SLOG, etc. Shelf the other drive as a "Cold Spare".

Of course do burn in all the drives to be sure they are up to par.

If what @Spearfoot says is correct, then I agree that you should get a real HBA.

First post please play nice.

You sir ask too much of us.. :p

CPU: 2x Opteron 285 cpu's. (2600mhz)

Might be an issue, but I don't run AMD so can't really say one way or another. Others who do may chime in and clear any concerns. @joeschmuck used to (and maybe still does) so that could be one data point.

+1 For a good first posting which actually included hardware specs and use-case info.

Lastly, Welcome!

 

[Spazzy]

Thanks for the warm welcome and replies.

To address some comments:
I do realize that I would only have 2TB (1.8TB usuable) available per drive per vdev until I upgrade all the drives and expand the vdev. That said, moving forward, lets assume they are all 2TB drives, and I will be sure regardless of the configuration, to keep at least one 2TB in each vdev to keep the math clean.

AOC-SAT2-MV8 is the correct model of the card, my bad & Good catch. They do see 3tb with no issues and
It is a HBA. I must have the revision that does seem beyond the 2TB barrier.
As much as I would like the other LSI card mentioned or any compatible SATA III controller, this card and motherboard combo are pci-x. not eXpress :-(, That being said, still fast enough for my purpose.

FreeNAS seem to be working out ok on the configuration and AMD CPU's
I did googlefu for people running similar hardware before I even thought of lifting the case up to bring it home. it is a beast.
3U 15bay Supermicro 933t or Riverbed interceptor 9020 (both the same thing) for the googlers.

So back to the original question

I have already been playing/testing
2 vdev 7 drives each in RaidZ2.
2 vdev 6 drive raidz2 and 9 drive raidz3 .
4 vdev 3 drive each raidz1 + 3 drive vdev raidz1 spare

They seem to have the same available storage space regardless, however, the 4 drive vdev raidz1 seem cleaner, does take follow the 4k recommendations

 

[Spazzy]

WTF oops, must of hit post before I finished typing. (stupid trackpad)

....And it copies data across itself much faster.
I guess I could of mentioned this in the previous post however, I really wanted to hear from others to learn from different points of view, and thank you for sharing.
I am leaning towards the 7 drive 2 vdev configuration however. and as I am a newb...

I am really curious why the 4 drive raidz1 configuration is actually not more efficient and redundant in this scenario. I know we automatically wanna say NO to raidz1. In this scenario, would it be just as good if not actually better?

ok, now you can bend me of the knee and break out the yard stick to teach me right from wrong ;-)

 

[Mirfster]

I have already been playing/testing
2 vdev 7 drives each in RaidZ2.
2 vdev 6 drive raidz2 and 9 drive raidz3 .
4 vdev 3 drive each raidz1 + 3 drive vdev raidz1 spare

They seem to have the same available storage space regardless, however, the 4 drive vdev raidz1 seem cleaner, does take follow the 4k recommendations

I am leaning towards the 7 drive 2 vdev configuration however. and as I am a newb...

I am really curious why the 4 drive raidz1 configuration is actually not more efficient and redundant in this scenario. I know we automatically wanna say NO to raidz1. In this scenario, would it be just as good if not actually better?

Just for clarity:

• Redundancy is not accumulative, it actually is vulnerable to the weakest level
  
• Loss of any vDev that is part of a Pool will result in loss of that entire Pool
• RaidZ1 only provides protection from loss of one disk in the vDev, lose two disk (like while one is re-silvering) and it is toast
  
• RaidZ2 vDev can withstand losing two disks

Think of it this way, say I have a Pool consisting of 6 x 4TB RaidZ3 vDevs and 1 x 5TB drive. Redundancy of the Pool is now all gone for all practical matters. Even though those RaidZ3 vDevs can withstand losing 3 drives each, if that single 5TB drive were to die then the entire Pool is lost...

 

[Arwen]

You can do this;

1 vDev with 7 disks in a RAID-Z2, (mixed 2TB & 3TB)
1 vDev with 8 disks in a RAID-Z2, (all 3TB)​

This will give you a bit more space. And when you replace the 5 x 2TB drives, that vDev
will jump up in size.

It's generally recommended to have the same drive counts between vDevs, as well as same
size disks. Plus, same redundancy, (Mirror or RAID-Z{1,2,3}), in a single Zpool. But,
for many home uses, like yours, the above configuration should work just fine.

I do prefer SATA DOMs or SSDs over USB flash drives, just like @Mirfster suggested.

 

[Stux]

Use 2 7 drive vdevs.

Leave the 15th port empty. You need an empty slot to upgrade a drive with maximum safety anyway and you plan to do that.

Eventually you can add an SSD for some reason, say slog, arc, boot, etc.

Or use it for a 10TB archive drive for backups.

 

[SFoskett]

Use 2 7 drive vdevs. Leave the 15th port empty. You need an empty slot to upgrade a drive with maximum safety anyway and you plan to do that.

This.

Either stripe 2x 7-drive RAIDZ2 sets (keep 4x 2TB drives in one and toss the extra) or stripe 7x mirrors (and toss one of the 2TB drives). The advantage of the mirror configuration (apart from performance) is quicker upgrades: As soon as you replace two of the 2TB drives, your pool is 1 TB bigger, rather than waiting to replace all four (running) 2TB drives. And in the future you can replace the drives in pairs to get much more capacity. But of course you've got only 29% overhead with 2x7 RAIDZ2+0 and 50% with 7x2 RAID1+0...

But leave that last SATA port empty. And when you do replace the 3TB drives, keep one or two around as a (cold) spare. You'll be glad you did.

 

[Spazzy]

I want to thank all you of for your input. It has been edumacational.
I think for the 15th bay...
I am not even going to connect it at the back-plane and just use it to store a cold spare.
I am sold on using it for a SSD.
I am going to pick up a couple of those slick PCI expansion slot trays for 2.5" disks and run sata port 0 from each card to them. I do have a few SSD's on hand, it would make a nice redundant mirror boot device.

Doing the 2 drive mirror vdev x 7 is attractive and clean.
I could handle the 50% overhead, that actually was one of the configurations I already played with. I mirrored based on their respective card port numbers. Splitting the mirror between the cards thinking it would help with the workload per card and heat and vibration per drive. It was nice splitting things evenly across the cards and fun to watch the lights as I blew things up.

Conclusion of that was, I am not sure how much I would benefit with the performance at the cost of space.
Still seems sketchy with the mirror even though the rebuild is much quicker and supposedly easier on the drives.
Not sure how much less riskier it really is.

I do appreciate all your advice and input.

 

[Stux]

Let's not discuss 5 3-way mirrors ;)

That's how people who need the performance of mirrors improve the relisbility.

33% storage efficiency!