20Tb RAID-Z3 build feedback (in development)

[solarisguy]

[...]
Regarding the pools...why two pools with one vdev each is better then one pool with two vdev? Somehow i'm missing the answer... :|

Two vdevs in two pools are not better, they are different.

Two vdevs in two pools:
You got two storage areas that you have to manage independently or you can manage independently, and that could be good or bad depending on usage.
You can have different hardware in each vdev (=pool).
You can upgrade each of them more easily.
A catastrophical failure of one does not make you loose all the data.
Some performance aspects could be better (with two users, each using a different vdev).


Two vdevs in a single pool:
You got one storage area, and that could be good or bad depending on usage.
You should attempt to have the same hardware in each vdev.
Upgrades are more of the hassle.
A catastrophical failure of one makes you loose all the data.
Some performance aspects could be better (reads are potentially substantially faster).

 

[solarisguy]

[...]
But i have one big ass problem...can't seem to find a HBA... :| [...]

You mean something like those two items that are shippable to Romania?
http://www.amazon.de/SilverStone-SST-ECS02-SFF-8087-9211-8i-Chip-Kontroller/dp/B00NNPL4X6 SilverStone, SST-ECS02
http://www.amazon.de/LSI-Logic-LSI00194-Adapter-8-Port/dp/B002RL8I7M LSI Logic LSI00194

They are both variations of http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9211-8i
The first one is made by somebody else. And the second one was made before Avago bought LSI.

 

[Xam]

well let me tell something about the vdev's maybe that will shed some light:

main purpose for the box is to be a transfer medium for LARGE amounts of data from my workplace to my home and back. When i say large amounts of data i'm talking about 10-15Tb files...each...
second purpose will be that of a media box.

therefore, i've "split" the workload into 2 vdevs:
1. composed of 10 hdd's (i know the max is 11) giving me a total usable space of 33tb (raid-z3) thus giving me the space i need to transfer the files to and from. The files will be accessed via a combination of NFS/SMB access (from windows and linux machines alike)
2. composed of the 4 hdd giving me a total usable space of 9.5tb (raid-z2) for files like music, movies, pictures, and so on, and space for plugins like plex/owncloud/bittorrent sync/etc that may or may not be played at the same time on 3 smart tv's around the house + other small devices (tablets, phones, etc).

SO...the workload on the second vdev won't be that high, but on the first vdev, just reading the files will be a stressful job for the server itself (or so i think).
BUT! these types of transfer happen once every few months...between them there will be smaller files like spreadsheets, documents, pdfs and so one...the usual shitload of documents one can have at any given point...

You are mentioning a catastrophic failure...can you be more precise?

 

[Xam]

http://www.amazon.de/SilverStone-SST-ECS02-SFF-8087-9211-8i-Chip-Kontroller/dp/B00NNPL4X6 SilverStone, SST-ECS02

this one is a bust is it does not support JBOD -> http://www.silverstonetek.com/product.php?pid=523&area=en

http://www.amazon.de/LSI-Logic-LSI00194-Adapter-8-Port/dp/B002RL8I7M LSI Logic LSI00194

this one's also a bust as the Compatibility Report available on the LSI website does not include the ST6000VN0001 Seagate hdd's i'm planning to buy... :|

 

[solarisguy]

this one is a bust is it does not support JBOD -> http://www.silverstonetek.com/product.php?pid=523&area=en


this one's also a bust as the Compatibility Report available on the LSI website does not include the ST6000VN0001 Seagate hdd's i'm planning to buy... :|

Both of them are OK.

You have to install IT firmware. The included product description is accurate for the default shipping configuration that has IR firmware. Read the forum for information about difference between those two firmware lines.

Current versions of IT and IR firmware can be downloaded from the page
http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9211-8i#downloads
You want
9211_8i_Package_P20_IR_IT_Firmware_BIOS_for_MSDOS_Windows
and most likely
Installer_P20_for_UEFI

Could you please provide the link to the Compatibility Report ?

 

[solarisguy]

A catastrophic failure means all the data is lost. For example, in the RAID-Z2 that has 6 drives, loosing 3 drives is a catastrophic failure.

 

[Scharbag]

A catastrophic failure means all the data is lost. For example, in the RAID-Z2 that has 6 drives, loosing 3 drives is a catastrophic failure.

Losing 3 drives should be avoidable if you have replacements as the odds of 3 drives failing at the same time are low.

Also, redundancy is for data availability and is not a replacement for backups :)

 

[solarisguy]

@Scharbag, no.

Having two vdevs independent of each other is a design decision that might be easy to implement.

Having a backup system is an easy (and obvious!) decision, that requires resources including real $$$$...

And, in the light of past 36 hours, your comment about data availability and backups makes no sense to me.

 

[Scharbag]

Data availability (uptime) is defined as MTBF/(MTBF+MTTR). System redundancy dramatically changes the MTBF (mean time between failure). MTTR (mean time to repair) would be how long it takes to fix the issue and restore from a backup. The MTBF of your system will include every component, including power supplies, RAM, MB, fans, drives etc. It just so happens that the least reliable component in a server is typically the spinning media (and many other component failures do not cause both loss of availability and loss of data). That is where RAID can help with availability.

If you have a system that can tolerate drive(s) failures and still perform as required, then the MTBF will be much greater than a system that cannot tolerate a failure. The more drive failures that your system can tolerate, the higher your MTBF will be (excluding other failures such as PS, MB, HBA, RAM, cables etc.). This is where a diminishing return on your investment comes into the picture as it cost the around the same to go from 99.9% to 99.99% (huge gain) availability as it does to go from 99.9999% to 99.99999% (very little gain) availability. That is what RAID (HW or SW) brings to the table.

Code:

Availability %    Downtime per year    Downtime per month    Downtime per week    Downtime per day
90% ("one nine")    36.5 days    72 hours    16.8 hours    2.4 hours
95%    18.25 days    36 hours    8.4 hours    1.2 hours
97%    10.96 days    21.6 hours    5.04 hours    43.2 minutes
98%    7.30 days    14.4 hours    3.36 hours    28.8 minutes
99% ("two nines")    3.65 days    7.20 hours    1.68 hours    14.4 minutes
99.5%    1.83 days    3.60 hours    50.4 minutes    7.2 minutes
99.8%    17.52 hours    86.23 minutes    20.16 minutes    2.88 minutes
99.9% ("three nines")    8.76 hours    43.8 minutes    10.1 minutes    1.44 minutes
99.95%    4.38 hours    21.56 minutes    5.04 minutes    43.2 seconds
99.99% ("four nines")    52.56 minutes    4.38 minutes    1.01 minutes    8.66 seconds
99.995%    26.28 minutes    2.16 minutes    30.24 seconds    4.32 seconds
99.999% ("five nines")    5.26 minutes    25.9 seconds    6.05 seconds    864.3 milliseconds
99.9999% ("six nines")    31.5 seconds    2.59 seconds    604.8 milliseconds    86.4 milliseconds
99.99999% ("seven nines")    3.15 seconds    262.97 milliseconds    60.48 milliseconds    8.64 milliseconds
99.999999% ("eight nines")    315.569 milliseconds    26.297 milliseconds    6.048 milliseconds    0.864 milliseconds
99.9999999% ("nine nines")    31.5569 milliseconds    2.6297 milliseconds    0.6048 milliseconds    0.0864 milliseconds

The important thing to know is that RAID is NEVER EVER EVER EVER to be considered a replacement for backups. End of story, full stop, nuf said. Anyone who says differently is lying to you.

If the data cannot be replaced, back it up. Irreplaceable data is quite manageable as we typically do not have TBs worth of it in our personal lives (at least I do not).

Cheers,

 

[solarisguy]

[...] System redundancy dramatically changes the MTBF (mean time between failure). [...]

That is the point I am trying to make...

 

[Scharbag]

A catastrophic failure means all the data is lost. For example, in the RAID-Z2 that has 6 drives, loosing 3 drives is a catastrophic failure.

But the above is only true if you do not backup. What it should say is that "A catastrophic failure means all the data is unavailable until restored from backups."

Cheers,

 

[Xam]

Hi guys,

Sorry for the late reply...

Could you please provide the link to the Compatibility Report ?

I've just searched for it but can't seem to find it anymore... :| will try and get the compatibility report from somewhere else.

@Scharbag
Thanks for the emphasis on the backups...but like i said before, this storage unit is intended as a transfer unit primarily of one to two large files per month...these large files combined can take up to 20Tb of storage space.
The second vdev will be for bringing together all the media sources i have around the house...bu that's not really as important as it sounds.

:)