20Tb RAID-Z3 build feedback (in development)

[Xam]

Hi guys,

Hello, greetings, aloha, buna seara from Romania :)

First thing first: i've been reading about ZFS and freenas for about a year (or so) so i decided it's time...to renounce the Synology NAS (although nice in it's functionality but horribly slow and/or costly hardware upgrades) and get myself a ZFS based NAS.

Now, for the past 2-3 weeks i've been trying to configure a box that will suite my needs:
- minimum available storage space: 20+ Tb
- relatively fast (wanna take full advantage of the LACP on the 24p Switch i have)
- minimize data corruption as much as possible (this being the primary reason i'm switching to ZFS)
- provide a way to maintain the entire multimedia database (plex jail possible in FreeNAS as i've tested it) and have enough cpu power to transcode without lagging on 3 streams (max) with full HD quality
- posibility to encrypt the entire ammount of data (20Tb)
- be able to provide iSCSI datastores to the esxi machines i'm running

So, i've come up with this (i would call this system version 213.5 :D )

Heat conductive paste
1x Spire WhiteGrease 0.3g
Model: SP-420/0.3G

Case
1x Cooler Master Cosmos II
Model: CACM1200KKN1
- 13 hdd slots
- plenty of cooling (or so i think)

SSD - for OS
1x Transcend 370 Premium Series 32GB SATA-III 2.5 inch

SSD - 1x cache and 1x logs
2x Kingston SSDNow KC300 60GB SATA-III 2.5 inch
Model: SKC300S37A/60G
- will be placed on the hotswap tray's of the case for easy replacement

HDD WD
10x SE 4TB SATA-III 7200 RPM 64MB
Model: WD4000F9YZ
- according to this calculator (https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/) the minumum hard drives required for a RAID-Z3 and have 20Tb of storage space available is 10.
- still trying to understand if i can later on switch the 4Tb drives with 6 or 8Tb drives in order to increase capacity and what would be the procedure to do this...

Power Supply
1x Seasonic Platinum 1200W SS-1200XP
- math time: 35Wx10=350W for hdd power consumption...the supply is 4 times larger...showld i get a smaller one? :-/

CPU
1x Intel Haswell Refresh, Core i5 4690K 3.5GHz box
Model: BX80646I54690K
- easily OC if needed

RAM
1x Corsair Vengeance LPX Black 16GB DDR4 2666MHz CL16 Quad Channel Kit
Model: CMK16GX4M4A2666C16
-still not decided if i should go for 32Gb or not...

Motherboard
1x ASRock Z97 Extreme6/3.1
Model: Z97 EXTREME6/3.1
- lots of SATA3 ports
- somehow i get the feeling that the MB is not the right choice for my situation :( because of non-ECC only supported RAM.

This system is supposed to be a some what of an end game as i'll be using it for a looong time from now on, but there is a possibility that next year i'll be building a bigger one than this (probably).

Havent bought the hardware yet as I will get them one by one during a 3-4 months time frame.

So...what do you guys think?
Hints/clues/recommendations/feedback is/are welcomed...

 

[CompuGlobalHyperMegaNet]

Hi guys,

Hello, greetings, aloha, buna seara from Romania :)

Welcome to the FreeNAS forum!

I'm gonna address several things below but the order of your post will be changed for the sake of clarity and not having to repeat the same point.

Case
1x Cooler Master Cosmos II
Model: CACM1200KKN1
- 13 hdd slots
- plenty of cooling (or so i think)

The biggest mistake I made with my FreeNAS build was the case. I did not choose wisely. If I was to share some advice with you in regards to the case, it'd be to fully understand what you want from the case and how and where it's going to be used. There aren't that many options out there for desktop case with a high drive capacity. I'd advise you to look at something that's a bit more utilitarian and cheaper than the Cosmos.

SSD - for OS
1x Transcend 370 Premium Series 32GB SATA-III 2.5 inch

Don't waste your money on an SSD for use as the boot device. The only time the performance of an SSD is noticeable is when booting the server. FreeNAS is loaded into RAM, so once it's booted up, there is no performance advantage with having an SSD instead on a USB stick. Speaking of which, FreeNAS was designed to be run from a USB stick. Do yourself a favour, save some money and just get a decent USB flash drive... or two (it's a good idea to mirror the boot device in the event that one of them fails).

SSD - 1x cache and 1x logs
2x Kingston SSDNow KC300 60GB SATA-III 2.5 inch
Model: SKC300S37A/60G
- will be placed on the hotswap tray's of the case for easy replacement

RAM
1x Corsair Vengeance LPX Black 16GB DDR4 2666MHz CL16 Quad Channel Kit
Model: CMK16GX4M4A2666C16
-still not decided if i should go for 32Gb or not...

Until you fully populate your motherboard's RAM slots, you shouldn't bother worrying about using an SSD as cache. You'll see far higher performance gains by adding RAM than adding an SSD... in fact, adding an SSD as an L2ARC can actually decrease performance when you don't have sufficient RAM.

HDD WD
10x SE 4TB SATA-III 7200 RPM 64MB
Model: WD4000F9YZ
- according to this calculator (https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/) the minumum hard drives required for a RAID-Z3 and have 20Tb of storage space available is 10.
- still trying to understand if i can later on switch the 4Tb drives with 6 or 8Tb drives in order to increase capacity and what would be the procedure to do this...

You can replace the drives with higher capacity ones at a later date. Although you won't benefit from the added capacity until all the drives are replaced. Basically, you have to replace a single drive, resilver the array, replace another single drive, resilver etc. etc. until all the drives have been replaced. Only then can you expand the array and take advantage of the added capacity.

You probably know this already but I'll say it anyway... you cannot add drives to a vdev. If you start with 10 drives in your RAIDz3 vdev, that's all you'll ever be able to have.

Power Supply
1x Seasonic Platinum 1200W SS-1200XP
- math time: 35Wx10=350W for hdd power consumption...the supply is 4 times larger...showld i get a smaller one? :-/

This system is supposed to be a some what of an end game as i'll be using it for a looong time from now on, but there is a possibility that next year i'll be building a bigger one than this (probably).

In that case, a higher than average efficient rating for the PSU makes sense... assuming the electricity in Romania isn't ridiculously cheap.

I'd go with a smaller PSU as long as you don't sacrifice quality and it actually represents a saving. Don't forget that PSU efficiency tends to be very poor below 10~20% load.

CPU
1x Intel Haswell Refresh, Core i5 4690K 3.5GHz box
Model: BX80646I54690K
- easily OC if needed


Motherboard
1x ASRock Z97 Extreme6/3.1
Model: Z97 EXTREME6/3.1
- lots of SATA3 ports
- somehow i get the feeling that the MB is not the right choice for my situation :( because of non-ECC only supported RAM.

To quote you "minimize data corruption as much as possible". Neither the CPU or mobo supports ECC. At this point I'd usually recommend an alternative CPU and motherboard but I know nothing about the retailers in Romania. If you tell me where you'll be buying the hardware, I'm happy to take a look but if you'd like to do the leg work yourself, then I'd recommend looking at SuperMicro's motherboards (such as the X10SL7-F) and CPU's that support ECC and have a PassMark score of over 6000. The general recommendation for CPU when it comes to Plex is a PassMark score of 2000 for each 1080p stream you wish to use. So factor in 3x 2000 and then some more for FreeNAS.

Havent bought the hardware yet as I will get them one by one during a 3-4 months time frame.

So...what do you guys think?
Hints/clues/recommendations/feedback is/are welcomed...

[/QUOTE][/QUOTE]

 

[Ericloewe]

SSD - 1x cache and 1x logs
2x Kingston SSDNow KC300 60GB SATA-III 2.5 inch
Model: SKC300S37A/60G
- will be placed on the hotswap tray's of the case for easy replacement

Those SSDs are very crappy. You'll probably want to avoid L2ARC with so little RAM and you probably won't need an SLOG.

1x Seasonic Platinum 1200W SS-1200XP

Don't overdo it. A Seasonic Platinum 650 would be good enough for 12 drives.

1x Intel Haswell Refresh, Core i5 4690K 3.5GHz box
Model: BX80646I54690K
- easily OC if needed

No ECC support, so it's immediately a bad choice. Overclocking a server is one of those ideas that really should never even be considered.

1x Corsair Vengeance LPX Black 16GB DDR4 2666MHz CL16 Quad Channel Kit
Model: CMK16GX4M4A2666C16
-still not decided if i should go for 32Gb or not...

What? That's DDR4. Unless you're going to end up with a Xeon E5 v3 system, you'll always want DDR3. Also, 4GB DIMMs are a poor choice. 8GB should be the minimum size.


Have you read the Hardware Recommendations sticky?

 

[Xam]

Hi guys,


Thank first of all for your valuable input. Appreciated!

Now, on to answering and asking questions

@CompuGlobalHyperMegaNet

I'd advise you to look at something that's a bit more utilitarian and cheaper than the Cosmos

Any ideas?
Unfortunately the project has a minimum data space requirement (20tb) and until now, I couldn't find anything that goes above 10 drive slots... :( This being the primary reason for choosing this case.
Also the second reason is that it is a tower case, has somewhat adequate (my opinion) cooling and (according to the dimensions of the case) should fits perfectly in the space i have destined for it.

Don't waste your money on an SSD

Yeah...I was thinking about that and I concur...unfortunately the MB doesn't have a internal USB slot on it...still not fond of setting the boot nas usb drive on the outside...but I guess I can live with that (external usb) if I have no choice :)
How does the Transcend Jetflash 520 16GB Gold acting as a boot drive sound to you? :D

Until you fully populate your motherboard's RAM slots, you shouldn't bother worrying about using an SSD as cache ... in fact, adding an SSD as an L2ARC can actually decrease performance when you don't have sufficient RAM

True that...lesson learned... :)

Although you won't benefit from the added capacity until all the drives are replaced

I'm aware of that as I’ve read this before and I’m painfully aware that this is a time consuming process, but I've read somewhere (?) that I need enough space (??) before I start doing this...this means that I need at least 4Tb of available space (as I will use initially 4tb drive) before I pull out the drive and change it/them with 6/8Tb drive...did I got this right?

you cannot add drives to a vdev

I know i know :) i also know that 11 drives is the max a vdev can have :) so...i'm pushing the limits :D

assuming the electricity in Romania isn't ridiculously cheap

Not that cheap but affordable :)

efficiency tends to be very poor below 10~20% load

Any feedback on this: Corsair HXi Series HX850i ???
I know it a bit over the 650W bar @Ericloewe put up but I’m thinking that I will be changing the drives in about 2-4 Years, so why not get a PSU that can go longer than that and still be able to provide enough power to 12x 8Tb drives…

If you tell me where you'll be buying the hardware

www.pcgarage.ro – primarily because I can use a voucher for 1% discount on any order unlimited times…so… at an order of 20000 RON (which is ~4500 euros I get to go home with 500 euros…that’s half my salary…don’t get me wrong, but my 1000 Euros/month salary is a pretty decent one)
www.emag.ro
www.price.ro - finds the smallest price for an item from all the partner shops...if it finds it...there are lots of shops registered in the site, so you might find something that I didn’t found…maybe…

==================================================================

@Ericloewe

You'll probably want to avoid L2ARC with so little RAM and you probably won't need an SLOG

I had an epiphany and will be dropping the SSD from the configuration

A Seasonic Platinum 650 would be good enough for 12 drives

Couldn’t find that model you were talking about…but found the one mentioned above (Corsair HXi Series HX850i) J

No ECC support, so it's immediately a bad choice

Thought so…and wasn’t planning to actually OC the CPU I just said:

- easily OC if needed

… you'll always want DDR3. Also, 4GB DIMMs are a poor choice. 8GB should be the minimum size

Already made up my mind with going for full RAM slot occupation on the MB.
Problem is I can't seem to find an Mb around the 300-400 euros price tag that will suit the needs of the many and still available within Romania.

Have you read the Hardware Recommendations sticky

Yes I did…even before posting actually, but I can't abide for all rules, even though I completely agree with them, mainly because of budget issues.
And can't seem to find a case to fit... :(

IPMI is awesome

I can't argue with this…I know it is…but my budget/month which racks up to 500-700 euros is not that awesome…that’s why I'm building it piece by piece over a few months.

SATA Port Multipliers at all costs

So…by that logic I should better be off with JBOD on 12 SATA ports in the MB…where can I find such moboat a reasonable price tag?!?! This…kind of sucks…:(

Do not try to change this philosophy, you will only hurt yourself

I’m not trying to change anything…just to make my point…but apparently the philosophy will do everything in its power to hurt my budget :(

Would love this case though :) -> http://www.supermicro.com/products/chassis/4U/946/SC946ED-R2KJBOD.cfm

 

[solarisguy]

Could you please check the price of a RAIDZ3 configuration with 8 * 6TB WD Red drives ?

Drives might be more expensive, but you do not need a weird case and a grossly oversized power supply. Additionally, less SATA connections required, would give you more motherboard choices ;-)

 

[solarisguy]

@Xam, could you please provide the starting page for the RAID calculator you had referenced in your initial post? Thanks!

 

[Xam]

@solarisguy

I'm not really fond of the RED line of WD to be honest...had 2 before, different batch, different year, both broke down in a two months period...and that's when i said stop to consumer range products especially when it comes to hdd's...

But what the heck...

Pricetag 4Tb drive (10 drives)
price/hdd - ~250 euros
price/vdev- ~2500 euros

Pricetag 6Tb drive (8 drives)
price/hdd - ~298 euros
price/vdev- ~2384 euros

Pricetag 8Tb drive (7 drives)
price/hdd - ~535 Euros
price/vdev- ~3745 Euros

Prices have been calculated at the exchange rate for today.

By this logic the 6tb would be cheaper if both all at once, problem is i was planning to buy two drives every month going with the 4tb drives...and probably starting in 2017 to buy one 6 or 8tb drive every month. :(

Link to calculator: https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/

 

[solarisguy]

How did you plan to arrange your storage? You cannot grow RAIDZ3 in steps! (And neither RAIDZ2, nor RAIDZ1.)

Let's say you start with 8 drives that are 500GB each. Your pool size would not increase until you replace all 8 drives.

 

[solarisguy]

@Xam, could you please provide the starting page for the RAID calculator you had referenced in your initial post? Thanks!

I went to https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/ but all I can do is to enter values in the fields provided. The results are always 0. It does not update anything automatically. It does not update if I press Result button. It could be my security settings though...

 

[Xam]

How did you plan to arrange your storage

I dont think i understood the question :(

Your pool size would not increase until you replace all 8 drives.

I know...but the fact that i can expand it gives me leverage in choosing a case that's not to expensive... :)

continuing were you left of, the process is like this:

initial config: 8*1tb drives in a 8 hdd slot case running raidz3 give me a total of 3.936TB usable space.

note1: the minimum amount of space required for an vdev size upgrade is 0.984TB so make sure it's free before the upgrade process.

note2:for each drive i pull out and insert a 2TB drive it take 82.43h for the vdev to recover/upgrade - I see this as a controlled vdev "crash" :)

process: eject and replace ONE drive every 82.44h but only after making sure the vdev is still there and alive

finale :) after 27.52 days the system will be done and the total amount of space will then be 7.872TB with the minimum recommended free space of 1.968TB

Off course, i may be wrong, so please...anyone...correct me if i am wrong...

It could be my security settings

It is...change and recheck :)

 

[Ericloewe]

Problem is I can't seem to find an Mb around the 300-400 euros price tag that will suit the needs of the many and still available within Romania.

Amazon.de ships to Romania, no problem, and even the X10SL7-F is currently under 300€. Even if that needs an additional 10% tacked on, from the exchange service, delivery and possible VAT difference, you're still looking at a better deal than you seem to get in Romania.

So…by that logic I should better be off with JBOD on 12 SATA ports in the MB…where can I find such moboat a reasonable price tag?!?! This…kind of sucks…:(

X10SL7-F.
Also, SAS expanders are fine. It's just SATA port multipliers that suck.

Corsair HXi Series HX850i

It's not bad, but Seasonic Platinum is better. Price should be mostly comparable.

Any ideas?
Unfortunately the project has a minimum data space requirement (20tb) and until now, I couldn't find anything that goes above 10 drive slots... :( This being the primary reason for choosing this case.
Also the second reason is that it is a tower case, has somewhat adequate (my opinion) cooling and (according to the dimensions of the case) should fits perfectly in the space i have destined for it.

Consider a regular ATX case with a lot of 5.25" drive bays and added hotswap bays (see my sig for such a setup).

 

[Xam]

@Ericloewe

Let me get this right...your case can hold up to 24 drives?!

I'm in... :))

I've started researching for parts on amazon...my wallet and i are more than happy at what were're seing...

Will update with a parts list probably tommorow based on the recommandations i've got so far :)

Thanks to all! ^_^

 

[Ericloewe]

Let me get this right...your case can hold up to 24 drives?!

No, more like 15 in a reasonable but crowded 3x 5-in-3 bay configuration. If you can find a case with 12 external 5.25" bays, you could do 20 drives.

Keep in mind that the 5-drive bays don't easily fit in all cases.

 

[Robert Trevellyan]

i also know that 11 drives is the max

This is not a hard limit. The strong recommendation is "not too wide", and 12+ is generally considered "too wide".

for each drive i pull out and insert a 2TB drive it take 82.43h for the vdev to recover/upgrade

First, note that one benefit of ZFS software RAID over traditional RAID is that it only resilvers in-use storage, not the entire drive capacity. Best practice is not to let your pool go above 80% full. Even at 80% your numbers seem way high to me. Please show your work ;)

 

[Xam]

Keep in mind that the 5-drive bays don't easily fit in all cases.

what? why? i was actually thinking of 3x 5-in-3 bay configuration in a case like yours... :(

This is not a hard limit

it is for me :) but one thing i still dont understand is this:

if the limit is for 11-12 disks max does this cover all the disks in the following config? or only to the 10 (respectively 3) drives in each vdev?

10 drives for RAIDZ3 vdev1 - 22.04 TB usable space
2 drives for hot spare - one for vdev1 and one for vdev2
3 drives for RAIDZ1 vdev2 - 6.298 TB usable space

only resilvers in-use storage, not the entire drive capacity

I know...the allotted times are for the full disk space occupation :) ... or so i think :(

 

[Robert Trevellyan]

or only to the 10 (respectively 3) drives in each vdev?

This - it's per vdev. For example, someone planing a 16-drive build would typically be counseled to go with two 8-drive vdevs.

the allotted times are for the full disk space occupation :)

How did you arrive at those numbers?

 

[Xam]

How did you arrive at those numbers?

By using this calculator: https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/

So i can only assume that the calculator will take into consideration MTTR for full data occupation of the drives as the minimum time to recover is 72h :)

 

[Robert Trevellyan]

Sorry, I don't understand that calculator. Maybe @Bidule0hm can help.

 

[Bidule0hm]

Please use the thread link because if I change something the url of the app will change too; I guess I really need to host the calculator myself to avoid this url problem

Regarding the MTTR question: yes, it's the worst case scenario. The 72 h you see is probably because of the MTTPR as it is 72 h by default and is included in the MTTR; you can set it to 0 if you just want to see the rebuild time ;)

 

[Robert Trevellyan]

the MTTPR as it 72 h by default

Is the MTTPR the estimate of how long it takes to obtain a replacement drive?

EDIT: setting that to 0 and leaving everything else the same gives a rebuild time of less than 11 hours, which seems much more reasonable to me.