Selecting raidz-level and buying new hdds

[otpi]

Hello there,

I'm just rambling here, but maybe someone finds this useful: I just decided on what number of hdds to buy. I have crawled the interwebs searching for the definitive answer to the optimal raidz setup and number of disks I thought I'd share my conclusion, and the whys.

My hardware, and experience so far:
I'm your typical home user searching for a good NAS solution. I need rock solid backup of family photos and "important documents", and I'd like to have my media collection available for streaming...

I got lucky and scored some free server hardware from work: HPs with xeon E5606s (4 cores, 2,16 Ghz), ECC memory and a bag full of used 1 TB drives. I set up a 6 wide raidz2. So far FreeNAS has been rock solid, in spite of my failures. I've had disks fail -> resilvered; usb boot drive fail -> reinstall to new usb; update failed (freenas 10 nightlies) -> reinstall; NAS out of commission for 6 months due to "life" -> installed the new version of freenas (11.2). Bottom line: all data is still intact.

So now I'm ready to "invest" in some new storage and consolidate all our datas:
I went back and forth between 4 vs 6 hdd setup, and whether to go for 4 TB or 8 TB hdd

(My) Conclusion: 4 hdd, raidz2, of 8 TB hdds.
- Yes that's 50% "loss of space", but I have grown to accept the price of redundancy.
- 4 TB and 8 TB have almost the same price/TB (today)
- 4x8TB is only slightly more expensive than 6x4TB. I consider 8 TB hdd more "future proof", and available TB is the same for both (~16).

4 disks are 2/3 the cost of 6, that's a lot. 33% (on top of today's black friday deal).

With a 4 hdd setup I can double the available capacity with 2 extra hdds, if needed.

"But you cant add hdd to vdevs?"
True, not today, but openzfs is working on it. AND; I will most likely discover the need to expand to a 6 disk z2 before it's a problem to offload my data to some other device temporarily.

Also, I only have 6 sata ports on my motherboard (unless I buy some pci-sata hba). A 4 disk vdev leaves one for a boot ssd (better than using usb). And I have an unused SSD from an old system.

That spare SATA port? Perfect for a backup hdd. Raidz is redundant, not backup, this way I can have internal backup, in addition to the copies stored on my devices. This is only for "important stuff", so I do not need lots of capacity, can probably use one of the old drives. Spare sata can also be used in case of drive failure, upgrading hdds etc. just so many options.

Last: According to backblaze stats I should expect ~1% failure rate. With 4 drives I'm actually less likely to experience a drive failure.

cheers :)

 

[hescominsoon]

I would get 6 sata HDD for your storage array and then get 2 X 64GB usb drives for boot. Freenas will automatically mirror the boot usb's then you can use all 6 for storage drives.

 

[Evertb1]

- 4 TB and 8 TB have almost the same price/TB (today)
- 4x8TB is only slightly more expensive than 6x4TB. I consider 8 TB hdd more "future proof", and available TB is the same for both (~16).

I am not sure if you did your calculations correctly concerning the cost and the practical usable storage. Personaly I never liked the economy of using 4 drives in a RAIDZ-2 vdev.
With the drive capacity of 4 or 8 TB you will not end up with 16 TB practical usable storage. You need to take in account that you not only will have "Slop space allocation" but also need to deal with a certain percentage of "Free space limit". I always calculate 20 % for that. And I use the ZFS calculator you can find here https://wintelguy.com/zfs-calc.pl.

I don't know anything about the prices in Norway but at my favorite reseller (not a price fighter mind you) I would pay around eur. 119,90 for a 4TB drive and eur. 232,90 for a 8 TB drive. And Seagate Iron Wolfs and WD Reds have about the same price tags.

Would I build with 6 x 4 TB, I would end up with around 11.1 TB practical usable storage and a total cost of eur. 719.40 (or eur. 64.54 per TiB).
Would I buidl with 4 x 8 TB, I would end up with around 10.8 TB practical usable storage and a total cost of eur. 931.60 (or eur. 86.19 per TiB).

You might say that the difference in price is very favorable for 6 x 4 TB.

About your concerns for the future. As soon as you start using your pool you know that the drives in it are only good for a certain life span. I will replace my pool when my drives are around 5 years old. Should a drive fail before that I will replace it with a similair drive or, if I am close to a planned replacement of my pool, I will replace the complete pool. And with drives that are mainstream at that moment and have a capacity needed for the next 5 years. If you calculate the capacity of your pool based on storage needs for the coming years you should be fine.

 

[Scharbag]

Firstly, boot from redundant USB keys and save your SATA ports for HDDs. FreeNAS loads into RAM for the most part and rarely needs to boot so there is almost no real benefit to using any other boot media.

As for the RaidZ choice: If I were you, I would go with a 6 drive RaidZ2. This way, your vDev is 6 drives to start with. The reason I say this is that you can NEVER (unless this changes some day...) change a 4 disk vDev to a 6 disk vDev. You can add another 4 disk vDev to the pool, but you need to have the physical drive bays available, which you do not at this time. On that note - avoid the use of SATA cards. I would recommend a proper SAS HBA controller like an LSI 9211-8i. If this is what you had in mind, then cheers.

As time goes on, you can upgrade each disk one at a time (detach and replace) and once every disk in your vDev is upgraded, the pool will auto-expand.

If you are going to use 4 disks, I would personally use them in a stripe of mirrors instead of RaidZ2. That does give you the option to add another pair of drives into the pool in the future AND you get some speed advantages through the use of multiple vDevs. Initially, you could even run 2 @ 2 disk vDevs with new disks and 1 @ 2 disk vDev with old disks to maximize your storage immediately. You would then have lots of spares for the old vDev and when you wanted to, you could replace the old disks with new to again increase the storage space available.

ZFS has many ways to skin the cat!!

Cheers,

 

[otpi]

ZFS has many ways to skin the cat!!

Indeed! Which is why I have spent a year in "lab mode" using only free hardware. I see that my rant was unclear in many ways... Happens! Maybe I can clear things up. I am aware of all the above, too some degree. Striped mirrors, as I understand it, do not give the same redundancy as plain raidz2, and I do not need the performance. There is a subtle difference between two disk failure and "any two" disks. If using stripes, it better not be both in disks in one vdev. That's my understanding at least.

I like having options, and not spending too much. Which is why I chose the 4x8TB option. More options (IMO), same price. I have the option to buy two more disks for a 6 wide z2 and ~double available capacity. Yes, I need to offload data, destroy old vdev, make a new one, then copy back data, but that's not a problem (ymmv).

I am not sure if you did your calculations correctly concerning the cost and the practical usable storage. Personaly I never liked the economy of using 4 drives in a RAIDZ-2 vdev.
With the drive capacity of 4 or 8 TB you will not end up with 16 TB practical usable storage. You need to take in account that you not only will have "Slop space allocation" but also need to deal with a certain percentage of "Free space limit". I always calculate 20 % for that. And I use the ZFS calculator you can find here https://wintelguy.com/zfs-calc.pl.

I don't know anything about the prices in Norway but at my favorite reseller (not a price fighter mind you) I would pay around eur. 119,90 for a 4TB drive and eur. 232,90 for a 8 TB drive. And Seagate Iron Wolfs and WD Reds have about the same price tags.

Would I build with 6 x 4 TB, I would end up with around 11.1 TB practical usable storage and a total cost of eur. 719.40 (or eur. 64.54 per TiB).
Would I buidl with 4 x 8 TB, I would end up with around 10.8 TB practical usable storage and a total cost of eur. 931.60 (or eur. 86.19 per TiB).

You might say that the difference in price is very favorable for 6 x 4 TB.

About your concerns for the future. As soon as you start using your pool you know that the drives in it are only good for a certain life span. I will replace my pool when my drives are around 5 years old. Should a drive fail before that I will replace it with a similair drive or, if I am close to a planned replacement of my pool, I will replace the complete pool. And with drives that are mainstream at that moment and have a capacity needed for the next 5 years. If you calculate the capacity of your pool based on storage needs for the coming years you should be fine.

Prices in Norway are about the same +- difference in VAT. I can see my wording was misleading, or frankly wrong... Price/TB is ~the same for both sizes, but "(today)" referred to a 20% black friday discount on the 8 TB disks. Which is why total cost "today" for 4x8TB and 6x4TB came so close (~€800 for 4x8, and ~€720 for 6x4TB.). However, I'd still go for the larger disks even if I'd have to pay the full price.

I agree that it is not the most efficient use of disks in terms of total too available TB compared to installed TB. Had I gone for the more "efficient" 6x4 TB setup, and discovered that it does not meet my requirements, I'd need to buy another 6 disks + an HBA (striping two 6 wide z2's). Potentially 12 disks, instead I have 4, potentially 6.

 

[danb35]

Striped mirrors, as I understand it, do not give the same redundancy as plain raidz2, and I do not need the performance.

You understand correctly, so far as you've said. One other consideration favoring striped mirrors, though, is that you can easily and safely expand your pool with just two more disks, by adding another mirrored pair to the pool. And, once 11.2 goes final (or if you like playing with fire and using beta releases), you can even remove mirrored vdevs from your pool (though I'd guess you'll need to do that at the CLI). This flexibility can be particularly useful for folks with smaller installations.