Scalable hardware setup

[danb35]

As long as one copy of all metadata is recoverable (assuming that's what you mean by "trash all of it") you should be okay.

Yeah, I wasn't very clear. Given that there's more than one copy of any given piece of metadata (and, IIRC, those copies are spread around the disk), the chance of any given URE affecting all copies of any given piece seems low. And if a clean copy is available, then your pool is not trashed.

 

[Kevin Horton]

Let's assume you are using RAIDZ2, and a disk has failed. You replace the disk and now the new disk must be resilvered. If there is an URE when data on the other disks is read, it is corrected from the redundant data in that block. The odds of two disks having an URE in the same block of data is extremely low, but if that occurred then the file(s) that used that block would be corrupted.

If you have eight disks, and one of them is resilvering, then the system must read data from the other seven disks. The odds of a URE in a 4K block size on one disk is 4096 * 8 bits/byte * 10^-14 = 3.3 x 10^-10. The odds of having an URE in the same block on one of the other six disks is 6 * 3.3 x 10^-10. We need both those events to occur to have a problem, so the risk of data loss in a given block is 6 * 10^-19. If we have 8 TB disks, then we have 8 * 10^12 bytes / 4096 bytes/block = 1.95 * 10^9 blocks. Thus the risk of losing data is (1.95 * 10^9) * (6 * 10^-19) = 1.3 * 10^-9 if I am calculating properly.

Boeing's published airline accident data for jet airliners in North America from 2005 to 2014 showed 26 fatalities due to accidents in about 200 * 10^6 flights, or about 1.3 * 10^-7 per flight. Thus you are about 100 times as likely that your routine airline flight in North America has a fatal accident as losing data during a resolver with 8 x 8 TB disks in RAIDZ2.

 

[Dice]

PSU: Seasonic Platinum 460W (barely used on on hand)

I reread OP (but didnt reread all replies) -
Kind of returning to the 'scaling issue' if you are looking to add a second vdev to your zpool you will need to look for a bigger psu.
On top of that I'd like to give some props for thinking ahead a little bit on your system. In my view too few users take into account the longer time perspective (or do not make smart hardware choices).

/

 

[Jokener]

Okay, so RaidZ2 it is :)
When considering the choices, protecting myself against UREs and protecting myself against full disk failures got mixed up a bit.
Knowing that I do not risk loosing the pool from URE's if (all) parity drives are down, my fears of losing the pool to UREs were unfounded.
If FreeNAS should report an error in a block during this time, I can restore just the one affected file from my backup. Problem solved.
And being able to replace a failed disk quickly, two parity drives should suffice.

Summary:
I don't have a lot of cash available right now, a broken kitchen faucet destroying the kitchen cabinets and damaging the parquet flooring in the entire ground floor of the house does bad things to your reserves...
So I will have to get by with as minimal a hardware setup as reasonably (!) possible right now and expand on that later.

Therefore the Supermicro boards with integrated LSI controllers, while interesting, are just too expensive at 450€ when I can have the normal ones at 200 to 250€.
If I ever need more drives, an HBA card will be a good and viable option, so the "small" Supermicro board is a solid choice and expandable later.
-> Supermicro X11SSH-F

FreeNAS likes lots of RAM and using non-ECC is a non-starter with regular scrubbing activated.
To be on the safe side for now, I want to use 16GB. I also want a single module to allow for 2x or 4x 16GB later.
I can't find the Crucial or Hynix modules listed as compatible on the Supermicro site, I won't use Samsung products for reasons.
Crucial guarantees compatibility of some of their modules for all Supermicro X11 series boards.
I have (one) experience in dealing with their support and been impressed by their fast, friendly and professional service and replacement.
Buying directly from Crucial Germany costs me just 10€ more than the cheapest vendor online, so I will buy directly from them.
-> Crucial CT16G4WFD8213

In terms of the processor, it appears that there are no real concerns against using even the lowest powered Skylake Celerons.
On the other hand, a Pentium G4400 only costs 10€ more, so I will choose it over the Celeron G3900, even though I lack a solid reason.
If the box is running, I can test the performance of the G4400 with Plex and everything else and upgrade to the Xeon E3-1240 v5 if needed.
-> Intel Pentium G4400

Those decisions are made, unless some very good reasons convince me otherwise.
I will re-use my Seasonic PSU for the time being, and replace it if I ever go beyond the onboard SATA ports.
If I were to buy a new unit right now, I would go with one thats a bit more powerful.
But I'd rather spend the money on the components that I will most likely keep around for longer.
Getting a new PSU and therefore skimping on the RAM, processor or mainboard would be ill-advised.
-> Seasonic SS-460FL2

And since I am now contemplating the disk-situation from a RaidZ2 point of view, I will read and think some more on those.
I might even go with a RAIDZ2 of the 8TB Reds, but that kills the elegant upgrade path when using smaller drives.
Gotta pick someone up from the train station now, so I will have to finish this later tonight...

And once again, THANK YOU all for your input.

 

[Arwen]

...
Oh, and as for the backup, I will use an online backup service to back up all the data on the server.
Re-downloading everything after a potential local data loss might take a month, but a second server, off-site and far enough away to protect against flood or other catastrophes... it would be too expensive.
...

One thing you can consider for backups, is a Seagate 8TB SMR, (Shingled Magentic
Recording), drive. It's also called Archive. They are not as expensive as plain 8TB
drives, and certainly not as fast for writting.

About 1.5 years ago I bought one for my backups of my home FreeNAS. Since I had 4,
4TB drives in a RAID-Z2, 8TB can hold an entire backup without spanning disks. Seems
to work okay, if you can accept longer backup times. But, MUCH faster than Internet
services! And you control where it's stored, like office or safe deposit box.

In someways using a SMR drive for backups is the opposite case. The SMR drives tend
to be better at reads, than writes. With backups being write mostly, performance kinda
sucks. But, heh, it's cheaper and seems to work fine.

One last note. You can buy a water proof, impact resistant case and static bag for storage
of any external backup drive. I use a Seahorse, but others use Pelican.

 

[Jokener]

First I want to get up and running with the FreeNAS box.
When that is done, I will start using an online backup service, because they are cheap and don't prevent me from additional/other options.
Simply using another server in the same house doesn't seem to do me much good with regards to disaster-protection.
But when my house burns down, I really don't care about my movie-collection THAT much to warrant additional safeguards.
Which is really the only disaster with any significantly non-zero likelihood. I live far away from the sea, major rivers, mountains, tornado-zones and so on.
The insurance for my entire house against flooding and earthquakes costs only an additional 2,14€ per YEAR. I think they knoe why :)

I might use some sort of second local box plus online service in the future.
Right now the priority is to get onto ANY form of decent standard, but looking to make smart and scalable choices in the process.

And with regards to the components, I have made another choice.
I will use a Supermicro SuperDOM 16GB as the primary boot drive.
It only costs 35€ for the 16GB SuperDOM and can sit right on the X11SSH-F board.
As a first backup, I will use the SanDisk Cruzer Ultra 32GB on the internal USB port.
And I will look into using the USB port on my router (AVM Fritz!Box 7490) to regularly back up the boot drive to a second SanDisk Cruzer Ultra.

Now the only remaining choice to be made is the size of the drives and the number of them to use...
I am not quite ready yet to make the call on that, because I still have to think more about my need for incrementally larger storage space over time.
As I will use a SuperDOM as the primary boot drive, I will always have the option to disconnect it later.
I can then use the Sata port to migrate a drive to a larger one while booting from the USB backup drive and move back to the SuperDOM after being done.
When I use only 4 or 5 of the larger drives, growing the pool would mean scrapping the original one instead of in-place expansion.
Whlie that IS of course possible, something in me wants to avoid it more than I can reasonably explain... so I'll still mark it as a negative argument.
Possible configurations would be (numer after the "~"-sign being usable space in RaidZ2 rounded to 0,1TB):
6x 2TB = ~6,3TB @ 510€ - barely enough space today after deleting useless junk, very close for even the 12-month-scenario. <- No
7x 2TB = ~7,9TB @ 595€ - expensive, better, SuperDOM acrobatics when upgrading, potential to grow very large <- No
5x 3TB = ~7,1TB @ 315€ - cheap, have 2 of these drives and could use them, limited growth potential <- Unlikely
6x 3TB = ~9,4TB @ 420€ - moderate price, but a lot more space, 2 drives on hand, better growth potential <- Maybe
7x 3TB = ~11,8TB @ 525€ - expensive, SuperDOM acrobatics, 2 drives on hand, potential to grow very large <- Maybe
4x 4TB = ~6,3TB @ 580€ - barely sufficient today, same price as 7x2, 4TB drives limit growth potential without scrapping the pool <- No
5x 4TB = ~9,4TB @ 725€ - same space as 6x3 at higher price, growth potential limited/burdensome <- No
4x 5TB = ~7,9TB @ 800€ - 6x3 still looking better, even less space than 5x4 <- No
4x 6TB = ~9,4TB @ 960€ - very expensive, 3x6 absolutely unfeasable, growth potential limited/burdensome <- No
3x 8TB = ~6,3TB @ 960€ - very expensive, could add second 3x8 pool without scrapping the first (but overhead), no selling of disks later <-Maybe

As I said, I am still a bit undecided. The 3x8TB would be expensive now and complicated to upgrade. I could immediately sell the two 3TB Reds for ~70€ each.
But 8TB are the top of what is currently out there, 6x8 will last me a long time and I would not have to sell 5/6/7 smaller drives after upgrading.
On the other hand, any losses from the depreciation in value of the 3TB drives are offset by the 2 drives I already have AND the expected price decrease of the 8TB Reds in the future.
Life... full of difficult decisions.

 

[Bidule0hm]

3x 8TB = ~6,3TB @ 960€ - very expensive, could add second 3x8 pool without scrapping the first (but overhead), no selling of disks later <-Maybe

You can't do RAID-Z2 with less than 4 drives.

Can you add the resulting price per TB, because the spaces aren't equal so comparing the total prices isn't very useful.

 

[jgreco]

You can't do RAID-Z2 with less than 4 drives.

Sure you can, we just use a different term... "three way mirror"

 

[Bidule0hm]

Then that's not a RAID-Z2, it's a three way mirror :D

 

[jgreco]

Then that's not a RAID-Z2, it's a three way mirror :D

That's a matter of terminology, not capability.

 

[Dice]

Good. Some projective analysis going on.

7x 3TB = ~11,8TB @ 525€ - expensive, SuperDOM acrobatics, 2 drives on hand, potential to grow very large <- Maybe

It is an important consideration to tkae into account how much the "next upgrade" will cost and what it will benefit.
Did some of those calculations myself, which put in me in the 7drive per vdev segment. Today, a couple of months later I'd probably would've aimed for 8 if I could've mustered the budget. (Budget was utterly annahilated once I realized the requirements...)
The reason is the neatness of upgrading using a HBA which commonly has capacity for 8 drives rather than 7 ;)
However, you do have a point - it is neat to have surplus of at least one SATA for replacing 'risky drives'. I've surplus of ports that makes this a non-issue.

Cheers

 

[Jokener]

Sorry, I didn't think about the fact that one data drive plus two parity drives results in both parity drives holding the same information as the data drive...
But that would be even better for me, because I could just remove one drive from the three way mirror, scrap it and place it in the new pool before migrating the data.
Thats the advantage of going with 8TB drives right away, eliminating the selling of the in-between drives needed for all other vdev combinations.
Going with the 3TB Reds would allow me to integrate the two drives I have, thus effectively reducing the need to buy capacity right now.

I will re-post the four viable options for the drives and add the respective cost per TB at the bottom of this post.
The pure cost per TB is almsost identical for all WD Reds, at least when purchased from official WD dealers in Germany. (see attached screenshot)
4TB, 5TB and 6TB drives are just not as attractive, because they reach the capacity I can afford right now with few drives.
That is good from a power consumption point of view, but suboptimal with regards to in-place expansion of the vdev.
I could only realistically go that route by using the Supermicro X11SSL-CF, because it has 8 SAS ports from an LSI controller on board.
The motherboard costs only 40€ more than the X11SSH-F and its only significant limitation is the C232 chipset with its 6 Sata ports.
But 30€ here for the SuperDOM, 10€ there for Crucial RAM with compatibility guarantee... it adds up and the only place I can really save money are disks.
For that reason I was already contemplating the WD Blue drives. Since I will only use them for 6 to 12 months I should be fine, even with the 2 year warranty.
The price difference is quite large, with 3TB Blues going for 80€ and 3TB Reds for 105€.

Now to the cost per TB analysis:
5x 3TB = ~7,1TB @ 315/525€ -> ~44,5 or 74,0 €/TB - cheap, have 2 of these drives and could use them, limited growth potential <- Unlikely
6x 3TB = ~9,4TB @ 420/630€ -> ~44,5 or 67,0 €/TB - moderate price, but a lot more space, 2 drives on hand, better growth potential <- Maybe
7x 3TB = ~11,8TB @ 525/735€ -> ~44,5 or 62,3 €/TB - expensive, SuperDOM acrobatics, 2 drives on hand, potential to grow very large <- Maybe
3x 8TB = ~6,3TB @ 820/960€ -> ~130,2 or 152,4€/TB - very expensive, could add second 3x8 pool without scrapping the first (but overhead), no selling of disks later <-Maybe
(The two prices per configuration are with and without using the two existing 3TB Reds. For the 3TB configs I could use them, with the 8TB config I could sell them to offset the price.)

This analysis speaks very clearly in favor of the 3TB drives right now and that is the way I am leaning as well.
What is being obfuscated through this method is the advantage of not having to sell used disks later.
The 8TB Reds cost 320€ a piece (40€/TB), so I would be spending two thirds of the money on parity due to the low number of large disks.
With the 3TB drives and when using my two existing drives, I am effectively "donating" both parity drives, so the cost per TB for all options is identical in this case.
And for the scenario that ignores the two existing 3TB Reds, the cost per usable TB goes down with the number of disks, because the portion used for parity is lower in the large pool.

Now to the part that will cause some people here to think about tarring and feathering me, the WD Blues:
6x 3TB = ~9,4TB @ 370/510€ -> ~40,0 or 54,3€/TB - cheap, only two year warranty, good growth potential, lots of space right now <- Looks tempting, but unlikely

As of right now I am leaning more and more towards the 6x 3TB WD Red option. It would cost me 420€ in disks, which I could just afford.
The 7x 3TB option would be stretching the budget, but I COULD manage it if I had to. Spending 820/960€ on disks is not possible at the moment.

 

[Bidule0hm]

That's a matter of terminology, not capability.

According to this thread you should use the proper terminology... :)

As of right now I am leaning more and more towards the 6x 3TB WD Red option.

This is what I recommend ;)

 

[anodos]

According to this thread you should use the proper terminology... :)

It's a zraid 3 mirror.

 

[jgreco]

According to this thread you should use the proper terminology... :)

Obviously, but the point is that the *capability* exists.

 

[Robert Trevellyan]

If we're having a pedantry-party, let me be the one to state that you can absolutely do RAIDZ2 with 3 drives. You can do RAIDZn with n+1 drives, for n=1-3. You have to use the CLI, because it's silly, but ZFS is happy to do it.

 

[NetSoerfer]

Disclaimer: I'm in a similar situation as Jokener, with 8TB of data partially backed up, and with similar requirements for functionality, size, and expandability. I've looked into FreeNAS for a while, and I've been going in circles thinking about the number of drives for the VDEV as well.

Now I'm not saying the below is the best solution or even just entirely correct, since my FreeNAS 'knowledge' is purely theoretical, and sketchy at that.

There may be another approach though, and I'm curious as to why this hasn't been mentioned at all in this thread. If this makes no sense for you, I'm happy to move this discussion elsewhere as I don't mean to hijack your thread, but maybe it's worth thinking about.

You're looking to build a Raid-Z2, possibly a Raid-Z3, to alleviate the risk of subsequent failures while resilvering that increases with volume size.

Have you at all considered mirroring instead of Raid-Z? With Raid-Z2, you 'lose' 2 drives (anywhere from 67% to 25% of your drives), with Raid-Z3, you 'lose' anywhere from 75% to 38% of your drives, not considering decimal-binary conversion and overhead. And as the volume grows bigger, the greater the need for Raid-Z3 (however great that need may ultimately be).

To make things worse, you cannot add drives to a VDEV, so your volume size is set in stone until you replace all drives - worst case, you're looking at more than a month of heavy strain on your disks, which at that point may already be getting a bit long in the tooth. Remembering a disk failure on my 3+1 disk Synology, that's a long time of sleeping badly.

With a 1-drive mirror, you'll 'lose' 50% of your disks which may be more than with a Raid-Z, but you have easy expandability (simply add a 2-drive VDEV to the volume), you can create separate volumes with different levels of redundancy (2-drive mirror for critical data?), and replace one volume without affecting the others. Most importantly though, if a drive fails, the other (old) drive only needs to be read once for resilvering, which is faster by an order of magnitude, greatly decreasing the risk of a drive failure while resilvering.

Again, I'm very, very far from being an expert on this, but applying what limited knowledge I have about ZFS, it seems like this would be a feasible alternative.

 

[Jokener]

Of course I have no problem with your comment here, since your arguments are exactly on the points I was pondering.

Unless unexpected things happen, I have now made the decision to go with a 6x 3TB setup.
In a RaidZ2 configuration that leaves me with 9,4TB right now and more than enough space so the capacity increase doesn't have to happen mere months after the box goes online.

There are multiple reasons for this configuration.
The strain on the remaining drives during a resilvering is only then great, if you are replacing a failed drive and thus don't have acces to the data on the outgoing drive.
My plan (knock on wood) is to not replace a failed drive, but a perfectly functional one.
And since I have a free SATA port with the 6x3 configuration plus 1x SuperDOM, the process is quite easy.
FreeNAS has all the tools on board to replace a functional drive.
I will just connect the new one to the free port, migrate the data from the outgoing to the fresh drive and then disconnect the old drive.
Since FreeNAS can access both drives at the same time, the rest of the pool is not stressed during a 1:1 cloning of the data.

Another advantage of that upgrade strategy is that I can buy one drive every two weeks or so.
That way any production defects can't hit all of my drives, as they might with one bulk order.
This has already happened in the past, it is not unheard of.

Any performance benefits of a mirrored configuration are lost on me, because I have so few users and most requests involve large sequential transfers/streams of multimedia files.

And aside from that, I simply feel the reliability and size arguments speak for RaidZ2 (or 3).
Again, this is with my use case in mind, YMMV.

Once I have built the FreeNAS server and transferred my data onto it, I am already a lot more protected than right now.
The next step is to blast my upload speed for a few months and create a Backblaze or Crashplan backup of my data.
With THAT I will be protected a LOT better.
And when I still feel anxious, I might build a second really slow box to create another backup.
For now, I will take things one step at a time.

 

[danb35]

To make things worse, you cannot add drives to a VDEV, so your volume size is set in stone until you replace all drives

Although much of your analysis is correct, this misses the fact that you can add vdevs to a pool to expand it. I'll use myself as a real-world example.

My pool currently consists of twelve disks, in two six-disk RAIDZ2 vdevs. The first vdev consists of three 2 TB disks and three 3 TB disks; the second vdev consists of six 4 TB disks. I'm running out of space--my pool is already over 80% full. I have to options to expand it: (1) replace drives with larger drives, or (2) add a third vdev. For (1), the most obvious thing to do would be to replace the three 2 TB disks with three 3 TB disks, which would expand the first vdev from 8 TB of net capacity to 12 TB (and would allow full use of the 3 TB disks that are already there). As you note, that involves a resilver for each replacement, though my system has enough room that I can do that without taking the old disk offline first and compromising redundancy. However, that still has me throwing away three perfectly good disks. Those are the oldest disks in my system, and as they (start to) fail, I'll probably replace them with 3 TB disks, but that's not my short-term plan.

The second option is to simply add a third vdev, which is what I'm going to do. Once I have six 6 TB disks that have satisfactorily finished testing (which is harder than I'd expected--four of the six white-label drives I bought are showing bad sectors, though the vendor isn't giving me any grief so far about replacing them), I'll add them to the pool as a third six-disk RAIDZ2 vdev, which will expand the pool capacity by about 24 TB. This arrangement does carry some risk--if any of my three vdevs completely fails, my pool is toast.

 

[Jokener]

Do you need all of your space to be in one single pool?
Or is there another reason you don't create a new pool for every vdev?

As I was thinking about future upgrades, one of the options was/is an HBA for more drives and then a new vdev.
But while I knew it was possible, I never considered the option of a multi-vdev-pool.
I somehow always chalked that up for the "metric shit ton"-of-drives scenario where the recommended number of drives per vdev would be exceeded.