Scalable hardware setup

[Jokener]

Hello everybody,

I am looking to build a FreeNAS server on a budget, but would like to scale it up later.
I studied all of the guides and tutorials, but they didn't contain a very clear path or recommendation for a comparatively cheap but scalable system.

The current situation is that I have approximately 7 TB of data, with almost none of it properly backed up.
Of the 7 TB there is a bit of data, less than 2 GB, that is almost irreplaceable. This is properly backed up.
Then there are 200GB of somewhat important files and the rest is not absolutely critical (I can re-download my 250 GB iTunes library and re-rip my movies from the discs if everything goes to hell).
But if I wanted it gone I'd have deleted the stuff, so now I want to store it properly and implement a working backup scheme.

The purpose of the server right now is to hold my data (obviously), serve up some movies throughout the house and ideally allow remote access to some of the files (documents, mostly).
I won't need any sort of serious IO performance right now, but would like to have an upgrade path to allow multimedia transcoding and at least twice the storage capacity later.

My preliminary plan for the hardware is as follows, with the reasoning in the parentheses.

Motherboard: SuperMicro X11SAE-M (S1151, 8x Sata onboard, mATX, affordable)
Processor: Intel Celeron G3900 (very cheap, ECC capable)
Memory: 1x 16GB DDR4 ECC-memory (ECC is important, want to go 2x16GB later)
PSU: Seasonic Platinum 460W (barely used on on hand)
Drives: 7x WD Red 3TB (already have 2, good drives, can be replaced with larger drives)
USB-Drives: 2x SanDisk Cruzer Fit Ultra 16GB or Corsair Voyager Vega 16GB (compact, fast, cheap, good manufacturer)
Case: Custom built wall-mounted case, alternatively a heavily modified Bitfenix Prodigy M

Does this processor have enough power to "just store and transfer some files"? Or is something stronger required as the baseline?
Sometime soon-ish I would replace it with an Intel Xeon E3-1245 v5, or something similar...
The Xeon should then be enough to handle everything I might think about throwing at it.

I want to use the 3TB WD Reds because this would allow me to replace them one by one with 6TB or larger drives.
The current plan is to run them in RaidZ3, not because the 7x 3TB drives necessarily require me to, but because a RaidZ2 configuration seems sketchy with the possible 7x 8TB or even 7x 10TB drives I would purchase in 6 to 12 months.
After the parity, 80% of 12TB still leaves me with 9,6TB usable space, more than enough right now.

When the FreeNAS is up and running, I will book an online backup service like Crashplan (still undecided which one).

Are there any critical errors in my plan?
Or would you go a different route with regards to either the pool-drives or the other hardware ?

Thanks for any input!

 

[anodos]

I don't really see any reason why raidz2 would be sketchy.

Pentium might be better than the celeron because you might find out that it's enough CPU for your needs (assuming you're not transcoding).

With custom cases in particular and NAS devices in general, make sure you're getting adequate airflow over your drives. You can monitor the temps using smartctl. Make sure you don't cook your drives.

Your PSU is kinda close the the bare minimum that I'd consider for those specs.

 

[Jokener]

With sketchy I don't mean the concept of RAIDZ2 itself, I mean the statistics and risks of disk failures and read errors.
If I were to use larger drives in the future, such as 8TB WD Reds or maybe even 10TB Reds (if they are launched by then), I would have a bad feeling regarding additional drive failures during a 7-disk or 8-disk RAIDZ2 rebuild.
I could immediately swap out any drives that are reported as defective, since the server is going to stand in my basement.
With a setup of 8TB drives however, I might be looking at some scary rebuild times with a high continuous load on the drives.
The rebuild duration itself and the degraded performance of the array during that time would be of little concern to me.
But when all the drives are the same age and a worst case of 6,xTB of data needs to be restored to a replaced disk...
Then there is a significant non-zero chance of another drive failure, leaving the pool unprotected against a third drive failure or read errors.
I have played a bit with the reliability calculators regarding the failures and their likelihoods, but have not come away sufficiently convinced RAIDZ2 would be adequate.
On the other hand, this may very well be a counter-reaction to knowing my files are not protected at all right now, without the FreeNAS server...
I have to admit, the Z2/Z3 decision is the one I can back up with the least amount of facts. On the other hand, the only downside is the capacity loss, right?
(At least for my use case. I am aware of multi-user and very high-performance scenarios with different requirements. None of the scenarios I am aware of apply to me.)

Regarding the CPU, I could choose the Pentium G4400, if someone were to tell me it makes a real difference.
Just from looking at the clock speeds and data sheets though, how great could the Pentium advantage be?
If I am just sacrificing some potential performance in terms of reads/writes over the network, I'd choose the Celeron for now.
Plex transcoding is very interesting to me in the long run, so I will have to upgrade to a Xeon anyway.
If system stability or data integrity were negatively affected by the Celeron, I would go with a Pentium or Core i3 for now.

The cooling performance will be more than adequate.
All drives will be in water-cooled enclosures and sharing a loop with the CPU.
That way CPU temperature will warn me of any (potential) cooling problems, such as pump failures.
The pump includes an onboard logic that can issue alerts as well, without any CPU overheating issues at all.
I have water-cooled all my desktop computers for 10 years and never once had ANY type of problem.
Also, the power consumption of a water cooling setup is significantly lower than air cooling, since it requires no fans.
(Fans exist, but won't run with the expected water temperatures during idle or low loads.)
As an automotive engineer the cooling system is right up my alley.

And you are really worried about the PSU?
It can provide 100W for the 3.3V and 5V voltages and 456W for 12V.
If anything, it seems overkill.

But I may be wrong, since I don't feel quite as confident in my server knowledge as I do feel in my ability to build and configure gaming rigs.
Anyway, thanks for your input. It really helps me...

 

[anodos]

With sketchy I don't mean the concept of RAIDZ2 itself, I mean the statistics and risks of disk failures and read errors.
If I were to use larger drives in the future, such as 8TB WD Reds or maybe even 10TB Reds (if they are launched by then), I would have a bad feeling regarding additional drive failures during a 7-disk or 8-disk RAIDZ2 rebuild.
I could immediately swap out any drives that are reported as defective, since the server is going to stand in my basement.
With a setup of 8TB drives however, I might be looking at some scary rebuild times with a high continuous load on the drives.
The rebuild duration itself and the degraded performance of the array during that time would be of little concern to me.
But when all the drives are the same age and a worst case of 6,xTB of data needs to be restored to a replaced disk...
Then there is a significant non-zero chance of another drive failure, leaving the pool unprotected against a third drive failure or read errors.
I have played a bit with the reliability calculators regarding the failures and their likelihoods, but have not come away sufficiently convinced RAIDZ2 would be adequate.
On the other hand, this may very well be a counter-reaction to knowing my files are not protected at all right now, without the FreeNAS server...
I have to admit, the Z2/Z3 decision is the one I can back up with the least amount of facts. On the other hand, the only downside is the capacity loss, right?
(At least for my use case. I am aware of multi-user and very high-performance scenarios with different requirements. None of the scenarios I am aware of apply to me.)

I have a sinking suspicion that an event that takes out a RAIDZ2 vdev is likely to take out a RAIDZ3 vdev as well. Note that I'm not trying to dissuade you from selecting RAIDZ3. If you're okay with using an extra drive for parity, that's fine. Just be sure that you also are backing up your data because no amount of redundancy can replace having proper backups.

Regarding the CPU, I could choose the Pentium G4400, if someone were to tell me it makes a real difference.
Just from looking at the clock speeds and data sheets though, how great could the Pentium advantage be?
If I am just sacrificing some potential performance in terms of reads/writes over the network, I'd choose the Celeron for now.
Plex transcoding is very interesting to me in the long run, so I will have to upgrade to a Xeon anyway.
If system stability or data integrity were negatively affected by the Celeron, I would go with a Pentium or Core i3 for now.

If you think you'll only have a couple plex streams going simultaneously, an i3 is probably sufficient for your needs. Some services (such as samba) are affected by the clock speed of the CPU. You probably won't notice this on large sequential transfers (saturating gigabit is quite easy with that), but if you transfer thousands of 16KB files your computer will probably benefit from having a higher clock speed (samba is single-threaded per data stream).

The cooling performance will be more than adequate.
All drives will be in water-cooled enclosures and sharing a loop with the CPU.
That way CPU temperature will warn me of any (potential) cooling problems, such as pump failures.
The pump includes an onboard logic that can issue alerts as well, without any CPU overheating issues at all.
I have water-cooled all my desktop computers for 10 years and never once had ANY type of problem.
Also, the power consumption of a water cooling setup is significantly lower than air cooling, since it requires no fans.
(Fans exist, but won't run with the expected water temperatures during idle or low loads.)
As an automotive engineer the cooling system is right up my alley.

Honestly, I don't think I've heard of anyone water cooling their freenas server. You'll still want to configure SMART monitoring on the hard drives so that they'll send an email alert if they go above what is deemed a safe threshold (this is all baked into the webgui).

And you are really worried about the PSU?
It can provide 100W for the 3.3V and 5V voltages and 456W for 12V.
If anything, it seems overkill.

Per jgreco's rough estimates, you're in the 550W range. https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

But I may be wrong, since I don't feel quite as confident in my server knowledge as I do feel in my ability to build and configure gaming rigs.
Anyway, thanks for your input. It really helps me...

Servers are a different sort of thing. You're trying to optimize for stability, reliability, redundancy, and ease of mainenance.

 

[anodos]

You should also look at a server motherboard like the X11SSH-F http://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSH-F.cfm
The X11SAE-M is a workstation board and contains crap you don't need. The IPMI included in the X11SSH-F (or similar server motherboard) is worth its weight in gold. IPMI makes maintenance and debugging much easier. For example, see here: https://forums.freenas.org/index.php?threads/good-news-everyone-haswell-i3s-do-work-with-ecc.43113/

 

[Dice]

I agree with what's being said by anodos.

After the parity, 80% of 12TB still leaves me with 9,6TB usable space, more than enough right now.

That is on RaidzZ2. RaidZ3 will be closer to 9.5TB.

This is the space calculator the forum lives n dies by:
https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

 

[Jokener]

@Dice: Thanks for the link, that is the calculator I used :)
If I use 7x 3TB drives in Z3, three drives are used for parity.
Of the remaining 4x3=12 TB, I can use 78.7% according to the calculator. -> 9,446 TB
I just rounded that to 80%/9,6TB as a rule of thumb for the configuration process.

And of course I will use all SMART features that FreeNAS supports on the drives.
The cooling solution is done primarily for reliability and silence.
A bedroom door is located just 2 meters from the server location, it has to be dead silent.
And saving the power for all those fans is also worth it, I believe.

Western Digital spec sheets list the absolute peak current as 1,73A for the 3TB drives (and 1,79A for 8TB).
Assuming they know their own drives, peak power consumption for the drives would be 7x1,73Ax12V=145,32W.
I am certain it will be a cold day in hell if this configuration ever manages to draw above 300W.
If I were buying a new PSU, I might choose the semi-passive 660W over the fully passive 460W just to be extra extra safe. But for now the one I have should be fine.
Note the 460W Seasonic design is a derivative of a larger unit and has plenty of reserves to exceed even the 460W rating.

With regards to the board, I did want to use one with IPMI.
It seems the price comparison website I used has falsely included the SAE-M in the IPMI selection.
So I will change the board to the SSH-F. That is also a mATX board and has all the important features of the SAE-M.
DisplayPort/HDMI will be missed, but IPMI is a worthwhile replacement...

Oh, and as for the backup, I will use an online backup service to back up all the data on the server.
Re-downloading everything after a potential local data loss might take a month, but a second server, off-site and far enough away to protect against flood or other catastrophes... it would be too expensive.

Thank you guys again for your input, it is much appreciated.

 

[Ericloewe]

I have a sinking suspicion that an event that takes out a RAIDZ2 vdev is likely to take out a RAIDZ3 vdev as well.

Almost certainly.

 

[JDCynical]

DisplayPort/HDMI will be missed, but IPMI is a worthwhile replacement

This.

HDMI/Displayport outputs are irrelevant with the IMPI/KVM on the board. In the many years I've had my three servers with the IPMI/KVM running, I've never had to attach a display to any of them to perform any non-hardware maintenance.

And for those times, I have a small LCD panel I can plug into the VGA port becasue I'm lazy and didn't want to walk back to the room with the workstation vs seeing the results while I'm at the machine in case the hardware has some sort of problem.

 

[Kevin Horton]

If you are the kind of person who doesn't ever go in an automobile or an airliner because the risk of dying is too high, then RAIDZ3 is for you. But if you are willing to do those things, understand that you probably have a higher risk of dying in a car accident or aircraft crash than losing data in RAIDZ2. Is your data worth more than your life?

 

[Jokener]

I'm the kind of person that designs, builds and operates formula race cars and boxing is a hobby of mine.
As far as I can recall, nobody has ever called me affraid to take risks

I may be misunderstanding some elementary basics regarding the reliability here.
My fear stems from the relatively high chance of an unrecoverable read error of the 8TB WD Red drives.
The chance for UREs in the Reds is 1 in 10^14 bits. When reading a completely full 8TB drive, this is a chance of 8% if my math is right.
And even for the 3TB drives I am planning to use for now, 3% is still dangerously far away from 0%...
It is my understanding that ZFS will detect any UREs through checksums and use the parity drives to check if any drives can deliver the correct data.
It is also my understanding that ZFS being unable to recover any UREs will result in me loosing the entire pool with all the data.
If that is not the case and I were to only loose that one affected file or bit, I would agree with you on RAIDZ2 being sufficient.
Otherwise, I would definately go with RAIDZ3 since a second failure during a weeklong rebuild seems not unlikely enough...

 

[Dice]

There are some good reads on this topic, some charts that plot raidz3 vs raidz2 likeliyhood of loosing data around here somewhere. I could've sworn I had them bookmarked. Just nowhere to be found.
Ie, without being able to provide you with a sound reasoning based off numbers from on top of my head, I'll at least return that what you are saying does not ring completely true to the freenas scenario (some of your logic would be better applied to hardware raid controllers work), granted that scrubs are done regularly.

The the forums is pointing yet towards raidz2 for home use, with less than 8 drives per vdev, as of 6TB drives. But - evidently it will change towards raidz3 becoming the new standard, question is when.

It might be helpful to consider migration strategies. For example moving from 8x4tb raidz2 to add another vdev of 8x10tb drives into the same zpool, users like you might not feel comfortable remaining at raidz2.
At that point you could create a new raidz3 zpool using the new drives, then copying the data from the old drives to the new zpool (for best practice, there is additional readings to do). Then destroy your old raidz2 completely and add the drives in raidz3 configuration to the newly created raidz3/zpool. This sort of migration pattern would work as long as the new drives are sufficiently large to hold the data of the previous pool.
Of course, if you do not need the space right now and would like to avoid this hazzle, it is cool to get raidz3 now.
But getting it for "URE calculations"-reasons is not really a logic that the forum acknowledges at this point in time. Raidz3 is usually motivated for boxes where it is more difficult to get to the servers in time to do a resilvering/replacing drive.

Cheers /

 

[Jokener]

Then I will do some more research on the forum regarding the reliability and URE-probabilities for RAIDZ2 and Z3.
Previous searches didn't really satisfy me, but I have not searched long enough or read enough posts either.

I have read about not using more than 8 drives per vdev (and also that some people have no problem with more than 8 drives).
That isn't of such a great concern to me, because there is simply no hardware solution that fits my needs and even allows for more than 8 drives.
I am indeed tempted to use 8 drives instead of 7, but am contemplating the consequences.
If I limit myself to 7 drives, upgrading them becomes easier down the road.
I can just add the new large drive to port 8 and transfer the data from a working drive to the new one, then disconnect the old drive.
That way I could very easily replace one drive per day, let the FreeNAS work its magic and be done after a week.
Using 8 drives would mean disconnecting one, adding an empty large drive and then resilvering for at least three to five days, possibly longer.
AND it places a much higher load on the other drives, which should be acceptable but is not necessarily desirable.

Its a typical scenario of mutually exclusive goals.
I shall look into USB-SATA-Controllers and if it is possible to connect en empty 9th drive via USB for the transfer and the move it to the SATA port of the drive it is supposed to replace.
That way I could have my cake and eat it, but I would be satisfied with 7 drives as well.

 

[anodos]

I'm the kind of person that designs, builds and operates formula race cars and boxing is a hobby of mine.
As far as I can recall, nobody has ever called me affraid to take risks :)

I may be misunderstanding some elementary basics regarding the reliability here.
My fear stems from the relatively high chance of an unrecoverable read error of the 8TB WD Red drives.
The chance for UREs in the Reds is 1 in 10^14 bits. When reading a completely full 8TB drive, this is a chance of 8% if my math is right.
And even for the 3TB drives I am planning to use for now, 3% is still dangerously far away from 0%...
It is my understanding that ZFS will detect any UREs through checksums and use the parity drives to check if any drives can deliver the correct data.
It is also my understanding that ZFS being unable to recover any UREs will result in me loosing the entire pool with all the data.
If that is not the case and I were to only loose that one affected file or bit, I would agree with you on RAIDZ2 being sufficient.
Otherwise, I would definately go with RAIDZ3 since a second failure during a weeklong rebuild seems not unlikely enough...

A URE will only result in data loss if there is no available parity information from which to recover. That is why a degraded RAIDZ1/RAID5 array is susceptible to data loss / corruption from these events. The extra parity information available in a RAIDZ2 vdev protects against this type of failure. In principle, the longer it takes to resilver your zpool after replacing a drive, the more likely you are to experience additional drive failure(s). In practice, it's complicated. I suspect that an array with new-ish drives will probably not experience multiple simultaneous hard drive failures without some other underlying cause (like out-of-control temps, vibration, elbonians licking them). At the beginning you're experiencing infant mortality. I have a server with an 8-drive raidz2 zpool. One drive failed after 6 months. I replaced it, and the array has been chugging along for almost two years without issue. On the other hand, if you have an array full of 7 year old drives (same make, model, sequential serial numbers) and one fails, you should start getting your backups ready. :D

 

[anodos]

Then I will do some more research on the forum regarding the reliability and URE-probabilities for RAIDZ2 and Z3.
Previous searches didn't really satisfy me, but I have not searched long enough or read enough posts either.

I have read about not using more than 8 drives per vdev (and also that some people have no problem with more than 8 drives).
That isn't of such a great concern to me, because there is simply no hardware solution that fits my needs and even allows for more than 8 drives.
I am indeed tempted to use 8 drives instead of 7, but am contemplating the consequences.
If I limit myself to 7 drives, upgrading them becomes easier down the road.
I can just add the new large drive to port 8 and transfer the data from a working drive to the new one, then disconnect the old drive.
That way I could very easily replace one drive per day, let the FreeNAS work its magic and be done after a week.
Using 8 drives would mean disconnecting one, adding an empty large drive and then resilvering for at least three to five days, possibly longer.
AND it places a much higher load on the other drives, which should be acceptable but is not necessarily desirable.

Its a typical scenario of mutually exclusive goals.
I shall look into USB-SATA-Controllers and if it is possible to connect en empty 9th drive via USB for the transfer and the move it to the SATA port of the drive it is supposed to replace.
That way I could have my cake and eat it, but I would be satisfied with 7 drives as well.

If you want 8+ drives you should look at getting an HBA such as the LSI 9211-8i. Refer to the hardware recommendations sticky here: https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/

 

[Jokener]

I know ZFS can correct UREs with working parity drives.
So RaidZ1 with one dead drive can not recover a URE and RaidZ2 with two dead drives cannot recover a URE.
But what is the result of experiencing any UREs while the parity drives are down? Data loss on the one bit/block/file or the loss of the pool?
If its just the file, I would go with RaidZ2 and not worry. If the risk is to loose the pool, then RaidZ3 it is.
Even with a backup, downloading 7TB of data from a sufficiently cheap service like Crashplan or Backblaze would take AT LEAST 1 month and possibly three months, depending on their transfer speed.
It is better to wait three months than to loose everything, but it is even better to pay for a backup you never need to use...

I feel more than 8 drives are really not a prudent decision as long as I haven't gone to 8TB drives.
At least considering my use case, I can see other scenarios where it would be.
But the 7 or 8 drive questions IS bugging me, just because I want to maximize the utility and data integrity at the same time.

I am running in circles and will read some more before making the final decision.
But at least I know I'm not totally off-base with most of my assumptions and tendencies.

And once more, thank you for the input.

 

[anodos]

I feel more than 8 drives are really not a prudent decision as long as I haven't gone to 8TB drives.
At least considering my use case, I can see other scenarios where it would be.
But the 7 or 8 drive questions IS bugging me, just because I want to maximize the utility and data integrity at the same time.

Well, the solution is obvious: get 12 drives and create a zpool with two 6-disk RAIDz2 vdevs.

 

[anodos]

I know ZFS can correct UREs with working parity drives.

Parity information is spread across drives. So as long as it's able to reconstruct from available parity, you're golden.

But what is the result of experiencing any UREs while the parity drives are down? Data loss on the one bit/block/file or the loss of the pool?

If the URE trashes your ZFS metadata you'll be in trouble. Otherwise you're probably fine.

Even with a backup, downloading 7TB of data from a sufficiently cheap service like Crashplan or Backblaze would take AT LEAST 1 month and possibly three months, depending on their transfer speed.
It is better to wait three months than to loose everything, but it is even better to pay for a backup you never need to use...

That's why cloud backups suck. The solution is a better backup strategy. :D

I feel more than 8 drives are really not a prudent decision as long as I haven't gone to 8TB drives.

I probably would not go far beyond 8 drives with a RAIDZ2 vdev.

Maybe if we're lucky @jgreco or @cyberjock will weigh in on these issues.

 

[danb35]

If the URE trashes your ZFS metadata you'll be in trouble.

But since the ZFS metadata is stored at least twice (and up to six times for the really critical stuff), having a URE trash all of it is fairly unlikely.

 

[jgreco]

Parity information is spread across drives. So as long as it's able to reconstruct from available parity, you're golden.

If the URE trashes your ZFS metadata you'll be in trouble. Otherwise you're probably fine.

That's why cloud backups suck. The solution is a better backup strategy. :D

I probably would not go far beyond 8 drives with a RAIDZ2 vdev.

Maybe if we're lucky @jgreco or @cyberjock will weigh in on these issues.

But since the ZFS metadata is stored at least twice (and up to six times for the really critical stuff), having a URE trash all of it is fairly unlikely.

As long as one copy of all metadata is recoverable (assuming that's what you mean by "trash all of it") you should be okay. Once metadata becomes unavailable, the state of the pool is no longer assured to be in a good condition, and you really need to destroy and rebuild the pool from scratch.

There's definitely a tradeoff between various factors. For example, how aggressively do you actually take care of the system? If a drive dies, are you right on top of that with a replacement? Do you keep one on hand? (If so, and you're doing RAIDZ2, why aren't you doing RAIDZ3?) Has your hardware been burned in, and is it therefore more likely to be trustable? How big are the drives and how likely an additional failure during rebuild?

The archival filer here is an 11 drive RAIDZ3 with a spare (12 drives total) and a cold spare sitting on the shelf. It is non-problematic and is not expected to become problematic. As drives die, the system will automatically put in the spare and rebuild. At 11 drives wide (8 data), that's probably about as wide as you really want to get. The only scary thing might be if I started to see rapid succession disk failures, maybe due to overheating, a bad PSU, or something like that.