Hello! Sanity check my first build?

[cship]

Hey everyone, finally made an account today but these forums have been a great resource while I was researching this build. I'm getting ready to pull the trigger on my first "real" NAS and wanted to make sure that there aren't any issues with my component selection in terms of compatibility or performance.

The machine this build is replacing is currently acting as a single-disk 4TB media server that hosts Plex, Sonarr, Couchpotato, SABnzbd, and qbittorrent as well as general network file storage. Unfortunately I've filled it up and it's 7 years old now, so I've decided to start fresh. I'd also like enough overhead to run some Dockerized containers for lightweight home automation/intranet webdev type usage.

Onto the specs:

Storage Disks: 4x Seagate IronWolf 8TB, 4x WD Red 8TB - I think I want to set this up as a RAIDZ2 array
Boot Disks: 2x SanDisk Ultra Fit 32GB mirrored
Mobo: Supermicro MBD-X10SRI-F
CPU: Xeon E5-2630v4 (10 core HT, 2.2GHz) w/ Noctua i4 cooler
Memory: 1x 32GB Samsung DDR4-2133 - I plan to pick up another one of these in a few months, since RAM prices might finally be trending down this year
Case: Rosewill RSV-L4412 - I know this isn't the best option in terms of quality, but it needs to live in my office for a year until I move somewhere I can build a server closet, so I like the option of using quiet fans. I plan to replace the front fans with Noctua NF-P12s and rears with NF-A8s to keep things cool and quiet.

I already have a PSU and SSD laying around I can toss in here too.

Couple other miscellaneous questions/thoughts... I'm not sure I understand the benefit of splitting up a RAIDZ2 array into multiple vdevs. It seems like a single vdev with all 8 disks would give me about 48TB out of 64TB useable storage but if I split that in two 4 disk vdevs I only net 40TB. In the latter scenario, does each vdev have double parity?

Also wondering about backup strategies... Is anyone using cloud providers like Backblaze? Do I need to invest in some crazy tape drive setup?

In general... am I crazy, or does this look like a solid setup?

 

[danb35]

I'm not sure I understand the benefit of splitting up a RAIDZ2 array into multiple vdevs.

With only 8 disks, I wouldn't split it up.

It seems like a single vdev with all 8 disks would give me about 48TB out of 64TB useable storage but if I split that in two 4 disk vdevs I only net 40TB.

No, two 4-disk RAIDZ2 vdevs would give you 32 TB, or 29 TiB, of which you could only really use 80%.

Also wondering about backup strategies... Is anyone using cloud providers like Backblaze?

FreeNAS 11.1 supports a variety of cloud storage providers including Backblaze, IIRC. Google Drive is another one, and if you have a paid Google account you should have unlimited storage there. Crashplan has been a popular option, but they've changed their offerings in a way that's less attractive than it once was.

or does this look like a solid setup?

It looks like a solid setup, if a bit of overkill. But I'd recommend a SSD for the boot device rather than the USB sticks.

 

[cship]

With only 8 disks, I wouldn't split it up.

No, two 4-disk RAIDZ2 vdevs would give you 32 TB, or 29 TiB, of which you could only really use 80%.

Thanks, that was my gut feeling. Probably screwed up the math, as you can tell I haven't used ZFS before.

FreeNAS 11.1 supports a variety of cloud storage providers including Backblaze, IIRC. Google Drive is another one, and if you have a paid Google account you should have unlimited storage there. Crashplan has been a popular option, but they've changed their offerings in a way that's less attractive than it once was.

I do use Backblaze and GDrive currently, but I wasn't sure if these things really mean "unlimited". I likely won't have a problem with this for awhile in any case :)

It looks like a solid setup, if a bit of overkill. But I'd recommend a SSD for the boot device rather than the USB sticks.

Cool, using the SSD makes sense. It's probably overkill for my current streaming usage, but I want enough overhead to do some general home server stuff on it in the future that doesn't quite justify investing in a separate beefy machine. I'll definitely be throwing more RAM at this one as I can afford it. Thanks for your help!

 

[Chris Moore]

Thanks, that was my gut feeling. Probably screwed up the math, as you can tell I haven't used ZFS before.

This is a good calculator to use for capacity planning.
http://wintelguy.com/zfs-calc.pl

 

[toyebox]

This is a good calculator to use for capacity planning.
http://wintelguy.com/zfs-calc.pl

I am not sure if i believe in coincidences, but me stumbling along this post while just browsing around is quite funny. I just found that page not even 20 minutes ago and i am flabbergasted by the insane amount of useful tools on there!

Edit:

A couple of points:

-The build looks pretty solid. I would definitely recommend getting that other stick of memory though. If need be, get less drives, and get the memory. As it sits, you wouldn't be hitting the recommended 1G of memory per physical TB of space.

-I am unsure what danb35 is saying about using an SSD for the boot device.. He has been on here much longer than I, but from my experience a USB will do just fine given FreeNAS is loaded into RAM on boot anyway. Just keep frequent backups of your config (once a week is sufficient, or after major changes to the config).

-I would strongly suggest mirrors, but this is a long running argument/conversation. The quick and dirty details:
when a failure of a disk in a mirror occurs, the re silvering takes a fraction of the time compared to a RaidZ. More importantly, a Raidz will stress all hard drives in the vdev in order to rebuild. A mirror will simply need to do a direct copy from the backup drive to the new one. Again, someone else might argue different points that could persuade you otherwise.

 

[danb35]

freeNAS is loaded into ram on boot anyway.

This hasn't been the case for years, and I wish people would stop repeating it. ZFS caching means that most of the OS lives in RAM most of the time, but the RAMdisk went away with the release of FreeNAS 9.3 in 2014.

Just keep frequent backups of your config

Good advice, but they're now made automatically in the .system dataset--though getting to that can be a little tricky.

As it sits, you wouldn't be hitting the recommended 1G of memory per physical TB of space.

That's a very loose, and deliberately vague, recommendation. And if he's wanting to put 8 disks in RAIDZ2 (which is a perfectly sensible configuration), he'll need to have 8 disks at the outset. More RAM is always good, but the proposed build should be fine.

a Raidz will stress all hard drives in the vdev in order to rebuild.

True, if by "stress" you mean "use." I have yet to see any convincing evidence that this so-called "stress" causes any problems.

 

[toyebox]

This hasn't been the case for years, and I wish people would stop repeating it. ZFS caching means that most of the OS lives in RAM most of the time, but the RAMdisk went away with the release of FreeNAS 9.3 in 2014.

Didn't know that, learn something new every day :) thanks for that!

Good advice, but they're now made automatically in the .system dataset--though getting to that can be a little tricky.

Again, something I was unaware of--i still use a bash script to make local copies on one of my pools.

That's a very loose, and deliberately vague, recommendation. And if he's wanting to put 8 disks in RAIDZ2 (which is a perfectly sensible configuration), he'll need to have 8 disks at the outset. More RAM is always good, but the proposed build should be fine.

I said nothing of the sort--that a raidz2 configuration was not sensible. I was only using my own experiences of drive failures during resilvering of a raidz2. I had an additional failure which caused a loss of the pool. Of course this is the reason we preach backups though.

As for the ram recommendation. In the OP he specifically mentions 8x8TB disks though. The recommendation of 1g per 1Tb of physical disk size is plastered all over this entire forum, so I don't think I'm out of line mentioning that he is short 32gb.

True, if by "stress" you mean "use." I have yet to see any convincing evidence that this so-called "stress" causes any problems.

Again, personal experience and opinion on my part. I agree, there are many variables that can play in the failure of a drive, and the "use" of every drive in my pool during the resilver process of a raidz to me is an extra risk I'm not willing to take when a simple CP function will suffice. Please don't take my opinions personally. I am simply giving the new member a look at all options is all.

 

[danb35]

The recommendation of 1g per 1Tb of physical disk size is plastered all over this entire forum

Lots of stuff is "plastered all over this entire forum"; not all of it is valid or correctly understood. The hardware recommendations guide says:

A rule of thumb for RAM sizing is 1GB per 1TB of storage. This rule is left deliberately vague as it is only a rule of thumb.
. . .
16GB is probably the sweet spot for most home users, but more RAM is generally an easy way of improving server performance. For heavy jails/VM/plugin usage, 32GB might be a better starting point.

...and the manual says:

An old, somewhat overstated guideline is 1 GB of RAM per terabyte of disk capacity.

It's a rough guideline, it's deliberately vague as to whether it's total disk capacity or net pool capacity, and at larger capacities it really overstates the RAM required. More RAM is always a good thing, but 32 GiB of RAM for 43 TiB of storage capacity should be fine.

 

[Chris Moore]

personal experience and opinion on my part. I agree, there are many variables that can play in the failure of a drive, and the "use" of every drive in my pool during the resilver process of a raidz to me is an extra risk I'm not willing to take when a simple CP function will suffice. Please don't take my opinions personally. I am simply giving the new member a look at all options is all.

The thing is, you are a new member also and in a mirror there is only one drive protecting the pool from total collapse where in RAID-z2, there are two drives. If you had a failed pool, more than two drives would have had to failed. I have replaced two drives simultaneously in a pool and with healthy drives that should not be a problem. Your described failure was likely due to you not monitoring the health of your drives.
The only time a pool of mirrors is preferable to RAID-z2 is when you need high IOPS.

 

[cship]

Thanks for the input everyone! I went ahead and ordered the parts, will post back with an update once I get everything up and running.

I have replaced two drives simultaneously in a pool and with healthy drives that should not be a problem.

This was my understanding, and I'm satisfied with that level of risk.

 

[danb35]

and the "use" of every drive in my pool during the resilver process of a raidz to me is an extra risk I'm not willing to take

So do you scrub your pool? Because that's exactly what happens to the remaining disks in your RAIDZn pool when you resilver a disk.

You're right that pool layout warrants some serious consideration, and OP should consider all the alternatives. And I'm aware of the argument for using mirrors, particularly in a smaller installation. But let's compare mirrors vs. RAIDZ2 in OP's case (to wit, 8 x 8 TB disks):

• Mirrors will give 50% space efficiency, RAIDZ2 will give 75%. That means that mirrors will have 32 TB available, and RAIDZ2 will have 48 TB available. This ignores overhead, reserved space, and the 80% rule, of course.
  
• RAIDZ2 will tolerate the loss of any two disks without data loss. With mirrors, it's possible that the second failure could destroy the entire pool. It's also possible to lose up to four disks without data loss. It all depends on which disks fail.
• Mirrors can be more easily expanded--either by replacing disks (you'd only need to replace one mirrored pair to expand the pool) or by adding new disks (expand the pool a pair at a time). To expand the proposed 8 x 8 TB RAIDZ2 array, you'd need to replace all eight disks with something larger, or add another eight disks in RAIDZ2*.
• Resilvers are reported to be significantly faster for mirrors than for RAIDZn. This has never concerned me greatly, but it apparently concerns some people.
• The mirrors are going to have higher IOPS, and may therefore be a better solution for VM storage.

* It's recommended that the new vdev match the old one, but it isn't required.

 

[toyebox]

The thing is, you are a new member also and in a mirror there is only one drive protecting the pool from total collapse where in RAID-z2, there are two drives. If you had a failed pool, more than two drives would have had to failed. I have replaced two drives simultaneously in a pool and with healthy drives that should not be a problem. Your described failure was likely due to you not monitoring the health of your drives.
The only time a pool of mirrors is preferable to RAID-z2 is when you need high IOPS.

Hello Chris,
Please don't mistake my "newbie" brand as me being a new member. I have been using freenas for many years--It was only till recently i made a forum account. I do appreciate you trying to share some knowledge with me, although i already knew this information (i understand you were acting on the premise that i was new). Your speculation that my second failure was due to not monitoring the health of my drives is really jumping to conclusions though. So you are saying you have never had a drive fail prematurely (given manufacture expected lifespan)?

So do you scrub your pool? Because that's exactly what happens to the remaining disks in your RAIDZn pool when you resilver a disk.

You're right that pool layout warrants some serious consideration, and OP should consider all the alternatives. And I'm aware of the argument for using mirrors, particularly in a smaller installation. But let's compare mirrors vs. RAIDZ2 in OP's case (to wit, 8 x 8 TB disks):

• Mirrors will give 50% space efficiency, RAIDZ2 will give 75%. That means that mirrors will have 32 TB available, and RAIDZ2 will have 48 TB available. This ignores overhead, reserved space, and the 80% rule, of course.
  
• RAIDZ2 will tolerate the loss of any two disks without data loss. With mirrors, it's possible that the second failure could destroy the entire pool. It's also possible to lose up to four disks without data loss. It all depends on which disks fail.
• Mirrors can be more easily expanded--either by replacing disks (you'd only need to replace one mirrored pair to expand the pool) or by adding new disks (expand the pool a pair at a time). To expand the proposed 8 x 8 TB RAIDZ2 array, you'd need to replace all eight disks with something larger, or add another eight disks in RAIDZ2*.
• Resilvers are reported to be significantly faster for mirrors than for RAIDZn. This has never concerned me greatly, but it apparently concerns some people.
• The mirrors are going to have higher IOPS, and may therefore be a better solution for VM storage.

* It's recommended that the new vdev match the old one, but it isn't required.

All very good points sir. I would like to note again that i am not against RAIDZ configurations as i have built many systems for customers that currently use RAIDZ. It really does give a member the warm and fuzzy that they can lose any 2 drives and still be alright.

I appreciate all the feedback about my comments, and i hope i didn't ramble on too much.

 

[Chris Moore]

So you are saying you have never had a drive fail prematurely (given manufacture expected lifespan)?

I have a server at work with 60 drives in it and they are the WD Red Pro drives with 5 year warranties. During the first year the server was in service, I had to replace six drives, three of those in the first month, the other three at fairly random intervals, but a long time before the 5 year warranty was over. It happens. Drives fail. Sometimes you even get a bunch all at once, like the three that failed on me over a weekend when the server was first being loaded with the (about) 210TB of data that it houses. I have found that if you use good practices with monitoring drive faults, you can replace drives when they first become problematic, before they cause data loss. Also, those large capacity drives take about 3 days to resilver, so I do worry about a second failure during the recovery time, which is why I use RAID-z2.

PS. I have been a computer professional since 1997 and the care and feeding of servers has been a component of that work since around 1999. I can't begin to estimate the number of drives I have replaced. They generally fall into two categories for me. The ones that miraculously survive their useful life and the ones that fail before you are done with them. I have some drives that are two small (20 or 40 GB) to be useful but they still work. Sometimes it is kind of amazing that a drive can last that long. Other times, they are such disappointments.

 

[Jatrabari]

FreeNAS 11.1 supports a variety of cloud storage providers including Backblaze, IIRC. Google Drive is another one, and if you have a paid Google account you should have unlimited storage there. Crashplan has been a popular option, but they've changed their offerings in a way that's less attractive than it once was.

How can Backblaze be used to backup FreeNAS server?

 

[danb35]

How can Backblaze be used to backup FreeNAS server?

http://doc.freenas.org/11/tasks.html#cloud-sync

 

[cship]

http://doc.freenas.org/11/tasks.html#cloud-sync

Ah, it looks like this is actually Backblaze B2 and Google Cloud Storage integration, those products are commercial solutions that are significantly more expensive than their consumer plans (Backblaze and GDrive with "unlimited" storage) but much more in line with what I would expect to pay to reliably backup terabytes of data.

 

[danb35]

I have no experience with Backblaze so can't say at all. But the Google piece should work just fine with Google Drive. I haven't used the integrated support, but it uses rclone for the backend, and that most definitely supports Google Drive.

 

[cship]

I have no experience with Backblaze so can't say at all. But the Google piece should work just fine with Google Drive. I haven't used the integrated support, but it uses rclone for the backend, and that most definitely supports Google Drive.

Interesting, thanks for the tip. I'll dig into this more when the hardware shows up :)

 

[danb35]

In the worst case, you can manually configure rclone to work with your Google Drive account (including encryption, if desired). See https://rclone.org/.

 

[Jatrabari]

Ah, it looks like this is actually Backblaze B2 and Google Cloud Storage integration, those products are commercial solutions that are significantly more expensive than their consumer plans (Backblaze and GDrive with "unlimited" storage) but much more in line with what I would expect to pay to reliably backup terabytes of data.

I am in the understanding that the standard Backblaze account is only for Windows and Mac and B2 is specifically for NASs for example. I would use the B2 service anyway. Is there any info about client side encryption for cloud-sync? I can't check this myself.