FreeNAS - New build to replace QNAP

[tealcomp]

Hi Fellow Forum Members:

I joined the forum back in September of 2016 and I have been reading posts and "recommended" documentation ever since. It has given me time to reflect on my objectives and how I should go about getting there. I have gone back and forth between a Supermicro 24 bay chassis solution and something a little more conventional. Finally, after a lot of deliberation, I decided to build a new FreeNAS solution that is more compact and easier to manage within a home environment

Objectives:

1. Replace an older, hardware RAID-6 based storage array (nearing EOL) that is presently used to backup my primary QNAP storage system.
   
2. Move the QNAP appliance into the secondary storage role, kept in sync using RSYNC.
   
3. Increase storage capacity with this new FreeNAS solution to allow for data growth.

Usage:

1. SMB accessible, shared file system.
   
2. Maximum of 5 concurrent users.
   
3. Mixed file types: media (audio/video) libraries, images, various documents, and backups of personal devices.

In addition to mirroring the data between servers, routine "offline" backups will be made. I do fully understand the necessity of continuous backups, in the event either the primary or secondary data servers were to fail.

Hopefully, all of my homework and reading and sometimes re-reading have paid off and it won't be too painful to correct any oversights I might have made.

Proposed build:

Case: Fractal Design 804

Power: Seasonic Platinum 760W, calculated from the forum guidelines.

Cooling: There are 3x R2 cooling fans included with the case I have chosen; I will likely need additional fans to optimize cooling, but I may wait and see what temps I get with the stock fans. I don't like buying equipment for what-if scenarios, but I am open to suggestions.

Motherboard: Supermicro X11SSL-CF-O

I selected the X11 because it will allow me to have up to 64GB of ECC unbuffered memory and should adequately support my stated goals.

CPU: E3-1240, v5 (likely overkill for the given tasks, but since I started out with the idea of an E5, I figured I was making a good compromise).

RAM: Crucial 32GB DDR4-2400 ECC UDIMM (CT7982583)

Taken directly from the Crucial website; this is the highest density and I plan to go either with 32GB (2x) or 64GB (4x) for this project. The above part number is a 32GB paired set.

Boot: (2x, mirrored), SanDisk SSD PLUS 120GB Solid State Drive (SDSSDA-120G-G26) [Newest Version]

As I am not a fan of USB fobs, I picked 2 reasonably priced (~$100 for 2) and rated SSDs, to host the OS.

Storage: 10x WD Red (WD60EFRX)

I normally would go with an enterprise class drive; the QNAP for example has 4TB RED PRO drives in it; however, I do like the lower wattage of the plain REDS, albeit with lesser warranty coverage. It's a toss up, but I figured, given the case I am choosing, if I could lessen the heat load without really negatively impacting the overall transfer speed, why not do it? Presently, I have a GbE network, with the ability to go to 10 GbE if warranted.

I also thought about going with SAS based drives with the built in controller, but then I would have needed to add an expander to get past 8 drives and that just seemed cost-ineffective when I already have 8 SATA connections from the built-in SAS controller (via break-out cables) and 6 additional SATA connectors with the chosen motherboard. Even with 10 data drives, and 2 SSD's for the OS, I have 2 spare SATA ports if needed.

Storage Configuration Options

For a ZFS VDev/zpool solution, at 10 drives I know I am pressing the "width" barrier. With that in mind, I have outlined a few potential options. Calculated capacities provided by the "ZFS Drive Size and Cost Comparison" spreadsheet created by @diedrichg

Option 1: 1 zpool, 1 VDev, 10 drives in a RAIDZ2 configuration with a projected capacity of ~37.79TB

• Better capacity at the risk of redundancy and recovery times.

Option 2: 1 zpool, 2 VDevs, 5 drives each in a RAIDZ2 configuration, with a projected capacity of ~14.17TB per VDev or total projected capacity of ~28.34TB

• Better redundancy at the cost of available space.

Option 3: 1 zpool, 1 VDev, 10 drives in a RAIDZ3 configuration with a projected capacity of ~33.06TB

• A compromise, better redundancy with a minor reduction in available space.

If Option 3 doesn't slow down immensely because of the 3 way parity calculations, it seems the best of all 3 options; otherwise, I am leaning towards Option 1.

Well, that about does it for now. I welcome all constructive criticism and advice, and I look forward to being part of this community of FreeNAS owners.

 

[diedrichg]

Thanks for the props.

Do some research on the 6TB drives. BackBlaze has data on the most reliable drives in production. I seem to recall the 6TB Reds not lasting as long as drives such as the 4TB.

 

[JohnDigital]

This thing will put your QNAP to shame. Congrats and salute!

120gb boot drives are much too large. I think go for 16gb boot SSDs and shoot for 2 larger SSD drives to hold docker containers VMs and or jails. Even with about 10 boot environments you wouldn't even be able to fill 16gb SSDs. You say you still have 2 open ports? 2 16gb SSD boot mirror, 2 240gb "jail" mirror, 10 6tb in Z3, ftw!

But I'm otherwise jealous of your build. I'm rocking Z3 just for piece of mind as I'm a little wider than recommended. Sounds like you got it figured out.

Good luck and welcome!

 

[snaptec]

A hint:
Test your qnap rsync.
I had some qnaps where rsync was damn slow and killed itself After a couple of hours.
Impossible to make rsync backups!


Gesendet von iPhone mit Tapatalk

 

[tealcomp]

Thanks for the props.

Do some research on the 6TB drives. BackBlaze has data on the most reliable drives in production. I seem to recall the 6TB Reds not lasting as long as drives such as the 4TB.

Thanks!

I did take a look at the Backblaze data as you suggested. It doesn't seem like they really use many WD products though, so I really can't draw any actionable conclusions from their limited product experience. I also spent a good deal of time researching this forum and the REDS appear to have a good reputation, specifically the REDS I am proposing. In my years of IT, I have used both WD and Seagate and both have been pretty reliable in my personal usage. It does look like the closest comparison to the RED in the Seagate line is the "IronWolf" model, but at 6TB they ramp up to 7200 RPM; I don't see any 5400-5900 RPM NAS rated devices in the Seagate product line, did I miss them? The lower RPM drive was one of my objectives in this new build. In fact, from what I can tell so far, WD is the only manufacturer of the big 4 that has a 6TB drive that runs at the lower RPM. I also do not use desktop rated drives outside of their intended purposes; I did that once before and it only took one time to learn that lesson :) Anyway, I do appreciate the heads up but I am sticking with this option for now.

 

[tealcomp]

A hint:
Test your qnap rsync.
I had some qnaps where rsync was damn slow and killed itself After a couple of hours.
Impossible to make rsync backups!


Gesendet von iPhone mit Tapatalk

The QNAP is definitely a drag when it comes to RSYNC; however, we have been using it successfully for over 2 years between the QNAP and an Ubuntu server that is RAID6 attached to a DAS. It has been entirely reliable in my experience.

 

[tealcomp]

This thing will put your QNAP to shame. Congrats and salute!

120gb boot drives are much too large. I think go for 16gb boot SSDs and shoot for 2 larger SSD drives to hold docker containers VMs and or jails. Even with about 10 boot environments you wouldn't even be able to fill 16gb SSDs. You say you still have 2 open ports? 2 16gb SSD boot mirror, 2 240gb "jail" mirror, 10 6tb in Z3, ftw!

But I'm otherwise jealous of your build. I'm rocking Z3 just for piece of mind as I'm a little wider than recommended. Sounds like you got it figured out.

Good luck and welcome!

John:

Thanks! I did look for smaller capacities with the SSD's. I will take another look and see if I can dig something up. For SSDs, I think 32GB is the absolute floor (assuming I can find it and NOT on eBay). I would like to get these parts ordered today if possible, so I will admit the SSD was one of the lesser researched items. I did look at the DOMS as well, but I suspect for roughly the same price at 1/4 of the capacity, the expected lifetime is probably not going to be as good. That all said, is there a reason (if the price and product is decent) I should NOT go with a 120GB? I cannot think of one off hand.

 

[SweetAndLow]

Seems like a good build. Make sure you are ready to burn everything in. It will probably take a week to do all the drives, memory and CPU.

You can also think about getting 11 disks and doing a raid z3. The numbers work out a little better and it's still a suggested size, even though it's at the upper end.

Sent from my Nexus 5X using Tapatalk

 

[diedrichg]

For future reference:
https://www.backblaze.com/blog/hard-drive-failure-rates-q3-2016/
Scroll down to the "Cumulative hard drive failure rate by model" section.

Hard drive data and stats
https://www.backblaze.com/b2/hard-drive-test-data.html

 

[tealcomp]

For future reference:
https://www.backblaze.com/blog/hard-drive-failure-rates-q3-2016/
Scroll down to the "Cumulative hard drive failure rate by model" section.

Hard drive data and stats
https://www.backblaze.com/b2/hard-drive-test-data.html

Thanks, odd that I missed the q3 data even when I went looking specifically for it :) At any rate, from my perspective, it is but one data point from one provider who already has had their data called into question numerous times. Beyond that, my configuration is an apples to oranges on usage; "how" they are using the drives is an important factor (high density, subject to higher vibration and temperature); but I truly appreciate the information. It was taken into consideration. I guess time will tell if I have success or am more of a statistic :)

 

[snaptec]

I always use 20gb ssds for FreeNAS, hypervisor etc.
if you need some drop me a pm.


Gesendet von iPhone mit Tapatalk

 

[JohnDigital]

John:

Thanks! I did look for smaller capacities with the SSD's. I will take another look and see if I can dig something up. For SSDs, I think 32GB is the absolute floor (assuming I can find it and NOT on eBay). I would like to get these parts ordered today if possible, so I will admit the SSD was one of the lesser researched items. I did look at the DOMS as well, but I suspect for roughly the same price at 1/4 of the capacity, the expected lifetime is probably not going to be as good. That all said, is there a reason (if the price and product is decent) I should NOT go with a 120GB? I cannot think of one off hand.

No. If you are happy, I'm happy. Its just a matter of wasting that other 115gb of space. You wont need more than 8gb of space. If you don't mind I certainly don't. I like the build.

I've had great luck with Samsung 16gb from eBay. At 15 bucks they are certainly as reliable as a flash drive. I've had those go out left and right.

 

[Stux]

Xeon E3-1230v5 is better value in my opinion. Only 4% slower than the 1240v5 but generally 10% cheaper.

I think either z2 or z3 with one vdev. I'd be tempted to go wider with z3 or to 12 disks with 6Way z2. I'm not a fan of 5-way z2. Just don't like 40% parity loss ;). At 40% it almost feels like you might as well use mirrors and have the benefit of flexibility.

 

[tealcomp]

Hi Stux:

Thanks for taking a look at my configuration, I appreciate it. Yes, I debated the whole Z2 vs Z3 thing for a few days; I like your suggestion; trouble is, I am not sure I can shoe horn 12 drives into the case I have chosen. I really like this case and a lot of time and research went into arriving at it as my candidate.
I haven't ordered anything yet, so I will take a look at the processor pricing, thanks for the tip.

 

[tealcomp]

Hi All:

I am about to let lose the credit card hounds and purchase the parts. I really appreciate the input I have received so far; if there is anybody else that would like to provide any last minute advice or recommendations, I am all ears. I am still seeing if I can get smaller SSD's for the boot drives.

PS I already did purchase the 6TB REDS, so I intend to keep everyone up to speed on how those perform for me.

Thanks again!
-Dan

 

[JohnDigital]

You are going to love the machine! It looks like it will be a humdinger. Welcome and let us know if you have any other questions. I know you arent a fan of the eBay, but HERE is where I get my 16gb SSDs. Very legit, 100% FB ratings. Good luck.

 

[tealcomp]

You are going to love the machine! It looks like it will be a humdinger. Welcome and let us know if you have any other questions. I know you arent a fan of the eBay, but HERE is where I get my 16gb SSDs. Very legit, 100% FB ratings. Good luck.

John:

Thanks, I did not have any idea there were drives any smaller than 32GB in MLC format; which begs the questions, what's the lifespan left on these drives and how old are they? Of course if I save the config elsewhere, for $12 seems like a darn good buy..is that your thinking?

 

[JohnDigital]

I did ask via the question system on how "used" they were. But this is some time time ago, and I believe the response was to the effect of very little, they come out of leased equipment of one form or another. Whatever the response was I felt good enough going forward to install them in my machines. They are Samsung branded drives, and mine have been running along without a single hiccup for well over a year. Like I said at 15 bucks, they are very legit.

 

[JohnDigital]

HERE is a different guys selling identical drives for a buck cheaper, faster shipping, and a 1 year warranty. The game has apparently changed.

 

[jdustinb]

This is right in line with what I have in mind for a build myself. I've just been trying to finalize on a CPU/MB combo. I'm curious if you came across a more powerful yet still cost effective solution from a MB/CPU standpoint as I plan on running this with Plex and require ~5 concurrent transcoding streams.