NAS Build options

[typhonragewind]

Hi everyone!

I'm hoping to soon join the ranks of FreeNAS users and replace my old single disk (shudder) 3TB WD MyCloud.
I've finally been able to digest most of the important info on building a FreeNAS system and i think I managed to design 3 acceptable builds.
However, since this is a bit of uncharted territory in terms of hardware specifications that i'm not used to, I'd like to ask you to not only please confirm if everything checks out in terms of compatibility but also your opinion on the system itself (what can be improved without going too over-budget).
I'll use the system for file storage (Main), Plex Server (Main), OwnCloud server (Main), small database website (secondary), calibre server (probably via Docker, secondary) and maaaybee a MineOS server.
People using it will be me, and probably one or two others occasionally on Plex and other servers.

So, without further ado, here are the 3 possible builds:


Build 1

Total Estimated Cost: 1143 USD ~ 1010 EUR
Estimated Peak Power Supply Needs: 360W

Motherboard: Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSL-F-O
CPU: Intel Core i3-6300
RAM: Timetec Hynix 16GB DDR4 2133MHz PC4-17000 Unbuffered ECC 1.2V CL15 2Rx8 Dual Rank 288 Pin UDIMM Server Memory RAM Module Upgrade (16GB)
Chassis: Fractal Design Node 804 No Power Supply MicroATX Cube Case FD-CA-NODE-804-BL, Black
Power Supply: Seasonic FOCUS 450 Gold SSR-450FM 450W 80+ Gold ATX12V & EPS12V
Storage: 4x WD Red 2TB
Boot Device: Transcend 32GB MLC SATA III 6Gb/s 2.5" Solid State Drive 370 (TS32GSSD370S)


Build 2

Total Estimated Cost: 923 USD ~ 815 EUR
Estimated Peak Power Supply Needs: 360W

Motherboard: Supermicro Motherboard Micro ATX DDR3 1600 LGA 1150 Motherboards X10SLL-F-O
CPU: Intel Core i3-4330
RAM: Samsung M393B2G70BH0-CK0 Samsung 16GB PC3-12800 DDR3-1600MHz ECC Registered CL11 240-Pin DIMM Dual Rank Memory Module Mfr P-N
Chassis: Fractal Design Node 804 No Power Supply MicroATX Cube Case FD-CA-NODE-804-BL, Black
Power Supply: Seasonic FOCUS 450 Gold SSR-450FM 450W 80+ Gold ATX12V & EPS12V
Storage: 4x WD Red 2TB
Boot Device:Transcend 32GB MLC SATA III 6Gb/s 2.5" Solid State Drive 370 (TS32GSSD370S)

Very similar to Build 1, not sure if Build 1 motherboard/cpu upgrade is worth the extra money.


Build 3

Total Estimated Cost: 1073 USD ~ 950 EUR
Estimated Peak Power Supply Needs: 360W

Motherboard+CPU: Supermicro Atom C2758 64GB DDR3 PCIE SATA USB Mini ITX DDR3 1333 NA Motherboards MBD-A1SRI-2758F-O
RAM: Samsung M393B2G70BH0-CK0 Samsung 16GB PC3-12800 DDR3-1600MHz ECC Registered CL11 240-Pin DIMM Dual Rank Memory Module Mfr P-N
Chassis: Fractal Design Node 304 Mini-ITX Hybrid Computer Case FD-CA-NODE-304-BL
Power Supply: Seasonic FOCUS 450 Gold SSR-450FM 450W 80+ Gold ATX12V & EPS12V
Storage: 5x WD Red 2TB
Boot Device: Transcend 32GB MLC SATA III 6Gb/s 2.5" Solid State Drive 370 (TS32GSSD370S)

Not possible to expand this one with more drives, only through bigger drives, which is a bit of a downside to this one. But it makes for a very compact and powerful system. This board might be a bit difficult to source, not sure.


You probably have noticed that there are no CPU coolers in those builds - this is because i have no idea what is the current gold standard for coolers, so an extra dose of help on this would be greatly appreciated!

Thoughts? (Also prayers once i start building it xD)

 

[joeschmuck]

You probably have noticed that there are no CPU coolers in those builds - this is because i have no idea what is the current gold standard for coolers, so an extra dose of help on this would be greatly appreciated!

The Intel CPUs in a Box come with heatsinks, no need to buy anything further. I would not purchase an aftermarket cooler unless you find something wrong with the boxed unit.

I like option 1 the most but then again it's almost the same motherboard I have so I'm biased. Make sure the RAM is compatable, I don't have time to look it up for you this morning.

You didn't specify how much storage capacity you desired nor the RAIDZ level you wanted. With a RAIDZ2 level you would have just over 3TB of storage. That doesn't sound like much to me. If you're not familiary with FreeNAS and ZFS please understand that you cannot add a single drive to expand you storage safely. Do a Google Search for "ZFS Primer FreeNAS" and check out this link as well.

The case and power supply should be fine as well.

 

[typhonragewind]

The Intel CPUs in a Box come with heatsinks, no need to buy anything further. I would not purchase an aftermarket cooler unless you find something wrong with the boxed unit.

I like option 1 the most but then again it's almost the same motherboard I have so I'm biased. Make sure the RAM is compatable, I don't have time to look it up for you this morning.

You didn't specify how much storage capacity you desired nor the RAIDZ level you wanted. With a RAIDZ2 level you would have just over 3TB of storage. That doesn't sound like much to me. If you're not familiary with FreeNAS and ZFS please understand that you cannot add a single drive to expand you storage safely. Do a Google Search for "ZFS Primer FreeNAS" and check out this link as well.

The case and power supply should be fine as well.

Thank you for your input!
You're right, i forgot about that. I'll be using RAIDZ1 and I'm considering to get some bigger discs - 6TB should be enough, but you never know. I've read about the ZFS quite a bit so i know the expandability limitations, and it's something that made me consider getting a Synology....but the FreeNAS allows for such a better system with a bit of planning that it wasn't worth it.

 

[joeschmuck]

You are correct that FreeNAS offers a better system for the cost. Thourghput is a major concern for most people here anmd if you are willing to work a little bit then you can save a lot of money to get a fast system.

 

[Pitfrr]

I'll be using RAIDZ1 and I'm considering to get some bigger discs - 6TB

Regarding the RAIDz level, it is not advised to use RAIDz1 with larger disks (around >1TB).
So therefore I'd go for RAIDz2 and advise also 6 disks (which I find to be the minimum number of disks for a RAIDz2 volume to have a nice raw/useable data ratio).
Which then brings an other problem: 6 disks and nowhere left to connect the boot device!!

Can't say much on the different builds, they look all nice to me. I'd like to try the C2758... (just to compare with the C2750).
One detail though on the CPU: if I had to choose I'd probably go for the i3-6300, since it supports VT-d and 64GB or RAM...

 

[joeschmuck]

Regarding the RAIDz level, it is not advised to use RAIDz1 with larger disks (around >1TB).

I've never heard that limitation before. The determining factor is resilvering time. A 2TB drive can resilver in 4 to 6 hours, the system configuration will make a difference of course. It's when you get into 6TB and above where resilvering can take longer than 24 hours where a risk comes in. Remember that a RAIDZ1 is one failed drive safe, if a second failure occurs then you have total data loss. I always recommend RAIDZ2 unless the user just doesn't need it.

So therefore I'd go for RAIDz2 and advise also 6 disks (which I find to be the minimum number of disks for a RAIDz2 volume to have a nice raw/useable data ratio).
Which then brings an other problem: 6 disks and nowhere left to connect the boot device!! :p

I too like six drives, it makes for a faster data pool. As for the boot drive, you can mount the SSD anywhere in the case, use strong Velcro tape or drill a few holes for mounting screws. It's small and doesn't generate a lot of heat so it can go anywhere you find some space. Use your imagination. Just don't block airflow.

 

[typhonragewind]

Regarding the RAIDz level, it is not advised to use RAIDz1 with larger disks (around >1TB).
So therefore I'd go for RAIDz2 and advise also 6 disks (which I find to be the minimum number of disks for a RAIDz2 volume to have a nice raw/useable data ratio).
Which then brings an other problem: 6 disks and nowhere left to connect the boot device!! :p

I'm undecided on this. RAIDz2 seems kinda overkill and would maim my budget a bit.

Can't say much on the different builds, they look all nice to me. I'd like to try the C2758... :) (just to compare with the C2750).
One detail though on the CPU: if I had to choose I'd probably go for the i3-6300, since it supports VT-d and 64GB or RAM...

My biggest thron in those builds is when when i see how old the components are. The CPU for example, is almost from 4 years ago, and even worse for the Atom.
But i'm a newbie at this kind of build so i couldn't figure something more recent within my budget (around 1000-1100$ including disks). I'm getting actual anxiety from this .

always recommend RAIDZ2 unless the user just doesn't need it.

How do you define the need for this?

 

[Pitfrr]

RAIDz2 seems kinda overkill

Well, one thing to take into account as well would be how much do you value your data?
If it's non critical data that can easily be replaced this gives you a different approach than valuable unreplaceable data. ;-)

About the hardware age... I think you have to make a compromise between price, performance and reliability (new HW might not be out long enough for proven reliability).

 

[typhonragewind]

Well, one thing to take into account as well would be how much do you value your data?
If it's non critical data that can easily be replaced this gives you a different approach than valuable unreplaceable data. ;-)

The vast bulk (about 2TB) is stuff that while annoying to get back is not truly irreplaceable. The rest, about 600GB of truly irreplaceable data is backed up on several devices and locations, and another 600GB is unique data that is currently only sitting on an old 1TB drive (which causes me no end of anxiety). So making a compromise between this on a budget, well....you can see xD. I'm probably going to give in to my paranoid side and go RAIDz2 AND get a backup solution for those 600GB.

About the hardware age... I think you have to make a compromise between price, performance and reliability (new HW might not be out long enough for proven reliability).

Do you have any specific recommendations?

 

[Pitfrr]

Do you have any specific recommendations?

Unfortunately I don't... The systems I'm using are also pretty old now and I haven't been keeping up with the latest CPU and so...
Since your most strong constraint is budget, this means you're probably not gonna aim for the latest (mostly expensive) hardware so I would try to aim for the best performance for that budget (in the selection you made that would probably be the i3-6300). But you could also consider acquiring second hand hardware (I wouldn't really mind for CPU, RAM and mainboard), but that is a personal preference.
The Pentium Gxxxx family might also be affordable (but I don't know if it makes a great difference compared to the i3?).

Independently of your RAIDZ1/2 choice, in any case, you definitely should get a backup solution for the irreplaceable 600GB!!
I would also advise, for your hard disk capacity choice, if possible, don't aim for the lowest capacity (i.e. 1TB)... but go for the best compromise TB/€ (therefore probably something like 4TB disks, I didn't make the calculation with actual prices).
I know it blows up your budget, especially if you start with 6 disks... but you have to consider the actual capacity needs and what is the expected growth (per year), also keep in mind that you shouldn't completely fill up your pool. You might also want to consider setting up snapshot (which could eat up some space as well)...
Just some thoughts...

 

[typhonragewind]

Unfortunately I don't... The systems I'm using are also pretty old now and I haven't been keeping up with the latest CPU and so...
Since your most strong constraint is budget, this means you're probably not gonna aim for the latest (mostly expensive) hardware so I would try to aim for the best performance for that budget (in the selection you made that would probably be the i3-6300). But you could also consider acquiring second hand hardware (I wouldn't really mind for CPU, RAM and mainboard), but that is a personal preference.
The Pentium Gxxxx family might also be affordable (but I don't know if it makes a great difference compared to the i3?).

I'll try to dig deeper, but as expected there doesn't seem to be newish (in terms of years since launch) equipment at a reasonable price. While i'm at that, what is your opinion on regular (non-server) boards for use in NAS? They will probably have worse power savings, but is that all and if so is it such a big difference?
I'm not very keen on second hand tech, since Ive had nothing but bad experiences with it.

Independently of your RAIDZ1/2 choice, in any case, you definitely should get a backup solution for the irreplaceable 600GB!! :)

Indeed!

I would also advise, for your hard disk capacity choice, if possible, don't aim for the lowest capacity (i.e. 1TB)... but go for the best compromise TB/€ (therefore probably something like 4TB disks, I didn't make the calculation with actual prices).
I know it blows up your budget, especially if you start with 6 disks... but you have to consider the actual capacity needs and what is the expected growth (per year), also keep in mind that you shouldn't completely fill up your pool. You might also want to consider setting up snapshot (which could eat up some space as well)...
Just some thoughts... :)

I'm pretty minimalist with my data, it took me over 5 years to accumulate all this data i spoke off and i don't expect a sudden increase in data usage speed. So even 9 or 12 TB is an upgrade that will last me probably a decade, without filling to 80% capacity (which documentation advises against).


Thank you so much for the time you're taking to discuss this with me!

 

[Pitfrr]

what is your opinion on regular (non-server) boards for use in NAS?

Non server grade mainboards often come with unnecessary things like sound card or other fancy stuff (wifi for example), that will just be useless.
Furthermore, the integrated LAN is often a realtek chipset and you'd want to avoid it (if you don't want to have crappy network performances). They also mostly don't support ECC memory.

Some interesting features of a server grade mainboard is the IPMI possibility. If you put your server in a cupboard or somewhere in the basement you'll be happy to have it (you don't need it very often but you're happy when you do!).

You shall get FreeNAS to run on a consumer grade board but performances might not be the same.
So unless there is a specific reason (or because you already have the hardware and no budget left), I wouldn't go for a non server grade mainboard (maybe for a backup server why not).

 

[joeschmuck]

How do you define the need for this?

I'm pretty minimalist with my data, it took me over 5 years to accumulate all this data i spoke off and i don't expect a sudden increase in data usage speed. So even 9 or 12 TB is an upgrade that will last me probably a decade, without filling to 80% capacity (which documentation advises against)

It appears your questions look to be answered but I'll put my own spin on it.

When determining how you want to design your pool (configuration of the hard drives and capacity) you need to consider these factors:
1) How much capacity do I need for the next 5 years (if this is affordable) and add 20% more.
2) How fast does my data throughput need to be?
3) How long should it take to resilver a failed hard drive?
4) How much protection do I need for all of my data. Am I fine if the data just disappears if I have a single or two hard drive failures, or do I need three drive failures?

Lets look at this closer:
Question #1 is purely about capacity. You desire 9 to 12TB.
Question #2 is about how fast your data must be able to read/write data. In your situation you have expressed little need for anything fast so any pool configuration will do for your needs.
Question #3 is about repair time. If you have short repair times then the risk of data loss is less. Drives with less data will resilver faster than drives with more data, that makes sense. So if you have six 2TB drives these would resilver faster than four 4TB drives holding the same amount of data. That was a crude example and the resilver time wouldn't be double the time but it would be close. There are other facts such as spindle speed but most people looking for a simple NAS also want a quiet NAS so slow spindle speeds help that out.
Question #4 is about redundancy and how many drives can fail without loosing any data, and it's where you consider how important your data is and what it would take to restore all that data if you had to. So a RAID Z1 allows for one drive failure and you data is still safe. Z2 = 2 drives can fail, Z3 = 3 drives can fail.

Lets put all this together...
Lets say you have 12 TB of storage available and you have 10TB of actual data. How much of that data is backed up elsewhere? If you say All of it, then the next question is How long will it take to restore all that data and it that kind of time acceptable? Maybe you have family photos and financial records that you do not routinely backup, if that was the case then a RAIDZ2 would be more favorable. And if you use large drives and your data is super important and needs to remain available all the time, a RAIDZ3 might be what you desire. And maybe you are using this system as a movie server and have a backup and could care less if it fails, then you could create a STRIPE pool and have no redundancy where any drive failure is a total loss of your data.

These questions have nothing to do with money but we all understand money can drive us to compromise. I did when I started out, I purchased four 2TB drives initially and a few months later I bought two more. I had to rebuild my pool but that was easy since I have very little data on the system at that time so I copied everything to an external drive and then rebuild my pool and transferred the data back. It took a little time but it's not difficult.

I have recommended a Stripe to a person here one time and they were made aware of the risks but that doesn't happen often. I always initially recommend a RAIDZ2 system because lets face it, most people are looking at the NAS being the backup and expect the data to be available. If it were easy then we would have a minimum requirement for RAID Z2 and you would have no choice, but you do have a choice.

I hope this helps.

 

[Constantin]

Never underestimate time value of money. I went Z3, which pretty much necessitates a 7- or 8-drive array to make sense.

Per typhonragewind's descriptions, a Z2 array makes sense to me, which brings the number down to 6 drives. For a 12TB array to meet it's recommended 80% fill ratio or less, that pencils out to 4TB drives. If power reduction is needed, go for 4 drives @ 8 TB ea. Consider going over Farmerpling2's disk drive resource, it is pretty incredible.

 

[typhonragewind]

It appears your questions look to be answered but I'll put my own spin on it.

When determining how you want to design your pool (configuration of the hard drives and capacity) you need to consider these factors:
1) How much capacity do I need for the next 5 years (if this is affordable) and add 20% more.
2) How fast does my data throughput need to be?
3) How long should it take to resilver a failed hard drive?
4) How much protection do I need for all of my data. Am I fine if the data just disappears if I have a single or two hard drive failures, or do I need three drive failures?

Lets look at this closer:
Question #1 is purely about capacity. You desire 9 to 12TB.
Question #2 is about how fast your data must be able to read/write data. In your situation you have expressed little need for anything fast so any pool configuration will do for your needs.
Question #3 is about repair time. If you have short repair times then the risk of data loss is less. Drives with less data will resilver faster than drives with more data, that makes sense. So if you have six 2TB drives these would resilver faster than four 4TB drives holding the same amount of data. That was a crude example and the resilver time wouldn't be double the time but it would be close. There are other facts such as spindle speed but most people looking for a simple NAS also want a quiet NAS so slow spindle speeds help that out.
Question #4 is about redundancy and how many drives can fail without loosing any data, and it's where you consider how important your data is and what it would take to restore all that data if you had to. So a RAID Z1 allows for one drive failure and you data is still safe. Z2 = 2 drives can fail, Z3 = 3 drives can fail.

Lets put all this together...
Lets say you have 12 TB of storage available and you have 10TB of actual data. How much of that data is backed up elsewhere? If you say All of it, then the next question is How long will it take to restore all that data and it that kind of time acceptable? Maybe you have family photos and financial records that you do not routinely backup, if that was the case then a RAIDZ2 would be more favorable. And if you use large drives and your data is super important and needs to remain available all the time, a RAIDZ3 might be what you desire. And maybe you are using this system as a movie server and have a backup and could care less if it fails, then you could create a STRIPE pool and have no redundancy where any drive failure is a total loss of your data.

These questions have nothing to do with money but we all understand money can drive us to compromise. I did when I started out, I purchased four 2TB drives initially and a few months later I bought two more. I had to rebuild my pool but that was easy since I have very little data on the system at that time so I copied everything to an external drive and then rebuild my pool and transferred the data back. It took a little time but it's not difficult.

I have recommended a Stripe to a person here one time and they were made aware of the risks but that doesn't happen often. I always initially recommend a RAIDZ2 system because lets face it, most people are looking at the NAS being the backup and expect the data to be available. If it were easy then we would have a minimum requirement for RAID Z2 and you would have no choice, but you do have a choice.

I hope this helps.

This is really in-depth and a great help! Thanks for the time it ook you to write all this!

Now i just need to finalize my CPU decision and it's wallet crying time!

 

[joeschmuck]

Now i just need to finalize my CPU decision and it's wallet crying time!

It is best to get too much CPU than too little. Keep in mind everything that you want to do, it's actually quite a lot if it's all happening at the same time. This also stands true for the RAM.

So lets look at your CPU usage:
1) Run FreeNAS as a data server (minimal impact)
2) Run Plex, likely will need to transcode. Will it be expected to handle 4K media? (high impact)
3) Own Cloud (minimal impact)
4) Three other servers (minimal to medium impact per server)

You add up all the impacts and it becomes significant. I doubt a 2 core CPU will do well.

What would I purchase? 32GB RAM with the ability to upgrade to 64GB and a fast 4 core CPU, remember that every server is going to want some CPU time and you are going to need to provide it. This type of setup also gives you the ability to expand and add more VMs when you feel like it. When I purchased my system I got the max RAM and almost fastest CPU I could obtain without breaking my bank. For me it was a very good investment.

How would I configure it? While you may be able to utilize Bhyve to create your VMs for the servers you desire, I would use VMWare ESXi (free). This can be a significant learning issue but if you find out that Bhyve just isn't working for you then you have an option to fall back on.

it's just my opinion and you know everyone has one. Good Luck.

 

[Pitfrr]

@joeschmuck: In the choice of the CPU, what would be the best to advise here (considering the use cases you listed): focus rather on high CPU clock (any high CPU passmark) or trying to get the highest CPU core?
I would go for the highest CPU core but I'm not sure.... (therefore the C2758)
If using VMWare definitely I would say; running FreeNAS bare metal, I'm not so sure (but probably considering Bhyve... but I don't have much experience there).

 

[pro lamer]

Some sanity chime in from a noob like me:

C2758... :) (just to compare with the C2750)

C2750 has more passmark points. May work better with Plex. Having said that: I haven't considered this

I doubt a 2 core CPU will do well

when writing the above

 

[pro lamer]

... Continued...

VT-d

+1

Excuse me I posted twice - I had some problem with dealing with my mobile phone. The screen started jumping up and down and finally I realized I had already posted something ;P

 

[typhonragewind]

It is best to get too much CPU than too little. Keep in mind everything that you want to do, it's actually quite a lot if it's all happening at the same time. This also stands true for the RAM.

So lets look at your CPU usage:
1) Run FreeNAS as a data server (minimal impact)
2) Run Plex, likely will need to transcode. Will it be expected to handle 4K media? (high impact)
3) Own Cloud (minimal impact)
4) Three other servers (minimal to medium impact per server)

You add up all the impacts and it becomes significant. I doubt a 2 core CPU will do well.

What would I purchase? 32GB RAM with the ability to upgrade to 64GB and a fast 4 core CPU, remember that every server is going to want some CPU time and you are going to need to provide it. This type of setup also gives you the ability to expand and add more VMs when you feel like it. When I purchased my system I got the max RAM and almost fastest CPU I could obtain without breaking my bank. For me it was a very good investment.

How would I configure it? While you may be able to utilize Bhyve to create your VMs for the servers you desire, I would use VMWare ESXi (free). This can be a significant learning issue but if you find out that Bhyve just isn't working for you then you have an option to fall back on.

it's just my opinion and you know everyone has one. Good Luck.

While my Plex won't be required to transcode 4k, i also had my doubts about a 2-core.

I've been trying to find a good new-ish CPU at reasonable price (i agree with finding something good that lasts instead of replacing it every so often) in order to compare it with the C2750, but it's been hard to find a Core series with ECC support.. Edit: found them