The $1000 FreeNAS 2019 prosumer build. Will it FreeNAS? Will it TrueNAS?

[Mr. Slumber]

Ok, some thoughts... Yes, you can build a FreeNAS system for less. And yes, you can spend much more money on a FreeNAS build.
Here's what I want to do: I want to build a FreeNAS system which only purpose is being a reliable filesserver! So no jails, no plugins, no Plex/Emby or something like that just pure FreeNAS. It shall be upgradable (storage- and ram-wise), it must be a rack server and being just a little quiet (compared to enterprise grade stuff) would be nice... (I'm not planning to install this in my living room but not wanting another screamer in my Startech rack ;))
Chasing unicorns? yeah, maybe, but let's see :)

Being used to enterprise grade hardware and loving the easy maintenance and the reliability of it I for sure could have just bought a new e.g. Dell R740 (way above 1000$) or e.g. an older Dell R710 (cheap, realiable but a little noisy under load) and upgrade it a little bit. Or I could've started with pure consumer grade hardware to stay in budget. But I thought maybe one could get a little bit of both worlds thats why I kept thinking of a "prosumer" build. Yeah I know we could debate for hours about this topic alone ;) I'm sure there a 1000 different ways to reach that goal but here's a way I thought of. Of course this build contains no storage HDDs because the individual storage needs may be very different from e.g. the 200GB photo archive backup dude to the 80TB data hoarder. (I only use enterprise grade Seagate Exos X10 ST100000NM0086 10TB drives. Take a look @Backblaze statistics :) They come with a 5 year warranty so no problems with bad drives and on this side of the pond they are just 5-10 bucks more than the WD RED NAS HDDs) EDIT: Exos 10 HDDs not recommended anymore (why? please take a look here)

So I'm really looking forward to your comments, productive criticism, thoughts and tips. Thank you in advance :)



Case: Intertech 4U 4129-N (in the US it's Norco RPC-450 4U, but it's sold there without the hot pluggable 120mm fans) (good airflow and lots of upgrade space)

Case rear fans: 2x Noctua NF-A8 PWM (silence please)

Backplane: 2x ICY DOCK FatCage MB155SP-B 5 Bay (room for up to 10 HDDs, fans have to be exchanged to silence them see below, no enterprise grade, still there's room for a third one = up to 15 HDDs in the case)

Backplane fans: 2x Noctua NF-A9 FLX (take the heat out in silence)

PSU: bequiet Dark Power Pro 11 550W (platinum grade efficiency, very silent, lots and lots of sata/sas rails for HDDs, 5 years warranty)

Motherboard: Supermicro X11SSM-F (it's Supermicro )

CPU: Intel Core i3-7320 (no jails, no VMs, no Plex/Emby, not more than 4 concurrently users using this machine)

CPU Cooler: Noctua NH-D9L (silent ice ;))

RAM: 2x 8GB ECC Ram Samsung M391A1K43BB1-CRC (to be upgraded later)

HBA: LSI 9211-8i (for up to 8 HDDS/SSDs, already flashed to IT mode)

Boot: 1x Intel SSD SSDSC2KW128G8X1 545s 128GB (slow write, cheap, 5 years warranty)

(open front view of the Intertech 4U 4129-N case, still more room for another ICY DOCK FatCage MB155SP-B 5 Bay in the middle section)

(detailed view from left to right: LSI 9211-8i (HBA), CPU Cooler: Noctua NH-D9L, PSU: bequiet Dark Power Pro 11 550W)


Thanks for your feedback. Will it FreeNAS? EDIT late march 2019: yes, definitely

EDIT 2019.03.19: added HBA and changed Boot
EDIT 2019.03.26: changed rackmount case
EDIT 2019.04.13 added some photos
EDIT: 2020.11.24 Exos 10 HDDs not recommended anymore (why? please take a look here)

 

[Danielx]

Have you considered a used x9 motherboard you could probably have 128gb ECC Ram and a faster processor and still be in your budget.

For the fans I suggest going with 4 pin fans if you want to replace them. I have a rosewill case the fans aren't loud at all.

 

[Mr. Slumber]

Have you considered a used x9 motherboard you could probably have 128gb ECC Ram and a faster processor and still be in your budget.

No didn't thought about this but thanks for mentioning.

 

[Mr. Slumber]

For the fans I suggest going with 4 pin fans if you want to replace them. I have a rosewill case the fans aren't loud at all.

Well, I got the case without the fans for 20bucks less so it was a pretty good deal.

The Noctua NF-A8 PWM come with 4 pin connectors (and a lot of adapters so the fit nearly everywhere).

Thanks for sharing that the built-in fans of the Rosewill case aren't loud that was something I wasn't sure about.

 

[Mr. Slumber]

CPU: Intel Core i3-7320 (no jails, no VMs, no Plex/Emby, not more than 4 concurrently users using this machine)

This let's me scratch my head a little still... Having read the real fantastic Hardware guide this CPU is for light usage of freenas. Ok, understood. But will it be enough e.g. to handle the backups of all my VMs (3 VMs together around 500GB which have to be backuped every night)

Thanks for your input :)

 

[Yorick]

Yes, that CPU will handle your backups just fine. If you’re not doing jails or VMs, then CPU use will be light.

You’ll be gated by the speed of the NICs more than CPU, for sure.

 

[Mr. Slumber]

Yes, that CPU will handle your backups just fine. If you’re not doing jails or VMs, then CPU use will be light.

You’ll be gated by the speed of the NICs more than CPU, for sure.

Thank you, I'm glad to here this. Well, the onboard NICs are an Intel brand (I've read here that Intel NICs are way better with FreeNAS) so hoping they will run just fine. But of course limited to 1Gb/s.

Link aggregation should be possible but as I've learned, 2x 1Gb/s NICs will not equal 2Gb/s transfer speed. Correct me if I'm wrong but it will help in case of many users simultaneously accesing the FreeNAS server (speaking >50 users) but in my case this won't happen.

But if one day I'll find some money laying around this build can be upgraded with a 10 Gb/s NIC. I like this kind of flexibility of the build. Of course: for satisfiyng a 10Gb/s card one will need to have more than just 1 vdev as I've understood so far so if one day I'm switching over to 10Gb/s then I'll also have to add some more disk/build some more vdevs, right?!

 

[Yorick]

Just about right on all counts. Yes, for a LAG to be useful you need multiple devices pulling data simultaneously. There’s no hard and fast rule, but 12 or more ought to do it. This is because a LAG load shares, it does not load balance. A “hash” is used to decide which client uses which link. A single client uses a single link. Two clients have a 50-50 chance of using the same link.

How you get use out of a 10Gb/s link depends a bit on your use case. Loads and loads of small files that get accessed over and over again? More RAM and maybe an L2ARC will help. Async writes? Yeah you need multiple vdevs so the disks can keep up. Reads with a dataset that doesn’t fit into ARC? Same.

Consumer 2.5Gb should be “around the corner”, a couple years at most. 5Gb might make an appearance too.

I have found that for my use case, 1Gb is more than enough. I’ll probably go mGig once it’s affordable just because I can, but there’s no actual need for it. I store backups and stream movies. The backup takes under 10 minutes incremental every day. A full backup, which happens only when the disk layout changes, takes a few hours: And for my use, there’s no meaningful difference between it taking 5 or 2 hours. Either way I’m not staring at it and it completes well within time.

 

[Mr. Slumber]

How you get use out of a 10Gb/s link depends a bit on your use case. Loads and loads of small files that get accessed over and over again? More RAM and maybe an L2ARC will help. Async writes? Yeah you need multiple vdevs so the disks can keep up. Reads with a dataset that doesn’t fit into ARC? Same.

Thanks for this explanation!

I’ll probably go mGig once it’s affordable just because I can, but there’s no actual need for it.

Yes, same over here. My VM backups of my Proxmox host take about 30-50min every night so 10Gb/s would be fun but not a "must have" for my usecase.

 

[Elliot Dierksen]

This is because a LAG load shares, it does not load balance. A “hash” is used to decide which client uses which link. A single client uses a single link. Two clients have a 50-50 chance of using the same link.

Indeed! This is one of my soapbox points regarding use of a LAGG. It also matters what hash algorithm your switch uses, and if you can change the selection. To get good distribution, the LAGG facing FreeNAS should typically be distributed based on source MAC/IP or a combination of source and destination. Most switches default to using destination which would give you almost no distribution between links at all. That assumes your clients and the FreeNAS are on the same layer 3 network. If your clients are on a different layer 3 network, then your switch would need to be able to do distribution by IP instead of MAC address.

 

[Yorick]

Really fancy switches can do L4 hash :). On eBay, such kit is actually affordable for home use. New, not so much, unless your home also has a Tesla X.

 

[Constantin]

I'd also consider a 2-core X10SDV-2C-7TP4F. Sure, it's $100 more than your solution but it offers a on-board HBA with 16 SAS / SATA channels, dual SFP+, and a 2-core, 4 thread CPU that can turbo to 2.6GHz but whose TDP is only 25W. All that for about $450. You don't get the ability to exchange processors, the thing is only running 4 threads, but as a SOHO file server, it will likely be 100% OK and serve up SMB with alacrity (seeing that clock speed is important for that protocol).

The SFP+ port is preferable in my book because it gives you the flexibility of using an inexpensive twinax DAC for short distances and a inexpensive fiber transceiver for longer distances. They even have copper wire transceivers now but I don't consider them cost competitive. Add a inexpensive Mikrotik switch with SFP+ and you get a solution that allows multiple clients to max out their gigabit connections to the server at a very low price point, without the complications (and in my case failures) associated with LAGGed interfaces.

Because it's a Supermicro, there are lots of boards in that family with fewer built-in options (SFP+, HBA, etc.) that can be had for less. Have a look and consider them.

 

[Mr. Slumber]

I'd also consider a 2-core X10SDV-2C-7TP4F.

Thanks for your suggestion! But I pulled the trigger just some hours before on the X11SSM-F as posted above... But for my usecase your suggestion would have a been a real real good match. But I was too impatient... ;)

 

[Mr. Slumber]

The SFP+ port is preferable in my book because it gives you the flexibility of using an inexpensive twinax DAC for short distances and a inexpensive fiber transceiver for longer distances. They even have copper wire transceivers now but I don't consider them cost competitive. Add a inexpensive Mikrotik switch with SFP+ and you get a solution that allows multiple clients to max out their gigabit connections to the server at a very low price point, without the complications (and in my case failures) associated with LAGGed interfaces.

I'll keep this in mind when it's time for the next step towards 10Gb/s... :)

 

[Mr. Slumber]

a 2-core, 4 thread CPU that can turbo to 2.6GHz but whose TDP is only 25W. All that for about $450

Wow, just took another look at the technical data and this for me seems to be an even better CPU package compared to the Intel Core i3-7320 (which I still quiet like very much ;)) with it's 2 cores (same), 4 threads (same), no turbo (worse), base frequency 4.10 GHz (better) and TDP 51W (nearly double! in comparison). The TDP is quiet an important corner stone over here for a machine running 24/7 and having nearly the highest electricity prizes worldwide. (just did a quick calculation: this +26W TDP difference results in a $71 electricity bill "upgrade" per year )

 

[Constantin]

(just did a quick calculation: this +26W TDP difference results in a $71 electricity bill "upgrade" per year )

Careful. The delta is unlikely to be a constant. Given that either CPU is likely to remain much at idle for most of the year, a better comparison would be the idle power consumption. ServeTheHome has tests for the D-15xx series, not sure they covered the i3 chips also. I'd research that a bit further.

 

[Mr. Slumber]

The delta is unlikely to be a constant.

You're right, idle will be the main task of the cpu ;). But next time: "Order not so fast I must. Patient I must be..." ;) before I buy my server parts.

 

[SweetAndLow]

Wow, just took another look at the technical data and this for me seems to be an even better CPU package compared to the Intel Core i3-7320 (which I still quiet like very much ;)) with it's 2 cores (same), 4 threads (same), no turbo (worse), base frequency 4.10 GHz (better) and TDP 51W (nearly double! in comparison). The TDP is quiet an important corner stone over here for a machine running 24/7 and having nearly the highest electricity prizes worldwide. (just did a quick calculation: this +26W TDP difference results in a $71 electricity bill "upgrade" per year )

Tdp is a measurement of max heat output not really of electricity cost. Those 2 CPUs will cost you the exact same every month to run 24/7.

 

[Mr. Slumber]

Tdp is a measurement of max heat output not really of electricity cost. Those 2 CPUs will cost you the exact same every month to run 24/7.

Had to "enlighten" myself speaking of TDP but after some reading I think I understand why you're right! Thanks! :)

 

[Mr. Slumber]

HBA. I wanted to ad a HBA card for up to 8 drives more later in the build but after reading again this very recommended Hardware recommendations Guide and having read many good things about the LSI 9211-8i I decided that I already will buy one. Found lots on eBay and bought this preflashed one LSI 9211-8i IT Mode
Correct me if I'm wrong but I thought that this model hasn't had to be flashed to IT mode but as I found out it also has to be flashed (updated to firmware 20) too and for me I decided that I would save me the "flashing trouble" (also it is not so hard according to some very good short guides here in the forum) and so I bought a preflashed one.
So with this card and my X11SSM-F motherboard I now can connect up to 16 drives. 15 can be fitted in my case so ok, plenty room for storage upgrades in the future. :)