FreeNAS First Build - 32TB - Backup/Sync/Media server

[b3rny]

Hi,

This month I'm building my first FreeNAS config. The goal is to have a system as quiet and economic as possible (I live in an apartment and the server will sit in the living room on a corner). I also want to have a budget below CHF 2000.- (around USD 2150). After carefully reading the official docs as well as some posts here on the forum and some Blogposts elsewhere on the web (which in retrospect seem to be really bad advice) for the past month, I came up with the following hardware configuration. I would like to get some feedback, in particular if there are areas where I can save some money/invest more money:

Supermicro X11SSH-LN4F: LGA1151, E5-1200v5 (LGA 1151, Intel C236, mATX) 270.–
Intel Xeon E3-1240L V5 (LGA 1151, 2.10GHz) 285.–
Crucial CT4K16G4WFD824A (4x, 16GB, DDR4-2400, DIMM 288) 751.–
4x WD Red 3.5" - 128 MB - 5400 RPM - SATA 6 Gb/s - 178MB/s - Optimized for 1 to 8-bay NAS - NASware 3.0 1'060.–
Fractal Define R5 Titanium (Midi Tower) 119.–

For a total cost of CHF 2485.- (USD 2647), which is a bit above my budget. Prices are from a local re-seller (Digitec) as I am a bit reluctant to order some of these on amazon and have ridiculous returning delays in case something is broken.

The use case for this machine is the following (In order of relevance):

1. Sync server: Up to 3 devices. I haven't decided on the sync software yet. I am leaning towards Syncthing. If you have a better alternative feel free to comment. I currently have a Dropbox pro account that I am planning to cancel as I don't want them to access my private data anymore. I might run multiple sync accounts (see below).
2. Backup server : Up to 4 devices. I haven't decided on the backup software yet. I will see that after I have a better idea of the config. I need to primarily backup home folders from my Android/Linux devices, then a windows device home folder. I also have a significant amount of pictures and finally a large media collection (for a home server, to which I will get below). Personal data requires to be encrypted and absolute security. The media collection not so much. I might run multiple backup accounts (see below).
3. Private Git server low private usage, with basic git wikis.
4. Private VPN server: This one is actually quite important as I am travelling to China quite often.
4. Media server. I have a Plex account and plan on having 1 stream as a typical usage within my home. Transcoding support is required and shouldn't impede use cases 1-3. I might share the access if I have the bandwidth to spare
5. Torrent box. I will probably run a basic torrent headless client such as transmission. This one feeds the media server above.
7. KVM /docker box (?) - I haven't decided on that yet but there is a 60% chance that for professional reasons I will need VMs to test basic stuff such as a Jenkins instance or a basic Ubuntu sandbox with some app inside to show to a client. I might rather use a cloud service for that though. We will see.

As you might have guessed, the Machine is a semi-professional setup : It will primarily be used by myself to backup my data/access my media/code/free internet, but as I am currently building my company I might create separate accounts for the company/some associates for usage 1-3, given that I have the extra bandwidth. Typically up to 3 users with 2 devices each, although I except the typical workload on a weekday evening at peak hours to be : 1 media stream and 2 devices syncing at the same time. Maybe one git commit. Backup is done in the middle of the night when only the torrent client might be running (and probably paused, as backup is absolute priority for me).

Because of the above requirements I will probably put everything behind an Nginx instance and when the need arises implement some basic access control so I can share some of the good stuff, but that's another story.

Note that I have a 1Gbps bidirectional optical fiber connection at home and plan on plugging all 4 Ethernet ports to make full usage of it (I'm not certain it is better than using a single one though, but I assume It ism at least in terms of latency, in a concurrent applications scenario. And its a cheap upgrade).

With the above configuration/requirements I am currently wondering the following:
- What PSU should I use ?
- Is it overkill ? If not, is it unterkill?
- Should I buy a fan for the CPU or is the stock one good enough?
- I am not sure about the case (is it quiet enough?, Is there a smaller one for the above config?)
- I am currently not sure about the pool configuration. Do I need RAIDZ2 ? can someone point me to relevant docs ? Right now I put 4x8TB and assume I will run a RAIDZ2 pool but I mostly do it because that is what everyone around here is doing... That is not a valid reason though (at least for me).
- Is there something I forgot I should be aware of ?

Thanks for the feedback
- Bernard

 

[Nick2253]

If you are looking for a dropbox replacement, a popular option is NextCloud. However, if you are only looking for the sync features, then I agree that Syncthing is probably the best option.

Backup is a tricky problem, and I keep bouncing around to different options. My current favorite is URBackup, but there are many other options out there depending on exactly what you are trying to accomplish. Another option I've looked at is BURP (which is a Bacula fork) for tape support (eventually, I plan to add a tape backup to my rotation, and I'll probably use BURP once I get my tape drive). BackupPC is an interesting option I used in the past, and it was super easy to use because it's clientless and my home network is on a domain, but it didn't support Volume Shadow Copy, so Windows file backups weren't necessarily consistent, which led to some data loss for me (a database on my main desktop was backed up in an inconsistent state, requiring me to recreate it).

- What PSU should I use ?
- Is it overkill ? If not, is it unterkill?
- Should I buy a fan for the CPU or is the stock one good enough?
- I am not sure about the case (is it quiet enough?, Is there a smaller one for the above config?)
- I am currently not sure about the pool configuration. Do I need RAIDZ2 ? can someone point me to relevant docs ? Right now I put 4x8TB and assume I will run a RAIDZ2 pool but I mostly do it because that is what everyone around here is doing... That is not a valid reason though (at least for me).
- Is there something I forgot I should be aware of ?

PSU: I would recommend checking out JonnyGuru, and looking at their PSU recommendations. Seasonic is typically a good brand, but not all of their PSUs are perfect, and many times the high-end OEMs sell to multiple brands (with multiple price points), and you can get a great PSU for a good price.

Overkill: I don't think your build is really that overkill. With all you are doing, you might be better off with a separate hypervisor if you scale much larger than a VM or two.

Fan: I'd recommend staying with the stock fan. No reason to spend the money for a different heatsink.

Case: Quiet is really relative. If you want silent, you'll need to look at some fan upgrades, which will add a lot of cost to your build. I would say it's quiet enough, but only you can really answer that.

Pool Config: I would actually recommend 6x 4TB drives in RAIDZ2. It's the same amount of usable space, and while the 4TB drives are marginally more expensive per TB, you're buying 8TB less storage space. That should help you save a couple hundred dollars.

 

[Chris Moore]

Note that I have a 1Gbps bidirectional optical fiber connection at home and plan on plugging all 4 Ethernet ports to make full usage of it (I'm not certain it is better than using a single one though, but I assume It ism at least in terms of latency, in a concurrent applications scenario. And its a cheap upgrade).

Connecting 4 NICs to the internet is not going to make it go faster. Two is more than enough. If you need faster connectivity for a system at your home, you might want to invest in 10GB networking, but your internet connectivity speed and latency is dictated by the provider circuit.
I would suggest getting a system board that is a couple generations older and uses DDR3 registered memory. It is significantly less expensive and at the moment memory prices are outrageous. For the purposes of a NAS, having a system board that is a couple generations old might cost you a few more watts of power but you don't loose any significant performance because the speed is mostly limited by the number and configuration of the drive pool.

 

[MatthewSteinhoff]

What Nick said.

1. Power supply... I like Seasonic and Corsair. Buy enough watts that it'll handle a case full of drives. More efficient is better than less efficient.

2. Your configuration seems reasonable. It really depends on what you do with VMs and local services. You're probably good.

3. Use the stock CPU fan.

4. Case.... The R5 is a solid choice, good build quality. I have the Fractal Design Node 804. I added a couple Noctua NF-F12 fans then undervolted them for quiet running. It still isn't silent but is not bothersome.

5. With just four drives, you'll get the same amount of usable space from a striped mirror with the added benefit of doubling both IOPS and throughput.

6. One NIC. One NIC only. Adding more increases complexity with little benefit.

Cheers,
Matt

 

[Chris Moore]

A board like this:
https://www.amazon.com/Supermicro-X9SRL-F-Motherboard-Graphics-X9SRL-F-B/dp/B00DTINJ12
Only $267.46 on Amazon right now and the memory will be much less expensive.
Here is a kit of 64GB for only $359:
https://www.amazon.com/Memory-Upgrade-PC3-14900-Registered-1866MHz/dp/B01FZPJQOA
EDIT:
or a kit of 32GB for only $115.51:
https://www.amazon.com/HP-Compaq-Workstation-Registered-PC3-12800-Tech/dp/B00WFDN6QU

and that system board can take up to 256GB of regular registered ECC. If you go with LRDIMM it can be as much as 512GB.

The Xeon E5-2650 v2 (not the L version) would be a good processor to pair with that system board. L version processors are a waste of time.
https://www.amazon.com/Intel-Processor-E5-2650-BX80635E52650V2-Cache/dp/B00F29LG1G

Save on the RAM spend a little more on the CPU. It works out better though, especially when you want more RAM later.

 

[Maelos]

I would suggest, as Chris suggested to me, that you look for used X9 hardware, especially if you can find it in a combo deal. If you look at my signature you will see what I have chosen and the journey I took. Well, I refreshed, and looks like I am a bit behind.

 

[Chris Moore]

I would suggest, as Chris suggested to me, that you look for used X9 hardware, especially if you can find it in a combo deal. If you look at my signature you will see what I have chosen and the journey I took. Well, I refreshed, and looks like I am a bit behind.

In business, I can understand that it is not always possible to go with anything but the, 'latest model,' for business management reasons. For home or personal use or even in business where there is some flexibility, I can't suggest anything newer than the X9 revision system boards with Xeon E5 processors because the newer boards all (if I recall correctly) use DDR4 memory, which is incredibly expensive right now. The registered DDR3 memory, which is almost half the price of DDR4, is the way to go, at least in my opinion. The newer processors and memory go faster, but in most storage implementations, the drive count is the limit on performance regardless of the CPU and RAM.

 

[Maelos]

Edited.

 

[joeinaz]

A board like this:
https://www.amazon.com/Supermicro-X9SRL-F-Motherboard-Graphics-X9SRL-F-B/dp/B00DTINJ12
Only $267.46 on Amazon right now and the memory will be much less expensive.
Here is a kit of 64GB for only $359:
https://www.amazon.com/Memory-Upgrade-PC3-14900-Registered-1866MHz/dp/B01FZPJQOA
EDIT:
or a kit of 32GB for only $115.51:
https://www.amazon.com/HP-Compaq-Workstation-Registered-PC3-12800-Tech/dp/B00WFDN6QU

and that system board can take up to 256GB of regular registered ECC. If you go with LRDIMM it can be as much as 512GB.

The Xeon E5-2650 v2 (not the L version) would be a good processor to pair with that system board. L version processors are a waste of time.
https://www.amazon.com/Intel-Processor-E5-2650-BX80635E52650V2-Cache/dp/B00F29LG1G

Save on the RAM spend a little more on the CPU. It works out better though, especially when you want more RAM later.

I have the exact same board Chris is suggesting. It does use inexpensive DDR3 memory. My 24GB was about $100. The only difference in my configuration is I went with the E5-2650 (v1). It's about 70% of the performance of the V2 and runs less than $75 on eBay.

 

[joeinaz]

I can't suggest the X10 revision system boards because they all (if I recall correctly) use DDR4 memory, which is incredibly expensive. I have been suggesting the X9 revision boards with Xeon E5 processors because they can use registered DDR3 memory, which is significantly less expensive. At least, it is for now.

Chris, My X10SAE is a Socket 1150 and uses DDR3 but it needs UDIMMs which are generally more expensive than DDR3 RDIMMs. I think the X11 boards need DDR4 and require higher end v3 and v4 Xeons.

 

[Maelos]

Some X10s, the newer ones, take DDR4. I think your best bet is just to confirm it here: https://www.supermicro.com/products/motherboard/

 

[Chris Moore]

Chris, My X10SAE is a Socket 1150 and uses DDR3 but it needs UDIMMs which are generally more expensive than DDR3 RDIMMs. I think the X11 boards need DDR4 and require higher end v3 and v4 Xeons.

I should have known that making a blanket statement would bite me. I remember that now. On the Xeon E3 compatible boards, when they went from the X9 to X10 series, changed the CPU socket and some other features of the board, like going from PCIe 2.0 to PCIe 3.0, but they didn't change to DDR4. The X11 series boards (in the Xeon E3 lineup) is where they went to DDR4 but all the boards that can take Xeon E3 processors (X9, X10 and X11) do not handle registered (RDIMM) memory. The Xeon E3 and the other processors compatible with those sockets (like the Pentium and i3) can't handle the truth, I mean RDIMM memory. The socket 2011 boards, when equipped with a Xeon E5, are able to use Registered (RDIMM) or Load Reduced (LRDIMM) memory. Again, I am sure there is a case where that is not true, so my blanket statement will probably come back to bite me... LRDIMMs might not be cheaper, I have not checked, but the regular RDIMMS (in DDR3) are the best price right now (March 2018) even compared to Unbuffered (UDIMM) DDR3 an almost half the price of DDR4. The 2011 socket Xeon processors are really inexpensive at the "v1" revision because they are a little oldish, but the 2011 socket Xeon E5 v2 chips are still a little on the pricey side compared to a new Xeon E3. The big advantage is the low cost memory and a Xeon E5 v2 should still run circles around a Xeon E3. The tradeoff is that the Xeon E5 is likely going to draw more power and make more heat. There are so many models to choose from that you can probably fine a situation where that isn't the case, but I can't know all the answers and many times I am going from memory, and I don't have ECC memory in my head.

 

[b3rny]

Hi,

Thank you all for your answers.

- For the PSU I am currently looking at Seasonic Platinum 400 Fanless (400W) (I assume it is enough power)

- For the Sync : Thanks @Nick2253 I will look into NextCloud as well. I haven't tried any of these but the usage will manly be dropbox replacement. Feel free to let me know your experiences with sync software

- For the Backup : Thanks @Nick2253 I will also look into URBackup. I am currently looking for something that can backup my android devices as well so I can finally get rid of all the "backup" crapware shipping with android/apps that actually does dubious stuff with your data.

- For the Pool config : I am confused, @Nick says 6x4TB is cheaper and possibly better (?). My understanding is that the fewer drives, the less likely one of them will go down. The price for 6x WD Red (4000GB, 3.5", NAS, Storage system) is 720.- instead of 940.- for 4x8TB so that's a pretty big discount. Also there was talk of doing a stripped mirror instead of RAIDZ2 @MatthewSteinhoff. Is that compatible with zfs ? I was under the impression that only 'ZX' Raid extensions were supported... I need to do more research. Right now my only goal there is to make sure I get at least 12TB (16 would be great) of arguably "unkillable" storage for as little as possible. I did 4x8 because I assumed with the board I chose I can always double that 2 years down the line if I need more storage.

- Finally for the older gen vs new gen chipset debate : I understand the concept as I am doing the same thing for Graphic Cards/laptops (previous generation high-end >> current generation middle-end). Unfortunately I have trouble finding DDR3 cheaper than DDR4 where I live. Keep in mind that I want to avoid shady no-name resellers from abroad in case I get a DOA/warrenty issues. I just saw newegg now ships to switzerland so I have to look into that but I have no experience with their customer service. Another point of contention with this solution is that if you check the above link, you will notice that branded DDR3 memory sticks (Crucial/Kingston) are actually only marginally cheaper than DDR4 sticks. The ones with a real discount are no-name sticks... (BlackDiamond, I don't know that brand or how good the stuff they ship is).

Anyway, I understand and agree with the concept, I'm just not sure it is applicable to me and where I live in this case... I have to do some more research.

I will come back with an upgraded config when I know more.

Thanks
- B

 

[Chris Moore]

For the Pool config : I am confused, @Nick says 6x4TB is cheaper and possibly better (?). My understanding is that the fewer drives, the less likely one of them will go down. The price for 6x WD Red (4000GB, 3.5", NAS, Storage system) is 720.- instead of 940.- for 4x10TB so that's a pretty big discount. Also there was talk of doing a stripped mirror instead of RAIDZ2 @MatthewSteinhoff. Is that compatible with zfs ? I was under the impression that only 'ZX' Raid extensions were supported... I need to do more research.

This slideshow will help you understand the options:
Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

 

[Maelos]

I have also found that the manual (User Guide), which I am reading now from front to back, repeats and has much of the same information in a lot of these threads/slideshows.
http://doc.freenas.org/11/system.html . We are always here if you have specific questions.

 

[Nick2253]

I will look into NextCloud as well. I haven't tried any of these but the usage will manly be dropbox replacement. Feel free to let me know your experiences with sync software

Some people have issues with NextCloud's sync capabilities (e.g., the app suddenly won't sync properly, and the only fix is monkeying with the server). However, not everyone has these issues. If you are syncing everything on a LAN, you'll probably be great with NextCloud. If most of your syncing is over the internet, SyncThings might be needed. Many users run both (though it takes some tweaking to make them both work properly) if they need more robust syncing with the drop-box features of NextCloud.

I am confused, @Nick says 6x4TB is cheaper and possibly better (?). My understanding is that the fewer drives, the less likely one of them will go down. The price for 6x WD Red (4000GB, 3.5", NAS, Storage system) is 720.- instead of 940.- for 4x10TB so that's a pretty big discount.

You are correct that, in general, the fewer drives in a vdev, the more robust it is. However, the decrease in reliability from 4x to 6x drives isn't terribly significant, especially when you have two parity drives (RAIDZ2). The biggest benefit, obviously, is the savings.

 

[Chris Moore]

finding DDR3 cheaper than DDR4 where I live

It isn't the difference between DDR3 and DDR4, the price break comes in when you are able to pickup registered DDR3 memory that has been surplussed from a data-center. You can get it for half the price of new memory and there is nothing wrong with it. To use registered memory, you must use a Xeon E5 processor. Take a look at the links. You don't have to buy from them, I was just showing you the model information. If you insist on buying new hardware, you will spend a lot more money than you need to spend.

- For the PSU I am currently looking at Seasonic Platinum 400 Fanless (400W) (I assume it is enough power)

Don't assume, read this thread:
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

 

[joeinaz]

Hi,

Thank you all for your answers.

- For the PSU I am currently looking at Seasonic Platinum 400 Fanless (400W) (I assume it is enough power)

- For the Sync : Thanks @Nick2253 I will look into NextCloud as well. I haven't tried any of these but the usage will manly be dropbox replacement. Feel free to let me know your experiences with sync software

- For the Backup : Thanks @Nick2253 I will also look into URBackup. I am currently looking for something that can backup my android devices as well so I can finally get rid of all the "backup" crapware shipping with android/apps that actually does dubious stuff with your data.

- For the Pool config : I am confused, @Nick says 6x4TB is cheaper and possibly better (?). My understanding is that the fewer drives, the less likely one of them will go down. The price for 6x WD Red (4000GB, 3.5", NAS, Storage system) is 720.- instead of 940.- for 4x10TB so that's a pretty big discount. Also there was talk of doing a stripped mirror instead of RAIDZ2 @MatthewSteinhoff. Is that compatible with zfs ? I was under the impression that only 'ZX' Raid extensions were supported... I need to do more research. Right now my only goal there is to make sure I get at least 12TB (16 would be great) of arguably "unkillable" storage for as little as possible. I did 4x8 because I assumed with the board I chose I can always double that 2 years down the line if I need more storage.

- Finally for the older gen vs new gen chipset debate : I understand the concept as I am doing the same thing for Graphic Cards/laptops (previous generation high-end >> current generation middle-end). Unfortunately I have trouble finding DDR3 cheaper than DDR4 where I live. Keep in mind that I want to avoid shady no-name resellers from abroad in case I get a DOA/warrenty issues. I just saw newegg now ships to switzerland so I have to look into that but I have no experience with their customer service. Another point of contention with this solution is that if you check the above link, you will notice that branded DDR3 memory sticks (Crucial/Kingston) are actually only marginally cheaper than DDR4 sticks. The ones with a real discount are no-name sticks... (BlackDiamond, I don't know that brand or how good the stuff they ship is).

Anyway, I understand and agree with the concept, I'm just not sure it is applicable to me and where I live in this case... I have to do some more research.

I will come back with an upgraded config when I know more.

Thanks
- B

With large drives, it's preferred to use RAIDZ2 due to the risk of drive failure during the time it takes to rebuild a drive after failure. If you assume the base overhead of 2 disks using RAIDZ2 array consider 3 options:

4x 6TB raw = < 12TB usable after Z2
6x 4TB raw = < 16TB usable after Z2
10x 2TB raw = < 16TB usable after Z2

Looking at the three options above, the ten 2TB disks likely offer the lowest purchase cost, highest performance and the fastest rebuild times. If your case has limited slots for disks, the six 4TB disks is a better choice.

 

[MatthewSteinhoff]

ten 2TB disks likely offer the lowest purchase cost, highest performance and the fastest rebuild times.

Highest performance? Fastest rebuild times? I'm not sure that's true. With RAIDZ2, your performance is pretty much that of the slowest drive in the VDEV.

Cheers,
Matt

 

[Chris Moore]

Looking at the three options above, the ten 2TB disks likely offer the lowest purchase cost, highest performance and the fastest rebuild times. If your case has limited slots for disks, the six 4TB disks is a better choice.

If we limit the discussion strictly to the drives, because if we don't it becomes very complicated, then ZFS performance is primarily influenced by the speed of an individual drive and the number of vdevs in the pool. I did the math on this earlier in the day today, and I am going from memory, meaning it may not be precise. A 10TB HGST SAS drive has a mechanical performance of about 230MB/s of data transfer. Using 24 of those in mirrored pairs, gives 12 vdevs and the combined performance of about 3680 IOPS. Use the same 24 drives but configure them in 3 RAIDz2 vdevs with 8 drives each and the IOPS would be somewhere between 1577 and 690, but because of parity calculations in RAIDz2 and other factors, it would likely be on the low side of that. It is hard to nail that down accurately just using what I know.
For talking purposes, if you were to put all 24 drives in a single RAIDz2 vdev (not recommended) you would only get the IOPS of a single drive. That would be about 230 IOPS and the reason for that is because ZFS is designed to write to all the vdevs in parallel, so more vdevs equates roughly to more IO based on the slowest drive in each vdev. It is really simple for mirrors, but with all the RAIDz levels (1, 2, 3) it becomes more complicated because of the way data is replicated within the disk group for redundancy purposes and the calculations become very complex. In simple terms, ZFS is trying to use all the drives in each vdev equally for the IO to that vdev and if you have more than one vdev, it is trying to equally use all the vdevs. It wants every player to have an equal opportunity, but it does have algorithms to favor fast vdevs.
If you want to dive deeper, here is a link to a blog post by Matt Aherns (co-founder of the ZFS project):
https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz