Updated New Build for Home NAS

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Hi,

I posted my original question about the build here but the idea changed quite a lot and I decided to post new thread.

Case: Fractal Design Meshify C ATX Mid Tower Case
Motherboard: Supermicro MBD-X11SSM-F Micro ATX LGA1151 Motherboard
CPU: Intel Xeon E3-1270 V5 3.6 GHz Quad-Core OEM/Tray Processor
Memory: 2x Samsung 16GB DDR4 2Rx8 2400 MHz, ECC Unuffered
PSU: SeaSonic CORE GM 650 W 80+ Gold Certified Semi-modular ATX Power Supply
CPU Cooling: Cooler Master Hyper 212 Black Edition 42 CFM CPU Cooler
Thermal Paste: ARCTIC MX-4 2019 Edition 4 g Thermal Paste

I've got all bookmarked ready to order, can I click order? :)

In addition to above I have 1TB WD RED I am planning to buy 2x 4TB WD REDs.

I learned recently that ZFS doesnt allow to add efficiently more drives (you can but there will be more redundancy). What would be usable disc size from those 9TBs? (4,4,1).

This will be mainly for FreeNas with Nextcloud and maybe some more plugins for learing. I will have Plex as well so I hope this will be able to transcode at least 1 4k.

I found also MB's that are called "MBD-X11SSM-F-O" but couldn't find what's "O" stands for on the Supermicro website, any ideas?

Thanks,
Pawel
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
I have already bought RAM. I found good deal on 2x16GB Samsung M391A2k43BB1-CRCQY.
 

Mark Levitt

Explorer
Joined
May 21, 2017
Messages
56
I learned recently that ZFS doesnt allow to add efficiently more drives (you can but there will be more redundancy). What would be usable disc size from those 9TBs? (4,4,1).

OK, quick and dirty background on ZFS... In ZFS, you create a storage pool. Storage pools are made up of one or more vdevs (virtual devices). A single vdev can correspond to a single physical disk, a pair of disks in a mirror, or a group of disks that provide redundancy against the loss of a physical disk (like RAIDz1, RAIDz2, RAIDz3). There are different levels of redundancy. RaidZ1 is the least redundant where you can only loose 1 drive in the pool without loosing any data. You can create a raidz pool with only three vdevs (here I am assuming each vdev is a single physical disk and they are all of equal size). A raidz2 pool allows you to loose 2 drives in the pool without loosing any data, but you need at least 5 drives of the same capacity.

You can mix different size vdevs in a single pool, but you cannot have vdev made up of disks of different size. In other words, your two 4TB drives can be mirrored, but your single 1TB drive will be a single vdev. If you create a storage pool using your mirrored vdev plus your 1TB drive vdev, I believe you would loose all of the data in the pool if your single 1TB drive fails. You could create two storage pools, one with your mirrored 4TB drives and a second one with your single 1TB drive. Then, if you loose the 1TB drive, you only loose the data on that pool.

You'd be much better off purchasing a third 4TB drive and create a raidz vdev storage pool. Then you could loose any one of the drives in the pool and still save your data.

Assuming drives of equal size, you can calculate how much space you'll have here:

Regarding the MBD-X11SSM-F vs MBD-X11SSM-F-O, I think they're the same board, but it may be one comes in retail packaging with the i/o shield and cables, etc and the other doesn't.

For your SSD boot drive, the motherboard you're specing has 8 SATA ports. You'd be better off using one of them for the boot drive vs connecting the SSD to a USB 3.0 adaptor.

Performance wise, I don't think you'll have any problem running plex and nextcloud.
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Thanks Mark for the detailed answer.

Okay, this means I need to buy 3 x 4TB physical drives. I will get 8TB of usable storage from these three drives (RAIDZ).

Is there any way I can make use of my 1TB and still have some data redundancy? The only way that comes to my mind now is to buy two more 1TB's and have another pool. This would be quite expensive just for 2TB unfortunately.

The other questions is, how I can add additional drives to the already existing pool? Again, the only way that comes to my mind is to copy all data somewhere else create new pool from old and new drives and then copy all data back.
 

Mark Levitt

Explorer
Joined
May 21, 2017
Messages
56
You could buy one more 1TB drive and create a mirrored pair. You'd only have 1TB of usable space with your two 1TB drives, but you would be able to loose one of the drives without loosing the pool. I believe you could then add this additional vdev to the pool with the other three 4TB drives and you'd have 9TB of space. Then you could, in theory, loose a single drive from the 3 drive raidz vdev and a single drive from the mirrored 1TB drives vdev without loosing your data.

I'm not an expert so you might want to research further, but no, you can't add additional drives to an existing vdev. You could do as you suggest and copy the data somewhere, delete the pool and recreate it with more drives, and copy the data back. A second option is to add additional drives in mirrored pairs and then add those to the existing pool. Finally, it is possible to expand the size of the pool by replacing the existing drives one at a time. I.e., you replace the first 4TB drive with an 8TB drive and let the pool resilver (rebuild the redundancy). Then replace the second one, then the third. Once you've replaced all of the 4TB drives with larger drives, the pool can be expanded with the additional space on the drives.

There are pros and cons to each approach. For the first, you'll want to think about where you're copying the data to. If it's to a single drive, then until you recreate the pool and copy the data back, if that drive fails you've lost everything. The second option is safest, but also has the greatest cost in terms of cost per TB. Also, you'll need to have a case big enough and with enough power for adding multiple drive pairs. For the third option, resilvering the pool is a disk intensive process and takes a long time. It's possible the added activity on the existing drives could cause one of them to fail before pool had rebuilt the new drive. If that happens, you've lost the pool. So, in short, you'll need to have enough spare space on another system to make a backup of the data and you'll need to ensure that backup itself is safe.

Obviously, the level of risk depends on what you're storing. There's a different between some movies that could be replaced vs. family photos vs. accounting records for a fortune 500 company.

Personally, I built my server with 6 4TB drives, giving me about 15.5TB of usable space using raidz2. After a few years, I still have about 10TB free (this is storing media for Plex, Time Machine backups for a household full of Macs, and a few other things). My goal was to massively overbudget on the amount of storage I needed so I wouldn't have to worry about it for a long time.
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Thanks Mark, taking into account all what you said above I'll go with buying 3x4TB's + additional 1TB (to mirror existing one) + Sata DOM Boot.

I think 9TB's should be enough for 2 PC backups + home photos and videos + some Plex stuff.

Thanks for mentioning the 'resilvering' method. That's may be (not-risk-free but still) way to go in future when expanding storage.

I'll probably wait for Black Friday to see if I can get any deals on chosen items.

Looking forward to any other opinions regarding my build! I don's see much critics therefore I'm quite worried.

In the meantime, slightly different question. I've got Netgear D7800 router that claims you can get up to 2.53 Gbps combined Wi-Fi speed (800 Mbps for 2.4 GHz and 1733 Mbps for 5GHz). Thinking just about the 5G in theory it should achieve 1733 Mbps = 216 MB/s. How much of this do you think I should be able to achieve on my laptop? (assuminig no other devices transferring and no interference from other Wi-Fi in neighborhood).

I am asking this question because it looks like the bottleneck will be the router which has only 1GBit ports. This means it will be able to achieve only 1000 Mbits or 125 MB/s in theory. As long as I can upgrade Motherboard with 10Gbit port the bottleneck will still remain the router.

Is there any way to get 2 or 3 Ethernet ports on router to revieve data? (I am using OpenWRT)
 

jayecin

Explorer
Joined
Oct 12, 2020
Messages
79
The CPU will struggle to transcode a 4k movies on the fly depending on the bitrate, compression etc.

Intel Xeon E3-1270 V5 - PassMark score of 8458

CPU Requirements

The most basic thing to remember is that the more Plex apps you have playing content at the same time, the more CPU power you’ll need. Generally speaking, if you have two Plex apps requiring transcoded content at the same time, that will require about twice the CPU processing power compared to if there was only one app playing content.


If you want very basic minimum suggestions:


  • No transcoding: Intel “Atom” 1.2GHz (NAS devices based on ARM processors should also be capable of at least one stream with no transcoding)
  • Single 720p transcode: Intel Core i3 3.0 GHz
  • Single 1080p transcode: Intel Core i5 3.0GHz
  • Single 4K transcode: Intel Core i7 3.2GHz

If you’ll need to support more than one simultaneous transcode, you’ll need a more powerful processor.


Related Page: NAS Devices

The Guideline

Very roughly speaking, for a single full-transcode of a video, the following PassMark score requirements are a good guideline for the following average source file:


  • 4K HDR (50Mbps, 10-bit HEVC) file: 17000 PassMark score (being transcoded to 10Mbps 1080p)
  • 4K SDR (40Mbps, 8-bit HEVC) file: 12000 PassMark score (being transcoded to 10Mbps 1080p)
  • 1080p (10Mbps, H.264) file: 2000 PassMark score
  • 720p (4Mbps, H.264) file: 1500 PassMark score

The CPU Benchmark website is a good resource to see what sort of PassMark score a particular processor received.
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Thanks for answer jayecin,

Unfortunately I've got a couple of constrains that probably (I assume, I may be wrong) won't allow me go much better that the Xeon E3-127x-ish processors.

The first is CPU ECC Support. I want this NAS to serve primarily as a data storage for good few years. Therefore I'd like to have the CPU that supports ECC which rules out all i5's and i7's (unless I am missing something). The most powerful Core i3's are only around 9500 passmark.

The second constrain is the cost. I don't really want this NAS to go over £600 (excluding drives). I estimate currently this will cost me around £560. I can't see any Xeon's E3's that goes above 9276 (9276 is Intel Xeon E3-1285 v6 @ 4.10GHz). The E3-1270 seems good compromise in cost-to-performance ratio. If I'd like to go above 1000 passmark then this means better Motherboard and CUP from new Xeon series.

I'd like to go better with passmark but cant see much room to go much better without stretching the cost significantly.

I would like be wrong on above and be missing some gap that would allow mi to go above 10000 passmark.

Looking forward to your feedback.

P.S. The only gap I can think of is having the Hardware supported acceleration in FreeNAS. As far as I have found, FreeNAS doesn't support GPU H/W acceleration. If at some point in future FreeNAS will support it, I may buy additional GPU.
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Something that I will have a lookup tomorrow is the combo of Xeon E5 and some Supermicro board supporting FCLGA2011.
 

jayecin

Explorer
Joined
Oct 12, 2020
Messages
79
Yes hardware transcoding done on the CPU is something else to consider, currently FreeNas does not support this, however the move to FreeBSD 13 on TrueNasCore does provide a pathway to support it. FreeBSD 13 supports the latest Intel iGPUs drivers and in theory should make it possible to utilize the hardware transcoding available on 9th and 10th generation Intels. However the Intel Xeon E3-1270 V5 you are using does not include an integrated iGPU which means you wont have that ability. https://www.intel.com/content/www/us/en/products/processors/xeon/e3-processors/e3-1270-v5.html The processor needs to support Intel® Quick Sync Video

This is why i chose to build with an i7 and non ecc memory. I understand that data integrity is critical, however I dont believe that ECC mitigates enough risk to make it the holy grail a lot of NAS builders perceive it to be. ECC corrects the random flipping of a single bit for data contained within the memory itself. ECC is primarily used for high critical applications and processes where the difference between a 1 and 0 can be $100,000 or $1,000,000 or the difference between a 5 minute software crash for a company earning $100,000 a minute or not. Random bit flipping while not 100% understood is primarily believed to be caused by electrical interference, like the massive amounts of EMF generated in a data center with hundreds of high density servers stacked as close as possible utilizing thousands of watts of power for each one. These are also environments where the amount of data being processed per second is on a magnitude far far greater than your home NAS will ever do in its entire life time. So with ECC what are you really getting? You are getting protection from a lottery statistic chance that if a single bit of information stored in your memory randomly flips it will get corrected and in the event that you would also happen to write that single bit to the hard drive. Its an absolutely astronomically small chance that ECC will ever protect your data in a meaningful way and even without it, whats really going to happen? 1 bit is flipped, a single pixel on your family photo turns from light grey to dark grey? 1 ms of static in your favorite MP3? Because thats really all thats going to happen, there are still many other checks and safety measures built into hard drives, cpus etc that work to mitigate and/or correct errors and their are way more reasonable ways to lose data that money would be better spent on.

But otherwise it looks like a solid build, I use that same case for my main gaming rig and I love it. They did just come out with the Meshify C 2 in case you didnt already own the case and were buying new, might be worth checking out improvements made.
 
Last edited:

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Thanks for reply @jayecin,

Regaring the first paragraph, I don't think I will rely on the support of external GPU or processors GPU. This doesn't seem to be well supported as you mentioned and even with the iGPU this is definitely not plug-and-play solution.

Regarding the ECC memory, I had a good think and few reads about it and it looks like you are probably right on what you've written. I am saying probably because it looks like there is a lot of controversial info regarding the ECC. Controversial from the point of view how bad the bit flip may affect ZFS and NAS overall. From the overall cost and bit flip statistics point of view the ECC is probably not required for my use as you said.

Altrough, I would still like to go with ECC. The reason is, the file storage will be the main task of this server. Plex and live transcoding will be the second of ever futher. I know myself and this server will probably serve me for next good few years (5-8?). Therefore I'd like to make it as good for the data as I can. And just have this feeling that there is no 'gap' in my approach to file storage. Overall ECC + RAIDZ5 + UPS + (maybe) some offline copy of the data will give me very good server for years.

Now, I did look for Xeon X5 and some Supermicro motherbord and I am quite surprised. I can basically get the 1.5 times passmark number with not that more money. I have found X10SRH-CLN4F-O for £280 and Xeon E5-2680 v3 for the same price as my original processor. Overall for £80 I am getting much better passmark and option to upgrade for best 2011R3 CPU in few years.

Am I missing something?
 

jayecin

Explorer
Joined
Oct 12, 2020
Messages
79
Id say the E5-2680 v3 is a solid choice then, should be able to brute force some 4k transcoding and give you lots of CPU power for any other expansion in the future.
 

pawel30w

Dabbler
Joined
Nov 7, 2020
Messages
19
Thanks jayecin,

Can anyone please recommended what's best single process Supermicro MB for Xeon E5?

Many thanks,
Pawel
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,909
I am not sure it fits with your performance requirements. But I have recently bought a couple of X9SRI-F that cost me less than 200 Euros with an E5 2870 (8 cores). And I could find quite a few 16 GB modules of DDR3 ECC RAM for 20-25 Euros per piece. But my use-case is lab virtualization for Kubernetes, so it may or may not be a fit. Also, these boxes will only run on demand and not 24x7, so power efficiency was less of a concern.
 

elorimer

Contributor
Joined
Aug 26, 2019
Messages
194
Re the 1TB drive: you could use it to back up datasets on your main pool. Presumably on your main pool you will have several datasets. Some subset of those datasets (the highest priority) could be backed up to the 1TB drive and provide a backup in case your main pool dies. Or, you could optimize your plex files (transcode in the background) and copy them to the 1TB drive for streaming. If the 1TB drive fails, you still have your main pool, so it is less important that it be mirrored, etc.
 
Top