Hardware tips for mid-high end home media server

[dirtynas]

Hi friends,

New user here. Somewhat technically literate, but still lost in the server and NAS spaces. Most simple guidance I find is for more affordable builds, so I could really use some help.

I am trying to build an all-in-one Home Media Server. For now, I am trying to understand which hardware to purchase, especially the critical upgrade limiting pieces like motherboard and case/chassis.

Currently considering Truenas Scale for this build.

Budget: 3-5000 US dollars (flexible)

Workloads:

• plex for high quality video (VR 8k)
• high quality video transcoding
• hosting 1 (or more?) game server (Minecraft, 'ARK: Survival Evolved')
• docker/container support/loads (nothing incredibly intensive)

NAS needs:

• 12-16 of 3.5 inch bays for 18TB HD (currently own 4 new 18TB WD Red Pro). Plan to install 4 drives now, but can do more if necessary
• support for 128-256 GB ram, where between 64-128 GB is installed now (I assume ddr4 is best)
• data protection is emphasized (ecc ram, anything similar)
• I suppose I want IPMI, though I do not understand it well
• I have no idea how to state my networking needs -- will need to access remotely

My Personal Philosophies for this Build:

• It is worth spending more on 'Quality of Life', security, data protection, and simplicity
• Selecting components that are not limiting to future upgrade possibilities is a plus
• Space considerations are nice, but ultimately are less important
• Used hardware can be considered, but if replacing the components are difficult from a technical perspective, I'd prefer to avoid it

Thanks for reading this far! Specific models/purchase links where possible are hugely appreciated. Advice for motherboards especially would be useful, but support for any of the following is ultimately beneficial:
- motherboard, cpu, ram, boot drive, fans/cooling, power supply, networking stuff, stuff-I-dont-know-I-need

 

[DigitalMinimalist]

If you can get this W680 Mainboard:

MW34-SP0 (rev. 1.0) | Workstation Motherboard - GIGABYTE Global

12th Gen Intel® Core™ Desktop Processor Single processor, LGA 1700Intel® W680 Chipset Dual Channel DDR4 3200MHz UDIMM, 4 DIMMs 1 x 2.5G LAN port (Intel® I22...

www.gigabyte.com

Paired with an Alder/Raptor Lake CPU and up to 128GB ECC DDR4 UDIMM.
13700K CPU with Quicksync for transcoding.

Alternative: AM4 Ryzen + dGPU

 

[Etorix]

NAS needs:
12-16 of 3.5 inch bays for 18TB HD (currently own 4 new 18TB WD Red Pro). Plan to install 4 drives now, but can do more if necessary

12-16 drives could be divided in two 6-8 vdevs in raidz2. Beware that it is possible to add further vdvs but that raidz# vdevs cannot be enlarged, so you need to decide on the final geometry and buy 2-4 more drives before creating the first vdev!

support for 128-256 GB ram, where between 64-128 GB is installed now (I assume ddr4 is best)

For such amounts, RDIMM is the best choice, so look for Xeon Scalable or EPYC platforms—quite possibly second hand.
W680 is "too new", and there's no DDR5 ECC UDIMM for it yet.

I suppose I want IPMI, though I do not understand it well

That's simple: It provides remote administration. You just log into a web page and manage the server from there…
Comes standard with any server-grade platform which supports RDIMM anyway ;)

I have no idea how to state my networking needs -- will need to access remotely

As fast as you want in your home network (10 GbE? more?).
For remote access, set up a VPN. Do not expose the NAS to the Internet.

 

[Davvo]

You might go server rack as well.

 

[dirtynas]

12-16 drives could be divided in two 6-8 vdevs in raidz2. Beware that it is possible to add further vdvs but that raidz# vdevs cannot be enlarged, so you need to decide on the final geometry and buy 2-4 more drives before creating the first vdev!

Good foresight. I will try for two 8 vdevs in raidz2, with 8 18tb to start.

For such amounts, RDIMM is the best choice, so look for Xeon Scalable or EPYC platforms—quite possibly second hand.
W680 is "too new", and there's no DDR5 ECC UDIMM for it yet.

What about one of the following two motherboards?
- GIGABYTE MD72-HB0 Extended ATX Server Motherboard Dual Socket P+ Intel C621A ->https://www.neweggbusiness.com/...
- SUPERMICRO MBD-X12SPA-TF-O Extended ATX Server Motherboard LGA 4189 Intel C621A -> https://www.newegg.com/...

Neither has 16 sata connections though. Would it be a problem for these motherboards to connect 16 3.5 inch Hard Drives?

Would using boards lacking 16 sata connections bottleneck a build with truenas scale and/or zfs in some way?
For the MD72 it mentions '2 x 7-pin SATA 6Gb/s ports' and '3 x SlimSAS with 12 x SATA 6Gb/s ports' for a total of 14 of the 16 needed drive slots. I assume the 12 x SlimSAS and the 2 x 7 pin slots are fine to use, but is there a performant way to connect more than 14 drives?

For remote access, set up a VPN. Do not expose the NAS to the Internet.

Are there any limitations related to hardware that I need to consider for setting up a VPN for remote access? Anything I have to ensure the motherboard includes or necessary network related peripherals?

 

[dirtynas]

You might go server rack as well.

I am under the impression home builds like this are better in cases (for heat/sound reasons), but I do not have a strong opinion about it. Although I would likely consider a vertical rack mounted on the wall if possible if I were to use a rack.

 

[ChrisRJ]

Are there any limitations related to hardware that I need to consider for setting up a VPN for remote access? Anything I have to ensure the motherboard includes or necessary network related peripherals?

I would strongly recommend to handle the VPN not in TrueNAS, but your router/firewall.

 

[ChrisRJ]

Neither has 16 sata connections though. Would it be a problem for these motherboards to connect 16 3.5 inch Hard Drives?

If your case does not have a backplane to support the required number of disks, you can just install a simple HBA. For additional information, please have a look at "Don't use RAID" from the "Recommended readings" in my signature.

 

[Etorix]

What about one of the following two motherboards?
- GIGABYTE MD72-HB0 Extended ATX Server Motherboard Dual Socket P+ Intel C621A ->https://www.neweggbusiness.com/...
- SUPERMICRO MBD-X12SPA-TF-O Extended ATX Server Motherboard LGA 4189 Intel C621A -> https://www.newegg.com/...

Any server motherboard will do, but i woudn't pick any of these two. X12SPA is a workstation motherboard, which is certainly capable but maybe not the first choice as a server. Dual Xeon would be a very big server, with LOTS of VMs.
I thought more of an X11SP_ motherboard for 1st/2nd generation Xeon Scalable because there will be more opportunity to get a second-hand CPU. Actually, even the older X10 generation and Xeon E5 may be worth considering.

Neither has 16 sata connections though. Would it be a problem for these motherboards to connect 16 3.5 inch Hard Drives?

With so many drives, one would expect a HBA to handle part or all of the drives.

I am under the impression home builds like this are better in cases (for heat/sound reasons), but I do not have a strong opinion about it.

Rackmount servers are typically very noisy but 16 drives are not going to be quiet, no matter the case! Do not expect to put 16 drives in a big E-ATX tower case and forget that The Beast stands under your desk, or in your bedroom.

If you can put the server away from ears, going rackmount is the easiest option. If so, I'd suggest to research past and present storage servers from Supermicro/Dell/HPE (at least to collect model numbers) and see if you can find a refurbished server or case (eBay or other).
A rackmount case will solve all questions about PSU and fans. In the best case, you'll find a whole server and that will mostly be it (possibly with extra RAM or replacing a RAID controller with a HBA). Else if it's "just" a case (with matching PSU/fans/backplane), then you'll go back to picking a motherboard to put in.

 

[dirtynas]

I would strongly recommend to handle the VPN not in TrueNAS, but your router/firewall.

Thanks @ChrisRJ for this comment. I will assume moving forward that a proper VPN setup does not require any special hardware considerations when building/buying a server, and rather the VPN should come from a separate solution with software/hardware related to my router/firewall

 

[dirtynas]

Thanks @Etorix and @Davvo for the recommendations for a pre-built used server rack, that seems to be the way to go. I have decided to be somewhat flexible and search for deals for 12-36 bay systems, where 12 bay can be seen as a short-term device and 36 bay is seen as a long-term device. I mostly am only considered builds with no storage installed, since few have high memory density (18TB HD).

I mostly looked through ebay for now but feel free to recommend better places to look.

I have managed to find the following builds after narrowing down obvious bad candidates, please share your opinions and/or your best/worst picks in the following, or share builds you think are better.

Show : $3020 -- Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 192GB 9361-8i w/ RAILS

Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 192GB 9361-8i w/ RAILS | eBay

Manufacturer Supermicro. Rails Included. Form Factor Rack Mount. If this happens, don't worry, we will stand behind our products! Storage NO HDD Installed | 12x LFF + 2 SFF Trays Included. Backplane BPN-SAS3-826EL1.

www.ebay.com

- 192GB DDR4 RAM Installed (12 x 16GB PC4-2133P) -> Maybe not ECC or RDIMM
- 2x Intel Xeon Gold 6140 24.75MB 140W LGA3647 (36-Cores Total)
- X11DPi-NT
- 12 bays

Show : $2506 -- Dell R730xd 24 Bay 2U 2x E5-2673 V4 2.3GHz 20C 512GB H730p 2x 1.8TB 10K 12G

Dell R730xd 24 Bay 2U 2x E5-2673 V4 2.3GHz 20C 512GB H730p 2x 1.8TB 10K 12G | eBay

Find many great new & used options and get the best deals for Dell R730xd 24 Bay 2U 2x E5-2673 V4 2.3GHz 20C 512GB H730p 2x 1.8TB 10K 12G at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- R730xd 24x Bay
- 512GB DDR4 RDIMM ECC Registered Memory (16 X 32GB)
- 2x Intel Xeon E5-2673 V4 2.3GHz 20 Core 55MB Cache 135W Processor
- 24 bays

Show : $3,310 -- Supermicro X10 4U 32 Bay LFF 4-Node 2x 10-Core E5-2640v4 128GB 2x NVMe Per Node

Supermicro X10 4U 32 Bay LFF 4-Node 2x 10-Core E5-2640v4 128GB 2x NVMe Per Node | eBay

Find many great new & used options and get the best deals for Supermicro X10 4U 32 Bay LFF 4-Node 2x 10-Core E5-2640v4 128GB 2x NVMe Per Node at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- X10DRFR-NT,CSE-F424AS-R1K28BP
- 4 nodes, total specs of them combined listed below
- 512GB DDR4 RAM Installed (32x 16GB PC4-2133P) -> Maybe not ECC or RDIMM
- 32 bays
- 8x Intel Xeon 10-Core E5-2640v4 (2 per Node)

Show : $4198 -- 4U 36 Bay SAS3 Server Xeon Skylake 2.6Ghz 28 Cores 256GB DDR4 2x 960GB NVME M.2

Pre-Built 4U 36 Bay SAS3 Server Xeon Skylake 36 Cores 256GB DDR4 1x 960GB NVME | eBay

Find many great new & used options and get the best deals for Pre-Built 4U 36 Bay SAS3 Server Xeon Skylake 36 Cores 256GB DDR4 1x 960GB NVME at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- X11DPH-T
- 256GB DDR4 ECC REG -> Maybe not RDIMM
- 2x Intel Xeon 6132 2.60GHz 14C Skylake (total 28 Cores)
- NIC: * Integrated Dual LAN with 10GBase-T with Intel® X722 + X557
- 36 bays

Show : $2160 + unknown shipping -- Supermicro 2U 24-Bay SFF 4-Node X10DRT-P 8x E5-2640v4 80-Core 256GB RAM & Rails

Supermicro X10drt-p Dual Socket R3 LGA 2011 Server Motherboard Node for sale online | eBay

Find many great new & used options and get the best deals for Supermicro X10drt-p Dual Socket R3 LGA 2011 Server Motherboard Node at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- X10DRT-P,CSE-217HQ+-R2K04B
- 8x Intel Xeon 10-Core E5-2640v4 (2 per Node)
- 256GB DDR4 RAM Installed
- 24 bays

 

[Etorix]

At this point, it may help to think about how many VMs/apps you intend to run, and how many cores you'd want. "More" is an easy answer, but that's usually contradictory with high clocks and SMB (if that's how you'll share) mostly needs high clock but few cores.

Too many CPUs is probably not helpful, unless you have many apps. And multiple nodes means that you can't pool all those bays.
R730xd: 2' bays in 2U must be 2.5"
Of your first list, that leaves the Supermicro 12 bays (replacing RAID controller with plain HBA) and 36 bays. Either would make a high-end NAS.

 

[elorimer]

All this for a home video server?

I don't see here how much storage is needed, where the content is coming from, how many simultaneous videos will be served up, and what the network demands are. Are we doing direct play to 8k TVs, or transcoding down? Also: backup.

 

[dirtynas]

Too many CPUs is probably not helpful, unless you have many apps. And multiple nodes means that you can't pool all those bays.

Fair point, I have opted for builds with less CPU with higher thread speed. Of each of the following, I am leaning towards the Dell R740XD, as it has an IPMI and larger power supplies, with some seemingly better networking stuff.

Show : $3870 -- Dell R740XD 12 LFF (2)Gold 6140 18-C 2.30Ghz 512GB H730P (2x)10G QSFP+ (2)1100W

Dell R740XD 12 LFF (2)Gold 6140 18-C 2.30Ghz 512GB H730P (2x)10G QSFP+ (2)1100W | eBay

Find many great new & used options and get the best deals for Dell R740XD 12 LFF (2)Gold 6140 18-C 2.30Ghz 512GB H730P (2x)10G QSFP+ (2)1100W at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- 512GB - (16x)32GB DDR4 Registered DIMM memory module -> Maybe not RDIMM
- (2x) Intel Gold 6140 18-Core 2.3GHz 25MB 140W LGA3647 P
- Dell PowerEdge R740XD
- 12 bays
- Remote Access Integrated IPMI 2.0
- Power (2x) 1100W 100-240V 80 Plus Platinum Hot-Pluggable PSU

Show : $3020 -- Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 192GB 9361-8i w/ RAILS

Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 192GB 9361-8i w/ RAILS | eBay

Manufacturer Supermicro. Rails Included. Form Factor Rack Mount. If this happens, don't worry, we will stand behind our products! Storage NO HDD Installed | 12x LFF + 2 SFF Trays Included. Backplane BPN-SAS3-826EL1.

www.ebay.com

- 192GB DDR4 RAM Installed (12 x 16GB PC4-2133P) -> Maybe not ECC or RDIMM
- 2x Intel Xeon Gold 6140 24.75MB 140W LGA3647 (36-Cores Total)
- X11DPi-NT
- 12 bays

Show : $3880 -- Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 512GB 9361-8i w/ RAILS

Supermicro 12 Bay Server X11DPi-NT 2x Gold 6140 36C 512GB 9361-8i w/ RAILS | eBay

Manufacturer Supermicro. Rails Included. Form Factor Rack Mount. If this happens, don't worry, we will stand behind our products! Storage NO HDD Installed | 12x LFF + 2 SFF Trays Included. Backplane BPN-SAS3-826EL1.

www.ebay.com

- 512GB DDR4 RAM Installed (16 x 32GB PC4-2133P) -> Maybe not ECC or RDIMM
- 2x Intel Xeon Gold 6140 24.75MB 140W LGA3647 (36-Cores Total)
- X11DPi-NT
- 12 bays

To complete the above builds I also need the following hardware. Am I missing anything? Will the HBA below work? I heard the HBA needs to have been flashed correctly in order to be used. Cheaper listings do not mention this flashing

Show : $140 -- LSI9300-16i 16-port SAS9300-16I 12GB/s Host Bus Adapter 03-25600-01B

LSI9300-16i 16-port SAS9300-16I 12GB/s Host Bus Adapter 03-25600-01B | eBay

Find many great new & used options and get the best deals for LSI9300-16i 16-port SAS9300-16I 12GB/s Host Bus Adapter 03-25600-01B at the best online prices at eBay! Free shipping for many products!

www.ebay.com

- any decent 2.5 inch ssd as a boot drive (seems the Dell build is missing a SFF slot though, so I suppose some adapter is necessary)
- 6 HDD for the first vdev

 

[Davvo]

any decent 2.5 inch ssd as a boot drive

I am using Seagate's IronWolf

 

[spanik]

I'm using a Supermicro H11SSL-i as my desktop. It has 16 SATA ports (and a single M.2), up to 2TB DDR4 ECC and IMPI. Network is only dual 1GbE and a separate 10/100 for the IMPI. Ticks some of your boxes. Case is a Coolermaster CM Stacker. Space for 12 3.5" sata HD's in cages of 4 with a fan at the front. Very obsolete as this is the 4th pc I build into it. But they still have comparable cases. Heavy tower case but can be fitted with castors :) Can take a second PSU.

 

[jgreco]

Very obsolete

I wouldn't call anything with DDR4 "very obsolete" just yet.

 

[spanik]

It's the case that is "very obsolete ;) Almost 20 years since it came on the market. But it remains a good box if you need to put a lot of 3.5 HD's in it.

 

[jgreco]

To complete the above builds I also need the following hardware. Am I missing anything? Will the HBA below work? I heard the HBA needs to have been flashed correctly in order to be used.

So long as you're aware that the PERC H730P and LSI 9361-8i controllers mentioned in the above listings WILL NOT be compatible with TrueNAS, and I also suspect that the 10G "QSFP+" (QSFP+ is actually 40G, and the Mellanox stuff is not really the best choice) may be problematic.

I spent a bunch of time looking into this due to Intel QuickSync not being generally available on the Xeon platform. I wanted a recent-ish CPU to experiment with and settled on building a "little hypervisor" out of an E-2388G on one of Supermicro's SYS-510T-WTR platform, maxxed out at 128GB of RAM. This is a Rocket Lake CPU so it covers lots of QuickSync options.

 

[jgreco]

It's the case that is "very obsolete ;) Almost 20 years since it came on the market. But it remains a good box if you need to put a lot of 3.5 HD's in it.

Mmm. Fine. But be aware that SATA 3Gbps has poor signal integrity, so there can be risks using old cases.