Homeserver built for NAS, VMs, Docker

[thimplicity]

Hi everyone,
I will be moving from QNAP to a DIY home server build and I am in the process of choosing the components, which seems to be very easy, as there is not a lot of choices available . My planned setup is:

• Proxmox as hypervisor
• TrueNAS virtualized
• Windows 10/11 virtualized (used maybe once or twice a month)
• MacOS virtualized (used maybe once or twice a month)
• approx. 10 docker applications in portainer (incl. Plex)
• Plex would need to stream a max. of three 4k sessions at the same time, but this might grow with 8k coming in the next 5-7 years :)

This is all I have planned so far, but I would also leave a little room to grow in the hardware I buy now. All those items shall run on NVMe SSDs (incl caching). My plan was to use "traditional" SSDs, but those take quite some room in the enclosure, and before digging into the topic, I had never heard of NVMe.

I will start with 4 drives, but would like to extend to 8 in the future. I would like this build to last 7-10 years without a complete rebuild if possible. I think I will not need more drives, as I will replace them on a regular basis with larger disks anyways. I would like to keep the enclosure as small as possible, so I was not sure whether to set the limit at 6 or 8, but the mATX standard for motherboards seems to be more widespread and also flexible, so I plan to go with that (until someone here convinces me otherwise). I also ask myself the same question as quite some people online: buy a recent CPU or buy an older Xeon from eBay.

My current favorite is the Intel Xeon E5-2650 or something similar, as it seems to be powerful enough for what I want to achieve.

My current hardware choices:

• Case: fractal design node 804
• CPU: Intel Xeon E5-2650
• Motherboards: tbd
• Graphics card: tbd
• 32 GB ECC RAM

I would also like to be able to extend the build in the future with:

• Additional NVMe: Xiwai 4X NVME M.2 AHCI to PCIE Express 3.0 Gen3 X16 Raid Card VROC Raid0 Hyper Adapter|Computer Cables & Connectors| - AliExpress
• 10GBit Ethernet card
• Graphics card
• Additional SATA card

As most mATX boards only have 3 PIC slots, I would need to go with a processor that has integrated graphics. Is there a similar CPU to the one I mentioned that has integrated graphics?

Before I dig deeper into the component choices - am I on the right track when it comes to the CPU type (cores etc) being powerful enough, enough RAM for now and extendability?

Thanks in advance!

​

 

[NugentS]

its as good as a piece of string is long.
ie it depends on being to see the future
Most Xeon don't have integrated graphics - look for some E3, but those are more like consumer chips and may lack PCIe lanes for your NVMe drives

Warning "caching" doesn't do what you think it does and doesn't work the way you think it might.

I suggest you do a lot more research into hardware. Start with Hardware Reccomendations

 

[thimplicity]

its as good as a piece of string is long.
ie it depends on being to see the future
Most Xeon don't have integrated graphics - look for some E3, but those are more like consumer chips and may lack PCIe lanes for your NVMe drives

Warning "caching" doesn't do what you think it does and doesn't work the way you think it might.

I suggest you do a lot more research into hardware

Thanks - Will do that. Seems like I need to also understand the topic of PCIe lanes better

 

[Arwen]

Yes, many PCIe slots don't do bifurcation. For you example, taking a 16 lane slot and being able to use 4, 4 lane NVMe drives. Which basically means you want the CPU, (or it's chipset if the lanes are coming from it), to support 4, 4 lane PCIe devices instead of 1, 16 lane device.

Their are some NVMe PCIe cards that include a PCIe switch chip, to take 4 - 16 lanes and make 2 or 4, 4 lane NVMe slots. But, if plugged into a reduced lane slot, and using the maximum number of NVMe drives, performance may be limited. (Of course, limited performance on a NVMe drive may be way better than SATA...)

So, make sure any NVMe PCIe card you buy, does what you need. The cheap ones will likely require slot bifurcation.


Note that AMD Epyc has lots of PCIe lanes, 128 total. Not all may be exposed for user use, as some might be for local peripherals, like network or BMC / IPMI. But the Epyc board I bought has 4 x 16 lane PCIe slots, with a bunch of other lanes in OCuLink connectors on the board.

 

[thimplicity]

Thanks for the feedback so far! As recommended by @NugentS, I dug a little more into the hardware side of things. Below are the components that will be the same, no matter which one I choose above:

PSU	550W Gold PSU - $75 - Brand recommendations welcome!
RAM	64GB brand to be decided ECC (2 x 32GB) DDR4 - $400
CPU fan	Noctua ... - $80
Boot-SSD	SAMSUNG 980 SSD 1TB M.2 NVMe - $110
Case	Fractal Node 805 / Fractal Design 5 - $150

From an HDD perspective, I will continue to use the 4 * 4TB Ironwolf that I already have in my QNAP today.

Now for the real four options I came up with based on what was in the hardware guide and what I have seen in forums:

	EPYC (SP3)	Ryzen (AM4)	Intel Xeon (FCLGA2011)	Intel Xeon (FCLGA1151)
CPU	7251 /7261 - $250	3600X - $250	E5-1650V4 - $175	E-2146G - $175
Motherboard (mATX)	Asrock Rack ROMED6U-2L2T - $510	Asrock Rack X470D4U - $320	Asrock Rack EPC612D4U-8R - $410	SuperMicro X11SCH-F - $350
Pros	10GbE onboard, plenty PCIe lanes, PCIe 4.0, IPMI, future-proof	PCIe 4.0, best run cost, fastest RAM click speed		Integrated graphics for Plex encoding, IPMI
Cons	Worst single thread rating, most expensive	16 PCIE lanes, no 10GB LAN, PCIe 3.0	40 PCIE lanes, no 10GB LAN, PCIe 3.0, run cost	16 PCIE lanes, no 10GB LAN, PCIe 3.0
Total price	$1,575	$1,325 (10GbE card in addition)	$1,330 (10GbE card in addition)	$1,245 (10GbE card in addition)

Based on the comparison I really like the EPYC setup, as my impression is that it would be the most future-proof of the four with PCIe 4.0, plenty of PCIe lanes and I could just replace the CPU when necessary in the future. At the same time, the clockspeed and therefore single thread performance is not great. The Ryzen seems to be the best choice out of the gate, but the PCIe lanes seem to limit expandability quite a bit when it comes to an NVMe card, 10 GbE, HBA, graphics card? The E-2146G has the biggest advantage in having a graphics card included, but as I habe IPMI in all boards, this is only relevant for Plex encoding, which would need to happen on the clients (Apple TVs). The E5-1650V4 seems to be a good mix, but is based on older tech and is in comparison to the AMDs not really that much cheaper. I find the AMD options more attractive, and as they only phase out the respective sockets in 2022, I would have 2-3 more years of CPUs I could put into the boards.

What do you guys think?

 

[ChrisRJ]

Have you looked at used enterprise gear? Something 4-6 years old will still provide more than enough speed for what you want to do.

 

[NugentS]

A minor point but

SAMSUNG 980 SSD 1TB M.2 NVMe - $110

is way overkill. TN needs a 16GB boot drive, and thats mostly empty. Get a 16GB Optane M10 from Ebay and use that instead. Its a lot cheaper.
---------------------------
As @ChrisRJ says try looking at second hand kit, from a dismantler (on ebay), not from a random John Smith on ebay selling off his overclocked for the last 5 years CPU. Its how I have built my NAS (although I did buy a new SuperMicro motherboard, rather than second hand)

Mind you prices seem to be going up - certainly RAM is anyway.

I had to move to twin CPU just to get enough PCIe lanes (never considered Epyc)

Also consider that SMB is single threaded, so fewer faster cores works better for this. But then as you are running VM's / Docker maybe it ain't quite that simple

 

[thimplicity]

A minor point but

SAMSUNG 980 SSD 1TB M.2 NVMe - $110

is way overkill. TN needs a 16GB boot drive, and thats mostly empty. Get a 16GB Optane M10 from Ebay and use that instead. Its a lot cheaper.

As I want to run TrueNAS as a VM in Proxmox, my idea was to partition the 1TB drive and make a portion available as TrueNAS boot.

Also consider that SMB is single threaded, so fewer faster cores works better for this. But then as you are running VM's / Docker maybe it ain't quite that simple

I did not know that - thanks!

Have you looked at used enterprise gear? Something 4-6 years old will still provide more than enough speed for what you want to do.

Which processor would you choose used for my needs?

 

[NugentS]

The Xeon is an older CPU (Q2'16, Broadwell) - but is only single socketed, so you won't get a full board of PCIe slots. Those PCIe lanes get used up rapidly.
Have a look at the supermicro boards, they usually have a huge choice and some have 10Gb (copper) on board

There is (unfortunately) no right answer as there are too many variables of cost, requirements, location and availability. There are however lots of wrong answers - which we can try and help with.

 

[ChrisRJ]

Which processor would you choose used for my needs?

I guess it will largely depend on the details of Plex and the VMs. Perhaps it makes sense to come up with a spreadsheet to gather more details, like what is going to happen in the VMs. Whether the latter will run an accounting program or something more demanding will make a huge difference. As to Plex, I have no experience with it. But the forum search should help you here.