It’s time to replace my Plex/VM/file server and could use some guidance. I am based in the US so everything would be sourced from Amazon, Newegg, eBay, etc.. I like the performance, power draw, and value of Ryzen CPUs and would like to modify a Supermicro 4U 24 bay chassis to accept an ATX power supply with a custom rear window, swapping the fans and having a quiet, powerful homelab server. I’m primarily interested if I can squeak by with enough PCIe lanes with Ryzen without having to jump the budget up substantially to Threadripper, EPYC or Xeon to get similar performance but have more pointed questions below.
Requirements:
- ESXI/TrueNas Core All In One
- 10-15 VM’s (TrueNas Core, Plex, large Docker VM, and others) on M.2 NVME SSD
- Large storage pool managed by TrueNas Core. HBA passthrough directly to TrueNas Core VM.
- Future proof with 10Gbe networking
- Ideally a Passmark score above 35k + GPU so transcoding 4k streams is never an issue when needed.
Specs:
OS: ESXI with TrueNas Core AIO
Case: Supermicro 24 bay 4U with SAS2-846EL2 backplane
Motherboard: AsRock Rack X570D4U-2L2T (open to suggestions but this seems to hit most of the marks)
CPU: Ryzen 9 5900x or Ryzen 9 5950x
CPU Cooler: undecided
Case Fans: undecided
ESXI Boot Device: Samsung 32GB USB Flash Drive
Storage: 6x18TB HDDs (RAIDZ2 with the ability to add 3x more vdevs in the future.)
M.2 Slot 1: 2TB Samsung NVME M.2 (For VM’s)
M.2 Slot 2: Empty at first, but are there enough PCIe lanes left to use this down the road without being bottlenecked?
RAM: 4x32GB (undecided model, but current server is 48GB and it is not enough and I am not even currently running TrueNas Core)
HBA: LSI 9207-8i or equivalent (open to suggestions)
GPU: Nvidia RTX 3060ti
PSU: EVGA 850w P2 or 1000w P2
L2ARC: Open to suggestions but from what I read this is not really necessary for Plex streaming, especially with the amount of RAM I plan to have.
SLOG: Open to suggestions here. Is this needed with 10Gbe NICs?
Questions:
1. PCIe Lanes - Does this motherboard/Ryzen combination have enough PCIe lanes for this setup? Between the GPU, HBA, PCIe 4.0 SSD, 10Gbe onboard NICs, I feel like it is cutting it close, and not sure if it leaves any room for future expansion if I need more SSD space (via second M.2 slot or SATA ports). This review of the motherboard breaks it down a little but I am still a bit confused.
[ASRock Rack X570D4U-2L2T Review an AMD Ryzen Server Motherboard](https://www.servethehome.com/asrock-rack-x570d4u-2l2t-review-amd-ryzen-server-motherboard/)
2. GPU Bandwidth - Utilizing both the x16 and x8 slots cuts them both down to x8. Is this a problem for the GPU? There is a lot of discussion online about the 3060ti working just fine on PCIe3.0x16 vs PCIe4.0x16 (1%-2% degradation) but not a lot about x8. Seems to me PCIe4.0x8 theoretically equals PCIe3.0x16 in bandwidth so it should be fine?
3. Thoughts on the vdev arrangement? It is painful losing 2x drives to parity PER 6 drive vdev when coming from hardware raid where you can add drives one at a time.
4. Any other tips/things that stand out would be appreciated!
Requirements:
- ESXI/TrueNas Core All In One
- 10-15 VM’s (TrueNas Core, Plex, large Docker VM, and others) on M.2 NVME SSD
- Large storage pool managed by TrueNas Core. HBA passthrough directly to TrueNas Core VM.
- Future proof with 10Gbe networking
- Ideally a Passmark score above 35k + GPU so transcoding 4k streams is never an issue when needed.
Specs:
OS: ESXI with TrueNas Core AIO
Case: Supermicro 24 bay 4U with SAS2-846EL2 backplane
Motherboard: AsRock Rack X570D4U-2L2T (open to suggestions but this seems to hit most of the marks)
CPU: Ryzen 9 5900x or Ryzen 9 5950x
CPU Cooler: undecided
Case Fans: undecided
ESXI Boot Device: Samsung 32GB USB Flash Drive
Storage: 6x18TB HDDs (RAIDZ2 with the ability to add 3x more vdevs in the future.)
M.2 Slot 1: 2TB Samsung NVME M.2 (For VM’s)
M.2 Slot 2: Empty at first, but are there enough PCIe lanes left to use this down the road without being bottlenecked?
RAM: 4x32GB (undecided model, but current server is 48GB and it is not enough and I am not even currently running TrueNas Core)
HBA: LSI 9207-8i or equivalent (open to suggestions)
GPU: Nvidia RTX 3060ti
PSU: EVGA 850w P2 or 1000w P2
L2ARC: Open to suggestions but from what I read this is not really necessary for Plex streaming, especially with the amount of RAM I plan to have.
SLOG: Open to suggestions here. Is this needed with 10Gbe NICs?
Questions:
1. PCIe Lanes - Does this motherboard/Ryzen combination have enough PCIe lanes for this setup? Between the GPU, HBA, PCIe 4.0 SSD, 10Gbe onboard NICs, I feel like it is cutting it close, and not sure if it leaves any room for future expansion if I need more SSD space (via second M.2 slot or SATA ports). This review of the motherboard breaks it down a little but I am still a bit confused.
[ASRock Rack X570D4U-2L2T Review an AMD Ryzen Server Motherboard](https://www.servethehome.com/asrock-rack-x570d4u-2l2t-review-amd-ryzen-server-motherboard/)
2. GPU Bandwidth - Utilizing both the x16 and x8 slots cuts them both down to x8. Is this a problem for the GPU? There is a lot of discussion online about the 3060ti working just fine on PCIe3.0x16 vs PCIe4.0x16 (1%-2% degradation) but not a lot about x8. Seems to me PCIe4.0x8 theoretically equals PCIe3.0x16 in bandwidth so it should be fine?
3. Thoughts on the vdev arrangement? It is painful losing 2x drives to parity PER 6 drive vdev when coming from hardware raid where you can add drives one at a time.
4. Any other tips/things that stand out would be appreciated!