System build review for TrueNAS, Mediaserver and Frigate NVR

[kiloohm]

Hi guys, I've spent the last 2 weeks reading page after page of hardware realted thread. There is a lot of information but being completely new to TrueNAS, I'm not sure if I'm headed in the right direction. The last system I built from scratch was probably a Pentium II so I'm rusty.

For starter, I'm trying to replace the following:

Dell Precision T7600, dual E5-2687W, 64 GB RAM, 14TB hard drive running ESXi (1 VM with a NFS share and a few other small test Linux VMs)
Intel NUC Exteme 12 running Proxmox VE (Plex and NZB downloader, Home Assistant, Frigate NAS)
QNAP TS-219P II

The T7600 used to be my main virtualisation machine. However, Plex transcoding was awful so I moved my Media Server to my NUC and then started playing with Home Assistant and Frigate.

I would like to decommission the T7600 to reduce my carbon footprint and rebuild everything on a single machine so that I can get back my NUC for other purposes. I would run TrueNAS Scale native and other VMs on it, hoping to passhtrough the iGPU to Plex and the Coral M.2 to Frigate.

• Motherboard: X12STH-F or X12STL-F ($357 USD)
• CPU: Intel Xeon E-2388G ($612 USD)
• Memory: 2 x 32GB ECC MEM-DR432MD-EU32 ($150 USD)
• Case: Fractal Design Node 804 ($125 USD)
• PSU: Vertex 850W GX Series 80+ Gold ($200 USD)
• Storage: 4x16TB WD RED Pro in RAIDZ2 --> ($900 USD)
• Boot Device: Kingston A400 120GB SSD SATA3 ($15 USD)
• Onboard M.2: Coral M.2 B+M EdgeTPU (repurposed from Intel NUC)
• Networking: Qlogic 57810S dual SFP+ (repurposed from T7600)

These are my questions right now and of course I'm open to other feedback.

Motherboard selection
Is there a benefit from one model listed over the other? I'm leaning towards the X12STH due to the additional SATA connectors. In the setup above, I would be using 6 SATA connectors so that leaves 2 free for future expansion before I need an additinal card.

RAM Sizing
I need about 32GB of RAM for the VMs to run efficiently. Would the remaining 32GB be sufficient for TrueNAS or should I bite the bullet and go to 128GB right away?

PCIe slots configuration
I would like to keep the PCIe x16 slot available. I have a Nvidia GPU in my Intel NUC at the moment that I may want to move. On Frigate, I use the Coral EdgeTPU for AI video processing but I may still want a GPU for transcoding. I have QuickSync allocated to Plex so I assume I can't share it also with Frigate.

That leaves me with 1 x PCI-E 4.0 x4 (in x8 slot) and 1 PCI-E 3.0 x4 (in x8 slot). The network card is PCI-E 2.0 x8 and I also have a PERC H710 in IT mode in my Dell T7600 that is also PCI-E 2.0 x8. Any recommended slot assignment or I should look at replacing those 2? I won't need the HBA right away, I just want to future proof this solution since this is a fairly large investment.

Additional storage
Do I need additional drives for the VMs or can I install them on the above RAID-Z2 storage pool?

 

[jgreco]

The last system I built from scratch was probably a Pentium II so I'm rusty.

That's probably fine.

Building, Burn-In, and Testing your FreeNAS system

I've been meaning to post some guidance here for a while now. We frequently see people come to the forums with hardware problems that should have washed out in the system build process, but since many of the users here are DIY'ers without...

www.truenas.com

• Motherboard: X12STH-F or X12STL-F ($357 USD)
• CPU: Intel Xeon E-2388G ($612 USD)

One of the few sensible options IMHO for this; Intel has gone off in a direction that makes this all harder.

• Networking: Qlogic 57810S dual SFP+ (repurposed from T7600)

This is not a good option for TrueNAS. Please see the 10 Gig Networking Primer for the list of good options, which really just boils down to "Intel X5xx/X7xx" and "Chelsio 520" based cards.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Is there a benefit from one model listed over the other? I'm leaning towards the X12STH due to the additional SATA connectors. In the setup above, I would be using 6 SATA connectors so that leaves 2 free for future expansion before I need an additinal card.

You have named families of mainboards, not specific boards. They're different form factors and the PCH is different. The X12STH is a C256 chipset with various form factors, while the X12STL is C252; the STL also comes in mATX form factor (X12STL-F) and mITX (X12STL-IF) form factor.

I need about 32GB of RAM for the VMs to run efficiently. Would the remaining 32GB be sufficient for TrueNAS or should I bite the bullet and go to 128GB right away?

Your choice of SCALE favors going larger on the RAM. ZFS on Linux does not interact well with Linux and ARC memory sizing is not automatic. iXsystems recommends using no more than half of the available memory for ARC. This generally means it is better to buy more.

Do I need additional drives for the VMs or can I install them on the above RAID-Z2 storage pool?

RAIDZ2 is not good at block storage; you can absolutely run a small number of non-busy VM's on it but don't expect anything stellar.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[kiloohm]

You have named families of mainboards, not specific boards. They're different form factors and the PCH is different. The X12STH is a C256 chipset with various form factors, while the X12STL is C252; the STL also comes in mATX form factor (X12STL-F) and mITX (X12STL-IF) form factor.

I was specifically referring to the the X12STH-F (not the X12STH-LN4F or another variant) or the X12STL-F (not the X12STL-IF). I'm leaning more towards the X12STH-F because of the additional SATA ports.

This is not a good option for TrueNAS. Please see the 10 Gig Networking Primer for the list of good options, which really just boils down to "Intel X5xx/X7xx" and "Chelsio 520" based cards.

I'm surprised that the BCM57810S gets a bad rep here. I've used those forever in FreeBSD (pfSense) and never had a hiccup. Based on the 10G primer, the only reference is for the 57710 and is marked as hit or miss. Given that I already have this card, I guess my best bet is to try it but plan to replace if it I run into any issue.

Aside from that, I will look to max out the RAM to 128GB and add 2 mirror SSD for OS/VM. I have 1.3TB provisioned on my current OS mirror drives on ESXi so 2x2TB would suffice and would be fairly cheap.

This whole project is a large investment so I will start by acquiring drives as I see sales. That will also give me a chance to acquire them from different batches.

 

[jgreco]

I'm surprised that the BCM57810S gets a bad rep here. I've used those forever in FreeBSD (pfSense) and never had a hiccup. Based on the 10G primer, the only reference is for the 57710 and is marked as hit or miss. Given that I already have this card, I guess my best bet is to try it but plan to replace if it I run into any issue.

I have some original Alteon Tigon I and II cards in inventory, so no one can accuse me of being biased here. The modern Broadcoms mostly descend from the Tigon II IIRC.

I'm not paid to do this, and keeping track of which Broadcoms work and which don't could well turn into a full time job, especially when you start looking at permutations such as "works with VLAN but not with bridged VLAN" or "breaks if used in link aggregation along with jumbo frames". I am hesitant to green light any card that has known gotchas, which some of the Broadcoms do, and while I am perfectly happy to believe that you may have found use cases where some Broadcoms work fine, I specifically warn people against them and other off-brand cards because most of them have caveats. I would rather see people get solid known-properly-working cards than halfarse cards like Mellanox which are landmines for newbies. If you actually have a card in hand and want to try it just to see, there's no harm in that.

 

[kiloohm]

I hear you on this and frankly a lot of this is a more of a hobby for me so I will play with it to see how broken or not it is.

I’ve spent a lot of time building pfSense appliances on Linux and manage a few servers at work and they all have these cards in it. The pfSense community seems to recommend them a lot (and I do use them with VLAN and TSO a lot but not LACP). On the other hand I’ve never seen 57710 recommended anywhere so I’m with you on this.

Anyway, I will report once I have my gear ready which will likely be another 2 months. Just ordered the first 2 hard drives…. Baby steps.

 

[Thebowler237]

Curious how the system build is going. Considering something similar for NAS, HA, frigate. An older generation of hw may be good enough for me though since I don’t plan to use plex.

 

[Fleshmauler]

I see no reasons to toss in an ssd or two for vms given how much you're already spending. Maybe a used gpu for transcoding. You're already spending over 2k, toss an extra $200 or so & have everything you want right away?