HW Build Review - TrueNAS Scale/Plex

[jonl]

Hi folks -

I've spent some time with a number of the other hw build review request threads (which led me to revisit my design several times already) so hopefully I've addressed the common things people get wrong with their first TrueNAS build. I've also reviewed the hardware guide and the Intro to ZFS (thank you!)

I have an existing QNAP build but I'm sick of how difficult it is to diagnose issues with QuTS Hero when anything goes wrong, and I also need to rebuild the hardware platform to something appropriate to locate in a primary living space as I move/downsize. I've noted the components that I'd like to reuse from my QNAP build if they're reasonably appropriate for the TrueNAS Scale system, and if they're not I'd love to hear about it. Also I never got rid of the NAS setup I had pre-QNAP, so fortunately I'll be able to use that while I rebuild.

From the other threads, I have some open questions as well:

1. I saw noted that sometimes the onboard SATA controller is not well suited for TrueNAS. Can someone expand on the reason for this, and how to determine which ones are good/bad? I was trying to figure out a mb/hba/graphics card combo that would work with ECC and found it very difficult to find an HBA that would also work with a graphics card, or more accurately an ECC capable motherboard that has more than one PCIe x 16 slot without dropping to 8/ea with the additional card. (So I hope I can use the on-board SATA ports on the AsRock)
2. I understand ZFS likes to scale memory with storage space, is 64G RAM reasonably sufficient for a ~100TB Plex server? Current specs have 2 x 32GB w/ 2 open slots in case I need to add more later.
3. I've got a pair of SSDs (Samsung EVO 870 2TB) from the QNAP build that I intend to use as a mirrored system drive (capacity overkill, but sunk cost), and a pair of M.2 NVME (Samsung 970 EVO Plus 2TB.) In the QNAP build, the M.2 are a mirrored cache drive. Would these be suitable for an L2ARC or would that dramatically increase the need for additional RAM? I need to reserve ~1TB on one of the SSD pairs for transcoding, what about splitting it for 1TB of L2ARC and 1TB "storage" for the transcoding temp folder?
4. I'm planning to host the main storage on an 8-wide RAIDZ2 vdev. Reasonable for Plex? ZFS Intro has a note about a 4-wide + 8-wide being "not horrible" : )

Current iteration of build specs:

• AsRock Rack W480D4U mATX Server Motherboard
• Intel Xeon W-1250 6 Core 3.30 GHz Processor
• 2x NEMIX RAM 64GB (2x32GB) Kit DDR4-2666 PC4-21300 ECC UDIMM Server Memory
• PNY TECHNOLOGIES Nvidia Quadro P4000
• Seasonic PRIME GX-850, 850W 80+ Gold Power Supply
• Fractal Design Define 7
• Fractal Design Celsius S36 Blackout 360mm AIO CPU Cooler
• 1x Kingston A400 120GB SATA 3 2.5" Internal SSD (Boot/system)
• 11x Seagate 18TB Exos X18 7200 RPM SATA 6Gb/s 256MB Cache 3.5-Inch Enterprise Hard Drive (Data)
• 2x Samsung EVO 870 2TB (Mirror - app/docker volumes)
• 2x M.2 Samsung 970 EVO Plus 2TB (1x for zfs cache, 1x for plex transcoding scratch)
• LSI 6Gbps SAS HBA LSI 9201-8i + Intel RES2SV240 SAS-2 Expander 24-Port 6Gbps Expansion Adapter

Thanks for reading! Appreciate any thoughts or insights you may have.

Edit: Updated processor for mb compatibility, video card for height (2 slot -> 1 slot) so I can add the LSI SAS HBA to provide additional necessary ports, added 120G SSD for system drive, updated case to current gen Define 7, updated drive purpose annotations

Edit2: Added an expander and 3 more drives now that we have zvol expansion

 

[NugentS]

The 2 x Samsung EVO 870 2TB as a system disk are a waste although I note you already have them. TN only needs 16 or so GB although I consider 32GB to be a better figure. Also as long as you keep a regular backup of the config using something like multi-report from @joeschmuck then rebuilding the NAS is literally no effort at all. Personally I think:
1. Boot from a "new" 32+GB disk of some kind
2. Keep a 2TB EVO as L2ARC - but don't use it immediately - you may not need it. Note you do not need to mirror an L2ARC (and may not be able to)
3. Keep a 2TB EVO as a cold spare or use it as a single scratch disk for transcoding
4. Keep the 2 970 EVO+ as spares OR use as a mirrored pool for system dataset and apps

You only have 8 SATA Ports and 2 M.2 so if you use the onboard SATA for the 8*Seagate EXOS then you have 2 M.2 left. One to boot and one for L2ARC - you are short on ports

 

[jonl]

SSD usage suggestions make sense, thanks. With just a read cache don't need a mirror - right.

You only have 8 SATA Ports and 2 M.2 so if you use the onboard SATA for the 8*Seagate EXOS then you have 2 M.2 left. One to boot and one for L2ARC - you are short on ports

Ahhh right you are. Do the M.2 slots use SATA ports that I'm effectively double allocating between the storage and SSD drives? Or am I only short the 2 for the EVO SSDs? (i.e. am I short 2 or 4?)

I could leave it as 6*18TB instead of bumping it to 8, but the extra capacity headroom did sound nice. If I do add an HBA to this system to get the extra ports, is that essentially a tradeoff with the video card, or is there a way to do both? (Second part of #1 in op)

I realize these are not TrueNAS specific questions but appreciate the help. I much prefer the software side of things!

 

[NugentS]

Normally (you would need to check the mobo manual) SATA Ports are shared if you use SATA M.2 drives. NVMe drives are directly attached to the PCIe bus so you could be OK.

And HBA is normally a X8 card and it looks to me like you have a spare X8 slot as you are only using a GPU. The X1 slot is all but useless

If the X16 drops to a X8 with both slots used - don't worry about it - a GPU will hardly notice the difference and you ain't gaming on this.

 

[jonl]

Oh rad -- I didn't realize it was just shared bandwidth for the two ports, I thought a x16 card simply wouldn't work if there was another card in the other slot. That makes way more sense. Slick, thank you! I'll look for an HBA for the storage and use the on-board for the system drives.

 

[jonl]

Oh looks like I can't edit my original post. Adding this to the parts list:

LSI Broadcom SAS 9300-8i 8-port 12Gb/s

 

[NugentS]

Good card in theory. BUT given where its come from it may (note may) be a fake (which is a real problem)
If I may suggest - for HDD's only: https://www.ebay.co.uk/itm/162958581156?hash=item25f1169da4:g:7sYAAOSwjMtcT8T8
Or if you think you might want to attach SSD's: https://www.ebay.co.uk/itm/163040642059?epid=2078952393&hash=item25f5fac40b:g:4hUAAOSw58BgBoVX

These aren't new, but they have the advantage of being genuine, and come pre-flashed with the correct firmware for TrueNAS - so you don't have the hassle of having to do it.

No association - he is just a well known supplier of good parts who knows what he is doing

 

[zachlovescoffee]

How did you build go?

 

[jonl]

How did you build go?

Hey thanks for asking. I spent a bunch of time narrowing down Intel procs that support ecc, but supporting ecc and matching the socket type was not sufficient, and it turned out that the mb didn't support that proc (xeon e series, only supports w series, which is weird but ok.) So that wasn't awesome and it didn't boot. Waiting on a new proc now.

Also the initial video card was too tall once I added the hba, so got a new one of those coming as well. Case is nice though, and while no load yet, the fans are super quiet. More to come when I get it booting haha

 

[zachlovescoffee]

I'll keep watching - Good luck! I'm slowly starting to try and understand TrueNAS and finding the right hardware for my use case, without blowing a hole in my wallet is becoming a challenge.

 

[jonl]

Thanks! I'm definitely prioritizing noise, capacity, and capability over cost for this one. I know you can do this more cheaply than I am. The challenge for me on this one was building with ECC capable components. That shifted all the core (mb/proc/ram) selection into brands/constraints I was less familiar with vs a PC build. Also the HBA selection with gaining enough SATA ports for everything (thanks again to NugentS for saving me from the potentially counterfeit controller.) Happy to answer any questions if it's useful to you.

 

[zachlovescoffee]

Good luck! :) I'll always take good advice on hardware as this is a new branch for me to climb out on!

I have a telco closet in my house where all my cabling, current NAS (2 bay Synology), modems, switches and routers start/terminate. So heat, moisture and noise isn't a problem. My current set up is just using two wire shelves mounted to exposed studs. It's really clean for being so amateur. What I'm trying to figure out is a cost effective way to have solid NAS for movies (1080-4k), shows (1080-4k), music (FLAC/AAC), photos/videos from phones (anticipated to grow exponentially in the next 3-5 years as my kids get phones), machine back ups (4 computers), etc with the biggest emphasis being on Plex high quality streaming/transcoding. Because I'm constantly acquiring content, I anticipate that I need at least 25-30 Tb to *start* with. I have 10Tb now (Seagate Exos) in RAID1 (2 disks) but I'm already 30% full after about a month of starting to back up all of my media. So-- if you have any recommendations on hardware I'd welcome it :) I have been reading the guides over and over again and they are helpful but man it's a lot to digest.

 

[NugentS]

I use an NVidia Shield as a Server. It works well and is more reliable than I have found running plex in a container or PC. Its also quite low power. You could then run a low power NAS as mostly just a storage device. A NAS itself doesn't need that much power for serving files but transcoding can unless you manage to direct stream everything.

Use big disks as that way you need less of them using less power, generating less heat and get less failures

 

[zachlovescoffee]

I use an NVidia Shield as a Server. It works well and is more reliable than I have found running plex in a container or PC. Its also quite low power. You could then run a low power NAS as mostly just a storage device. A NAS itself doesn't need that much power for serving files but transcoding can unless you manage to direct stream everything.

Use big disks as that way you need less of them using less power, generating less heat and get less failures

That’s basically what I was considering. I bought a high end NUC (11th gen i5) and was going to use an external 5-bay hard drive enclosure and just run JBOD on 3 disks and then have redundancy on disks 4-5 of certain content from disks 1-3. Using mdadm via Ubuntu server.

But then I stumbled and fell down the TrueNAS hole. Anyway, I don’t want to thread high jack! I have another thread going for new user support :)

 

[jonl]

Update I suppose.. https://www.truenas.com/community/threads/bad-controller.109941/

 

[jonl]

Solved via BIOS update. I think I'm finally done with this build! If anyone wants pix or perf stats when I move the workload onto this lmk.

 

[rawkus]

How has the build been performing?

 

[jonl]

How has the build been performing?

Beautifully. I knew it was overkill and it is haha. Averages 1-2% cpu.