Recommendation on TrueNAS setup

[pinterpass]

I'm new to TrueNAS and am in the process of migrating away from Netgear's EoL'd ReadyNAS line, which has served me well for 10+ years. This is my very first homebuilt NAS.

Based on what I've researched, I'm leaning toward mirrored drives (decent redundancy, faster recovery). But because I don't know what I don't know and this whole pool–dataset–share paradigm is new to me, I was hoping to solicit advice on the best way to set up TrueNAS pools and datasets for my use case.

Which is:

• Plex: Our household's NAS is our media hub. We rip CDs and DVDs/Blu-rays to lossless audio/video (increasingly 4K HDR MKVs) for local streaming, sometimes with multiple family members streaming media simultaneously. A Raspberry Pi-based Logitech Media Server instance (yes, we still use Squeezeboxes!) also draws on the audio repository.
• Time Machine: We have multiple laptop/desktop Macs and like to have network backup.
• Multi-user network storage: We use the NAS to access common files from multiple clients, to offload files from clients (archival storage) and to have user-based repositories. Basically, it's a giant attic for infrequently used files and large backups.
• Possible virtualization: I primarily use a Mac but occasionally use Windows-only software. I do have a dedicated Windows gaming-oriented desktop that I can access locally or remotely. But rather than clutter up my gaming rig with productivity software, it might be nice to have a work-oriented Windows VM that I could access from any machine.

Up to now, I've been using the ReadyNAS paradigm of one big data pool that you parse out through different shares. And it's worked well enough.

However, as I understand ZFS/TrueNAS, the idea of using dedicated pools for different datasets is appealing. My assumption is that performance would benefit if, say, one user is streaming 4K video from the dedicated Plex pool of HDDs and another user is backing up large files to the dedicated data pool of HDDs.

My physical storage media in this new build is: 2x 4TB NVMe SSDs, 6x 20TB HDDs. I also have two spare 4TB SATA SSDs that I could add, although that would max out my motherboard's eight total SATA slots. I would like to be able to add more high-volume storage in the future if necessary without resorting to PCIe SATA cards.

What I was envisioning was having an audio library stored on 2x 4TB NVMe SSDs (1x mirrored pair), the video library on 4x 20TB HDDs (2x mirrored pairs) and all the remaining storage on 2x 20TB HDDs (1x mirrored pair).

But here's my question to the TrueNAS veterans and experts: If you had the above use case, what might your approach be?

In other words, does that strike you as the optimal setup? Would it be better to have the NVMe drives reserved for metadata caching? Does the Plex app need a dedicated pool (in which case, splitting out audio and video into two separate pools might be unwise)?

Thanks in advance for any advice!

 

[Ericloewe]

However, as I understand ZFS/TrueNAS, the idea of using dedicated pools for different datasets is appealing.

I have to strongly disagree. Random bits of storage spread out are a pain in the ass.

My assumption is that performance would benefit if, say, one user is streaming 4K video from the dedicated Plex pool of HDDs and another user is backing up large files to the dedicated data pool of HDDs.

Maybe? Seems contrived and I could imagine scenarios benefiting either option.

But here's my question to the TrueNAS veterans and experts: If you had the above use case, what might your approach be?

Your scenario does not seem particularly tricky. Particularly if you're just running 1 GbE, it sounds like you're prematurely optimizing for problems that don't exist. Use a single pool, possibly forget about the SSDs even.
Fake edit: Your signature does mention a 10GbE NIC, but the overall point stands. We can discuss details...

For posterity, and because signatures are not always visible, let's go over it:

• SuperMicro X13SAE-F
• Intel i5 13600K
• 64GB Micron ECC RAM
• 6x 20TB Seagate Exos20/22 HDDs
• 2x 4TB WD SN850x NVMe SSDs

Fine. 64 GB of RAM doesn't sound like that much for six 20 TB disks, but it really is enough in many cases.

1x 2TB PNY NVMe SSD (boot)

Massive overkill.

Solarflare SFN6122F Dual Port SFP+ 10GB NIC

Not great, not terrible.

 

[danb35]

as I understand ZFS/TrueNAS, the idea of using dedicated pools for different datasets is appealing.

I'm curious what would have contributed to this understanding, as "pooled storage" is a large part of ZFS' design. Sure, you can have multiple pools--as many as you like--but in most cases, the most sensible configuration is "one big pool."

 

[pinterpass]

I'm curious what would have contributed to this understanding, as "pooled storage" is a large part of ZFS' design. Sure, you can have multiple pools--as many as you like--but in most cases, the most sensible configuration is "one big pool."

Based on some cursory forum browsing and YouTube videos. I saw a lot of folks creating discrete pools for different purposes. I also have a 10GbE network with several 10G/2.5G clients, so I don't have the 1G network bottleneck.

And I might reclaim the 2TB boot NVMe someday, but I repurposed it from another build to save on money. I'd already dropped enough on this build.

If the SolarFlare NIC is "not great, not terrible," what's the recommended option? I researched NICs more than any other component in this build, but it was hard to find up-to-date, reliable, affordable, plug-and-play recommendations for TrueNAS Scale.

At any rate, my uncertainty has less to do with the hardware than the pool/dataset configuration.

 

[danb35]

If the SolarFlare NIC is "not great, not terrible," what's the recommended option?

The SolarFlare is fine; better options are Chelsio and Intel. The 10G primer in the resources section is as current as it gets.

As to pools, the "default" configuration is to build one single pool. Exceptions would be:

• You have markedly different performance requirements for some of your data
  • e.g., you're storing both bulk data and VM images--the former might be in a RAIDZn pool, with the latter on striped mirrors or SSDs
• You have markedly different redundancy requirements for some of your data
  • You might in this case have a single-disk or striped pool for "scratch" or disposable data, and another redundant pool for data that actually matters
• You're just paranoid
  • The thought that loss of a vdev means loss of the entire pool just freaks some people out

 

[Ericloewe]

(Xenforo failed to submit my post, though it adds little at this point, it's written, so here it goes... )

Most youtube videos can safely be disregarded as junk, slightly better than ChatGPT.

what's the recommended option? I researched NICs more than any other component in this build, but it was hard to find up-to-date, reliable, affordable, plug-and-play recommendations

I can't imagine why, because the recommendation has been the same basically forever: Intel or Chelsio. The Intel X520/82599 in particular is ubiquitous, extremely well-supported, available in all sorts of form factors (except OCP 2.0 and 3.0). If you're looking for a more advanced feature set, that's where the X710 comes in.
The only catch is that you need to navigate between artificial roadblocks when choosing SFP modules - these do not exist when using DACs.
Chelsio, historically at least, had a small edge in driver quality on FreeBSD, though Intel hardware is said to be better. Overall, it's a wash.

The Solarflares have been showing up a lot lately, and people seem reasonably happy with them, so there may be a decent path there for you.

creating discrete pools for different purposes.

This only makes sense when the workloads are wildly different. Think archival storage of large files vs. heavy duty backing store for VM disks. For your scenario, three of four use cases are basically the same - normal, boring file storage.

 

[Redcoat]

If the SolarFlare NIC is "not great, not terrible,"

The SolarFlare is fine;

My experience with 2 Solarflare SFN6122F 10G between my TrueNAS boxes has been absolutely without issues - they "just work". I would use them asgain without hesitation. Last time I looked they were still inexpensive and readily available.

There have been occasional cautions here about cooling as they, just like HBAs and other "server components", were not expected to be jammed into consumer pc cases with limited through-air flow.

 

[pinterpass]

Thanks, everyone, for your input. Despite all the tutorials that I watched, I'm starting to think that I got conceptually muddled and was referring to pools when I meant vdevs, and by extension was thinking of vdevs as datasets.

Before pursuing this any further, I might just noodle around with TrueNAS in different demo setups and then settle on a configuration that looks like it makes sense. That hands-on experimentation might be the best way to figure things out.

I do think that I'm going to sideline all the SSDs for now and just get another 64GB of RAM. That looks like it'll be the most effective in terms of performance for 99% of my intended use case.

As for 10G networking, I saw the Chelsio and Intel discussion in the 10G thread, but those NICs a) looked like they ran hot and b) could be of questionable authenticity when they were in my ideal price range. I initially bought a 10Gtek SFP NIC that billed itself as "comparable with Intel
X520-DA2" but suffered constant network dropouts. The SolarFlare stood out to me because it supposedly ran cooler and a server pull NIC was only about $30 from a reputable seller.

 

[danb35]

looked like they ran hot

I think this is going to be true, or at least a concern, of any 10G NIC that doesn't have its own active cooling.

could be of questionable authenticity

I don't recall seeing that as much of a concern for the Chelsio cards, but it can be for Intel.

Everything I've seen about the Solarflare cards has been adequate, and I haven't had any complaints about mine (though it's under Proxmox, not TrueNAS--though both are based on Debian). If you already have one, go ahead and use it. If it gives you problems, you might try looking for Chelsio or Intel, but otherwise don't worry about it.