Help designing my pool

[wctschumy]

Hi all,

I'm moving from a Windows Pro for Workstation environment to TrueNas Scale. The machine I'll be using is a Ryzen 3800 (which will go to a 5950) with 128gigs of ram, 2 1tb nvme's and 16 6tb spinning drives connected via a 12g SAS HBA. The 16 drives are in a storage space running REFS and I'm using PrimoCache (read/write cache) on top of the storage space.

I've been mostly happy with it, but am bumping up against some limitations, notably iSCSI.

The machine is used for three main workloads:

1. It's the plex server for the house. Part of that is taking recordings from the two tuners I have and transcoding them to H265. That's currently hardware accelerated. We typically have 3 or so concurrent streams
2. It's an NVR for our IP security cameras. I'm using Blue Iris + DeepStack for object detection. There are 6 camera streams, either 4k or 1080p. The CPU usage on this is crushing the machine right now, hence the upgraded processor
3. It's fileshare for all the PCs on the home network. In particular, I want to put my and my kid's steam library on it, vs adding yet more storage to one of our machines, hence the iSCSI need

BlueIris is Windows-only, so I'll need to run a Windows VM. I will likely also run Plex in that VM with the encoding infrastructure - I'll pass through the GPU to that VM and hopefully that will just work.

I've spent a fair amount of time reading through different approaches to pool design, but I can't say I feel confident about any of my conclusions. My initial thinking was to put them all into one Pool with 4 4 disk VDEVs, but looking through the forum, I'm not sure that's the right approach. Any guidance is appreciated.

Thanks!

 

[Nick2253]

I would actually suggest running a hypervisor (like HyperV or Proxmox), and then visualizing TrueNAS (passing the SAS HBA though to the TrueNAS machine). Use the two 1TB NVME drives in a mirror for OS and VM storage.

For your use case, I would not recommend a SLOG or L2ARC device. For NVR use, sync writes are pretty much unnessesary. I believe that the default sync buffer is around 5 seconds, so you could potentially lose that much video. Otherwise, ZFS will ensure a consistent state on disk. And if you use async writes, then a SLOG provides no benefits.

With the RAM capacity of your system, an L2ARC is only beneficial if you are regularly accessing *lots* of the same data. For an application like Plex, at best an optimally cached file would slighly reduce the buffering time upon opening the video file. For file sharing, an L2ARC would make sense if you are regularly accessing a large quantity of small file that would exceed the capacity of you L1ARC (RAM).

TrueNAS CORE uses bhyve as its hypervisor, and in my opinion, it's a bit lacking compared to more established hypervisors like Hyper-V, VMware, or KVM (Proxmox and oVirt use KVM). If you virtualize TrueNAS, you can use these more established and feature-rich hypervisors to get more out of your system.

For pools, I'd recommend 2x 8-drive RAIDZ2 vdevs. I would personally only use 4x RAIDZ1 vdevs if I had a solid local backup strategy and replacement HDDs on hand. Restoring that much data from the cloud is slow and expensive.

If you really need to maximize performance, then striped mirror are the obvious solution. You might also run multiple pools: 1x 8-drive RAIDZ2, and 1 pool of 8 striped mirrors.

EDIT: I just realized that we're talking TrueNAS SCALE, which uses KVM as it hypervisor. From that perspective, I'm slightly more inclined to support running VMs from within TrueNAS. At the same time, I'm generally opposed to the "hypercoverged" philosophy, and prefer to use software in the role that it's best designed and supported for. As such, I primarily want to use TrueNAS in its main NAS role.

 

[wctschumy]

Thanks for the fast reply! Very helpful. I realized there's one other requirement - I want the Windows VM to have access to all the stored data. It's really slow, but I use one of the cloud backup offerings for offsite backup. It's definitely slow, but it's there and I've got all the data uploaded there.

IOPS would be substantially higher with 4 x 4RAIDZ1, vs 2 x 8RAIDZ2, correct? I'm not too worried about getting replacement drives - they're easily obtainable quickly.

Thanks! Will.

 

[Nick2253]

"Substantially" is relative. In theory, it will be 2x (though in practice, slightly less than that). However, it shouldn't increase throughput, since you still have the same number of data disks.

If you share all of your datasets using SMB, you can mount those on the Windows VM for backup purposes. I do something similar in Proxmox (not TrueNAS, but SMB sharing between two VMs). Using the Virtio network driver, the network connection is nearly as fast as the hardware HDD to HDD.

A trick for mounting SMB shares in Windows is to use mklink to attach the shares to a path in the folder structure. For example, use C:\NAS\<dataset1>, C:\NAS\<dataset2>, etc. If you use "map network drives", then those drives are only available on a user-by-user basis. For services to access network shares, mklink allows you to easily manage the shares without needing to do some complex wrapping script for the service that mounts the network shares before starting.

 

[wctschumy]

I'm getting ready to pull the trigger here - I'm trying to decide pros/cons vs putting all the data I want to back up into an iscsi share and running the back-up software on the PC that mounts the target. If I took that approach, I'd need to SMB share from the PC that mounted the iscsi folder, vs from TrueNAS, correct? Is it possible to have TrueNAS share it instead?

I've decided against the virtualization approach - I'm building a separate, dedicated machine for the server. I'll use the TrueNAS box as just a file server.

 

[Nick2253]

I'm trying to decide pros/cons vs putting all the data I want to back up into an iscsi share and running the back-up software on the PC that mounts the target.

If you go the iSCSI route, you're using a zvol, and that puts a lot more strain and capacity restrictions on your system. If using zvol, the general recommendation is that your usable space is 50% of your pool, or else the performance hit will cripple any writes to it.

Unless you *have* to have a block device, I'd never recommend using a block device. Putting the files directly on ZFS and sharing via NFS or SMB is usually more performant and will give you much better control via TrueNAS.