TrueNAS SCALE for hyperconverged homelab

[iceypoi]

I'm looking into building a hyperconverged homelab like I mention in the original thread. The context is, that I've got 10Gb symmetric internet and I need a NAS / Hypervisor combo.

For the NAS the workload would be fairly small, albeit 10Gb internet is definitely a lot more parallel downloads than just 1Gb.

On the other hand, for the hypervisor, I need raw power. That's why I'm getting 2x2TB Samsung PM9A1, 256GB ECC RAM, and a 3960x or 3970x Threadripper.

If I split the boxes, I'd need to connect them with a 100Gbe interconnect to get anywhere near the performance, that's why I was hoping to make them into one hyperconverged server.

From what I gather TrueNAS SCALE may very well be what I'm looking for. TrueNAS can take care of storage and the light workloads like background downloading, backup, etc. And then I can either create a container or a KVM vm and pass-through 48 / 64 threads and, say, 128GB RAM to the high-performance vm.

Here are my storage considerations:

• The motherboard has 8xSATA port and 3xM.2 ports
• Get 8x8TB WD Gold drives. They can do about 250MB sustained, so I need at least a 4x mirrors to saturate the 10Gb for downloads.
• I'd stick in 2x2TB PM9A1 in mirror.

What would you suggest to do with the last M.2 slot? I could stick in, say, a 500GB 980 Pro as L2ARC (probably not worth it with the amount of available RAM?), I could stick in a 59GB Optane 800p as (non-mirrored) SLOG? Or maybe stick in a 3rd 2TB PM9A1 and RAIDZ1 them together?

Additionally, I'd like to use some of ZFS's flexibility with partitions:

• shave off e.g. 20GB partitions for a mirrored boot pool (I could theoretically boot off a USB stick, but would prefer to use the NVMe drives)
• shave off e.g. 10-20GB for a SLOG for the spinning rust pool, can be mirrored if we already have 2 NVMe drives. Alternatively, I could use the 58GB Optane 800p, if we get it.
• shave off e.g. 20GB partitions for a mirrored metadata vdev for the spinning rust pool.
• Use rest of the NVMe drives for data for the high-performance VM / container.

Given, I know ZFS lets you use partitions like that, because I did similar things when I was running on NixOS; how do you use partitions in TrueNAS SCALE?

It could also be, that my requirements are somehow completely at odds with TrueNAS SCALE... What else should I use if TrueNAS SCALE is a poor choice?