Can someone review my ZFS pool concept?

[Gramdalf]

If there are any ZFS chads out there I would greatly appreciate your input.

I've already figured out where I want my containers running, and how I'm going to link my storage between Proxmox and TrueNAS, so ignore that aspect for the purpose of the following (unless there's something you think I should really know)

I'm running TrueNAS, and planning my ZFS layout after many months of waiting. This is my current hardware setup:

Code:

Dell Poweredge R810
├─ OS: Proxmox VE @ v7.x
├─ CPU: 4x Intel Xeon E7540 @ 2.00GHz
├─ RAM: 128GB (32x4GB) DDR3 @ 1333 MT/s
├─ HBA: LSI SAS 9201-16e @ FW 20.00.07.0
│     └─ HP SAS expander 468405-001
└─ TrueNAS Scale 22.12.3 @ VM
      └─ LSI SAS 9201-16e @ PCIe Passthrough

Connected to the SAS expander are some breakout cables, connecting to my mix of SAS/SATA drives (all CMR). There are 16 disks:

• 11x 2 tbyte
• 5x 4 tbyte
  • One of these is currently non functional, I suspect hardware failiure - still to be inspected

Connected directly to the LSI HBA (with SAS 8088 > SATA breakout cables) are (currently)

• 2x 512gbyte SSDs

I have

• 3x 256gbyte SSDs

available to use (I already have cabling and whatnot). I want to have at least raidz2 redundancy on all storage, excepting maybe cache (not sure what I'm gonna do for that, looks like l2arc and slog aren't critical to have mirrored in my situation https://www.reddit.com/r/zfs/comments/4lkv5v/can_loss_of_slog_or_l2arc_failure_on_modern/). I've read through the TrueNAS ZFS Whitepaper, and some blog posts about different use cases etc, and this is my current thought.

Main (capacity based) pool: Vdev 1

• 11 wide 2tbyte @ raidz2

Vdev 2

• 5 wide 4tbyte @ raidz2

l2arc vdev?

• 1 wide 256gbyte @ stripe

slog vdev?

• 2 wide 256gbyte @ mirror

Secondary (speed based) pool: Vdev 1

• 2 wide 512gbyte @ mirror

Questions

• Should I bother with encryption?
  • The server is in my own house, in the basement, and there is nearly always someone home.
    • Furthermore (at least according to ChatGPT - don't judge, it's a quick alternative to scouring google), my CPU(s) don't support hardware accelerated decode, and could therefore have a (somewhat significant) performance impact
    • All in all, the risk of a physical breach is limited, is there anything else I should be concerned about?
• Is this a sane layout?
  • The plan for this layout is to be used as a NAS (on the capacity pool), a Minecraft server host (on the speedy pool), and various other self-hosted things (vaultwarden, jellyfin, etc, also on the speedy pool), mostly docker/LXC with some VMs in the mix if need be.
    • I will also have an offsite backup, so the setup will follow the 3-2-1 strategy - raidz2 (imo, correct me if I'm wrong) counts for two of the three copies, and the offsite backup (regularly synced) counts for the third copy.
    • Capacity is of notable importance - there are a large number of files (mostly photos) currently waiting to be added to the array, and a large amount will be added regularly (one 128gbyte SD card holds about 1,500 DSLR photos)
    • Am I correct in thinking that raidz2 (plus an offsite backup) is ample protection?
• Is there anything else I should consider?
  • I believe that I've covered everything I can, and done ample research - mostly from here (https://www.truenas.com/community/r...em-build-notes-plus-new-user-advice-help.216/), but I could easily be missing something.

Thanks for your time!

 

[Etorix]

Should I bother with encryption?

Any mismanagement of keys here would make for a nice self-ransomware. Your call…

Furthermore (at least according to ChatGPT - don't judge, it's a quick alternative to scouring google)

I hate to write that, but Google (or any other traditional search engine) is vastly better than GhatGPT and its ilk. At least you get to see where the information comes from and can decide whether the source is trustworthy. While ChatGPT is known to happily make up case law or "technical" information (as in: TrueNAS CORE is based on Gentoo Linux and can run on 2 GB RAM…).

Is this a sane layout?

It is obviously designed to get maximal storage out your current mix of small, old drives. But because it is raidz2, you'll then get stuck with this layout forever—which implies that it is not a sustainable design. To increase space, you'd need to either

• bring in yet another vdev (and get stuck with it forever as well);
• replace all five 4 TB drives (which are currently the largest ones); or
• replace all eleven small 2 TB drives.

None of these options is really attractive.
Three 5-wide raidz2 vdevs would be more balanced and more sustainable.
If you can bring in two more drives, three 6-wide raidz2 vdevs would be even better.

With old Xeons and lots of small drives, the system will use lots of power for not so much of storage capacity (30 TB raw: Just four 16 TB drives in raidz2 could best that).

SLOG is for sync writes, and for sync writes only. Cheap consumer 256 GB SSDs are unlikely to be acceptable as SLOG.
And no L2ARC until you have tested the system under actual load and found that L2ARC could actually be of use.

I will also have an offsite backup, so the setup will follow the 3-2-1 strategy - raidz2 (imo, correct me if I'm wrong) counts for two of the three copies, and the offsite backup (regularly synced) counts for the third copy

Each NAS counts for a single level. If your house goes up in flames, all redundancy in the raidz2 is lost in a single "incident".

Am I correct in thinking that raidz2 (plus an offsite backup) is ample protection?

"Ample" is a relative term, and subordinate to one's degree of paranoia.
Let's say that raidz2+offsite is "fair". "Offline" is still missing to complete "3-2-1".