ZFS: Optimum Configuration w/16x Enterprise SSDs, 12x HDDs

[eduncan911]

Believe it or not, I tried to this keep this post as short as possible... Am I crazy for trying to plan for TrueNAS to do all of this?

So I'm new to ZFS (and kicking myself for not investing into this earlier). I have amassed some hardware for a previous local-disk Ceph cluster, that I am now leaning towards TrueNAS as the storage for a Proxmox cluster.

• 12 x 4 TB old Desktop HDDs (two models, 4x 5400 and 8x 7200)
  • Would want to keep 1x or 2x on the shelf for cold spares
• 16 x 240 GB new Samsum SM883 Enterprise SSDs that are Supercapacitor backed (instance fsync jobs, no wait)
  • However, I'd like to commit no more than 6x - maybe 8x - drives to this project - but I have more if needed.

And I already have:

• 24-bay chassis I'll dedicate to this project (to expand with additional vdevs as I grow)
  • The SAS2 backplane can reach 48 Gb/s as tested, not the normal 24 Gb/s variety.
• 40 Gb/s dual-port fiber in LAG across all servers (workstations are 10G and 40G, but can upgrade all to 40G if need be)
  • I can add more 40 Gb/s dual port cards if network sees a lot of contention to this NAS box.
• I have a choice of mobo/cpu/hardware:
  • High-speed 16C/32T ThreadRipper 2 w/64GB ECC DDR4
  • Or, low-speed 28C E5 V4 Xeons w/64GB ECC DDR4
    
• Optionally available, but prefer not to use:
  • 8-bay hot-swap chassis, running either the ThreadRipper or Xeon listed above
    • Or, i could use this chassis to "Expand" the 24-bay above for 32 total disks.
  • 24-bay 2.5" SAS2 Dual-Node chassis (both nodes share backplane and has an 10.5 GB/s interconnect between the nodes!)

Of course, everyone wants high read/write throughput, however:

• Lots of video and 3D animation videos my daughter and I remotely edit and use are > 50 GBs at a time, so it should be fast.
  • IOW, very fast bandwidth and file seeking remotely.
• This TrueNAS will be the backend for my Proxmox cluster, for everything:
  • ~11 VMs and some Windows (mostly idle)
  • ~29 Docker volumes, some in HA in K8s (various workloads, but some need high IOPS for Postgres/MySql)
  • K8s backend
  • Everything media related (low bandwidth, low IOPS, but large stripes/clusters)
  • Software development/compilation >100x, remotely, on the dataset for my work

As you can see, there's a potential for a lot of resource contention with this planning.

I can't see a lot of IOPS in any of the configurations, except some of the databases and some K8s etcd. They are rarely used; however, I do expect them to perform and now crawl.

Originally, the Ceph cluster was going to have 4 x 4 TB disks across 3 boxes - giving "local" access to about ~13 TB of storage across the entire cluster, across all services and applications. So that's where the performance would be (the SSDs would be for the block.db, for high IOPS). But stepping back and looking at the power usage, I'd prefer to downsize to a single TrueNAS ZFS box - if this is possible.

Or, is all of this not the ideal use-case for TrueNAS - to be the storage backend for a (home-use) performing cluster of Proxmox?

Thanks!

 

[eduncan911]

I guess I am stuck on the expected performance out of 2x vdevs (raidz2 6 disks), mirror will be fast I know but resource contention as a remote NAS for such a large-ish cluster, etc.

 

[jgreco]

Hopefully you ran across and read

https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

 

[eduncan911]

Hopefully you ran across and read

https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Actually, I hadn't. Thank you! Skimming seems to confirm many data points I've already gathered from Arstechnica and other well-known outlets.

It's just I've never used a "NAS" or iSCSI as a remote storage platform for VMs, containers, deployments, etc before. I am not sure what kind of troubles I may run into running applications that expect a locally-attached disks/performance - over the network.

Given, it's all fiber. But still, the TCP stack does increase latency.

 

[jgreco]

Everything increases latency, but the speed of a network can be made to beat the speed of rotational media seeking to find the right sector, so maybe it isn't as awful as you think.

You win in some ways and can lose in others. If you throw enough resources at ZFS, you can get SSD-like performance out of HDD's for many typical use cases, but you might not like my definition of "enough resources." It's all compsci trickery, exchanging one thing for another.

Please do more than skim through that post, it is designed to be a primer for anyone doing block storage on ZFS who hasn't dealt with ZFS's quirks before. You will probably have further questions, and this is very much a "the dumb question is the one not asked" type of scenario, so please ask. Don't be shocked if you get pointed at another prewritten post though. ;-)

 

[eduncan911]

Please do more than skim through that post, it is designed to be a primer for anyone doing block storage on ZFS who hasn't dealt with ZFS's quirks before. You will probably have further questions, and this is very much a "the dumb question is the one not asked" type of scenario, so please ask. Don't be shocked if you get pointed at another prewritten post though. ;-)

Oh yes, I will study it it. it's just <work work, busy, kids screaming, work work, something something> right now.