eduncan911
Cadet
- Joined
- Jun 27, 2021
- Messages
- 4
Believe it or not, I tried to this keep this post as short as possible... Am I crazy for trying to plan for TrueNAS to do all of this?
So I'm new to ZFS (and kicking myself for not investing into this earlier). I have amassed some hardware for a previous local-disk Ceph cluster, that I am now leaning towards TrueNAS as the storage for a Proxmox cluster.
As you can see, there's a potential for a lot of resource contention with this planning.
I can't see a lot of IOPS in any of the configurations, except some of the databases and some K8s etcd. They are rarely used; however, I do expect them to perform and now crawl.
Originally, the Ceph cluster was going to have 4 x 4 TB disks across 3 boxes - giving "local" access to about ~13 TB of storage across the entire cluster, across all services and applications. So that's where the performance would be (the SSDs would be for the block.db, for high IOPS). But stepping back and looking at the power usage, I'd prefer to downsize to a single TrueNAS ZFS box - if this is possible.
Or, is all of this not the ideal use-case for TrueNAS - to be the storage backend for a (home-use) performing cluster of Proxmox?
Thanks!
So I'm new to ZFS (and kicking myself for not investing into this earlier). I have amassed some hardware for a previous local-disk Ceph cluster, that I am now leaning towards TrueNAS as the storage for a Proxmox cluster.
- 12 x 4 TB old Desktop HDDs (two models, 4x 5400 and 8x 7200)
- Would want to keep 1x or 2x on the shelf for cold spares
- 16 x 240 GB new Samsum SM883 Enterprise SSDs that are Supercapacitor backed (instance fsync jobs, no wait)
- However, I'd like to commit no more than 6x - maybe 8x - drives to this project - but I have more if needed.
- 24-bay chassis I'll dedicate to this project (to expand with additional vdevs as I grow)
- The SAS2 backplane can reach 48 Gb/s as tested, not the normal 24 Gb/s variety.
- 40 Gb/s dual-port fiber in LAG across all servers (workstations are 10G and 40G, but can upgrade all to 40G if need be)
- I can add more 40 Gb/s dual port cards if network sees a lot of contention to this NAS box.
- I have a choice of mobo/cpu/hardware:
- High-speed 16C/32T ThreadRipper 2 w/64GB ECC DDR4
- Or, low-speed 28C E5 V4 Xeons w/64GB ECC DDR4
- Optionally available, but prefer not to use:
- 8-bay hot-swap chassis, running either the ThreadRipper or Xeon listed above
- Or, i could use this chassis to "Expand" the 24-bay above for 32 total disks.
- 24-bay 2.5" SAS2 Dual-Node chassis (both nodes share backplane and has an 10.5 GB/s interconnect between the nodes!)
- 8-bay hot-swap chassis, running either the ThreadRipper or Xeon listed above
- Lots of video and 3D animation videos my daughter and I remotely edit and use are > 50 GBs at a time, so it should be fast.
- IOW, very fast bandwidth and file seeking remotely.
- This TrueNAS will be the backend for my Proxmox cluster, for everything:
- ~11 VMs and some Windows (mostly idle)
- ~29 Docker volumes, some in HA in K8s (various workloads, but some need high IOPS for Postgres/MySql)
- K8s backend
- Everything media related (low bandwidth, low IOPS, but large stripes/clusters)
- Software development/compilation >100x, remotely, on the dataset for my work
As you can see, there's a potential for a lot of resource contention with this planning.
I can't see a lot of IOPS in any of the configurations, except some of the databases and some K8s etcd. They are rarely used; however, I do expect them to perform and now crawl.
Originally, the Ceph cluster was going to have 4 x 4 TB disks across 3 boxes - giving "local" access to about ~13 TB of storage across the entire cluster, across all services and applications. So that's where the performance would be (the SSDs would be for the block.db, for high IOPS). But stepping back and looking at the power usage, I'd prefer to downsize to a single TrueNAS ZFS box - if this is possible.
Or, is all of this not the ideal use-case for TrueNAS - to be the storage backend for a (home-use) performing cluster of Proxmox?
Thanks!