Have stuff ready for 1st FreeNAS build, looking for a little direction

[jcl123]

Hello,

I happily ran a 10-slot QNAP box for many years until the backplane died and they don't sell replacements.
So I want to switch to FreeNAS finally after reading about ZFS and following along for years.
I re-purposed the QNAP into an external JBOD with SAS expander, and have gathered a bunch of other hardware, as follows:
They were throwing away an old EMC SAN at work, so I harvested the 4TB SAS disks and SSDs from it
- I just had to re-format these drives from 520 to 512 sectors and they are good to go
- Going to follow the 3-2-1 rule with two servers + cloud

Server 1 (new)
- Case that can hold 12x 3.5 in drives
- Older Supermicro X9SCM-F w/E3-1280v2 and 32GB ECC RAM
- Broadcom 9400 16i 12Gig SAS tri-mode HBA (good deal on e-bay, more future-proof)
- Intel RES3TV360 12Gig SAS expander in jbod
- Intel x550 10Gig NIC
- A large number of Seagate 4TB SAS2 enterprise drives
- A large number of 200Gig Samsung SAS2 SLC SSDs
- A large number of 400Gig HGST SAS3 TLC SSDs (seem to be firmware locked to SAS2)

Server 2 (old desktop computer, this is where my data from the QNAP currently is)
- Case that can hold about 9x 3.5 drives
- Gigabyte motherboard with Intel i5 CPU and 8GB RAM
- 7x 10TB WD SATA drives "shucked" from Best Buy
- Intel x550 10Gig NIC
- Currently running Server 2019 DS with Storage Spaces

Thoughts:
- I am thinking Server 1 will become the new "main" server, and server 2 will be to backup server 1
- Going to buy another Broadcom card for Server 2, might change it to also be a SM board with Xeon and ECC RAM
- In total, Server 1 can practically take 22x 3.5" drives between the case and the JBOD, plus I can fit a couple of the SSDs in the expander as well
- I have around 20TB of data currently, mostly family pictures/videos, a hold bunch of BD ISOs, and so on
- My primary uses are general file services, might play with PLEX or something like that
- I work in IT and do allot with VMware and software defined storage at work

So, here are my questions:
1. How would you layout the drives?
- I could do up to 20x 4TB drives in RAIDZ2 or RAIDZ3 for 72TB or 68TB respectively
- Since I don't need that much space, maybe do 14 drives - 10 for data, 3 parity (Z3) and one hot spare? 40TB usable? (even 30TB would be OK)
- If I only use some of the slots, I could take a bunch of the 400Gig flash drives and make another all-flash disk group for VMs or something
- The SLC SSDs I have would work well for a mirrored ZIL and L2ARC and such, if I actually need both
- One of my concerns is that, last I checked you can't "expand" ZFS unless you add another disk group, or replace all the VDEVs with larger disks, so I want to kind of plan for the day when I do need to expand. With 40TB, I might actually go the life of those drives anyways before that happens (3-5 years). And one of the reasons I have two servers is so that I could blow away one of them when the time comes so that I could re-configure it for whatever the new need is, such as time to expand.

2. How best to replicate to the 2nd server:
- I am thinking the 2nd server will stay off most of the time, and I bring it up say once a month to run a backup
- I know there is the snapshot replication thing, but there are also some good plugins available
- So, what would ppl recommend for performing the backup and why?
- For the cloud part, I will probably use Backblaze or something like that

Any other general suggestions would also be appreciated.

Thanks

-JCL

 

[jcl123]

Reading through some of the guides, some of them mention to not create a vdev with more than 11-12 drives in it, is this still the case? Does that include the parity drives?