I've got a bit of experience in the smaller ZFS range (we run five 8 disk supermicros), but it's time to grow and build a larger/faster system. Workload will be iSCSI to ESXi, so sync writes all day long. We currently have about 75 vms with growth planned, and would really like to increase performance above all else (though more capacity is also welcome).
My current thoughts are building this:
System: SuperMicro SSG-6049P-E1CR24H https://www.supermicro.com/products/system/4U/6049/SSG-6049P-E1CR24H.cfm
CPU: 2x Intel® Xeon® Bronze 3104 Processor 6-core 1.70GHz 8.25MB Cache (85W) (not married to this, it's just the cheapest option. Not really sure how to gauge what cpu I need, would love feedback)
Controller: Board ships with 1 Broadcom 3108 AOC for 8 SAS ports. Is this a decent card to run in JBOD for ZFS? I have little experience here.
RAM: 12x 16GB for 192GB ECC
NIC: 1x Intel x710-da4 for 10gbE (+onboard 1gb for management/failover)
Disk pool: 8x 4TB Hitatchi SAS in 1 pool, striped and mirrored for 16TB, probably ~14TB usable?
SSD Pool: 4x1.6TB Intel S3520 in striped mirror for 3.2TB or ~3TB usable?
L2ARC: 1x 960GB intel dc s3520 (came to this via 192GB ram x5 for l2arc size based off https://forums.freenas.org/index.php?threads/formula-for-size-of-l2arc-needed.17947/#post-97362) would only apply l2arc to spinny pool
ZIL: 2x 150GB intel s3520, one per pool.
I *think* I've got the pools designed and sized right for what we need, but would really love any input on the chasis/CPU/SAS controller. I would not mind if there was a way to get a 12 disk rust pool, something like striping two 6disk raidz2. I'm trying to build as much resilience as possible, hence the s3520s all around. Any better options, or should I cheap out a bit on the SSD pool and let ZFS handle the resilience? I could also do m2. PCIe Samsung 960 Pros for ZIL or l2arc (though I dont think they have super caps?)
Obviously I have some idea what's going on, but am not 100% confident. Any help is appreciated!
My current thoughts are building this:
System: SuperMicro SSG-6049P-E1CR24H https://www.supermicro.com/products/system/4U/6049/SSG-6049P-E1CR24H.cfm
CPU: 2x Intel® Xeon® Bronze 3104 Processor 6-core 1.70GHz 8.25MB Cache (85W) (not married to this, it's just the cheapest option. Not really sure how to gauge what cpu I need, would love feedback)
Controller: Board ships with 1 Broadcom 3108 AOC for 8 SAS ports. Is this a decent card to run in JBOD for ZFS? I have little experience here.
RAM: 12x 16GB for 192GB ECC
NIC: 1x Intel x710-da4 for 10gbE (+onboard 1gb for management/failover)
Disk pool: 8x 4TB Hitatchi SAS in 1 pool, striped and mirrored for 16TB, probably ~14TB usable?
SSD Pool: 4x1.6TB Intel S3520 in striped mirror for 3.2TB or ~3TB usable?
L2ARC: 1x 960GB intel dc s3520 (came to this via 192GB ram x5 for l2arc size based off https://forums.freenas.org/index.php?threads/formula-for-size-of-l2arc-needed.17947/#post-97362) would only apply l2arc to spinny pool
ZIL: 2x 150GB intel s3520, one per pool.
I *think* I've got the pools designed and sized right for what we need, but would really love any input on the chasis/CPU/SAS controller. I would not mind if there was a way to get a 12 disk rust pool, something like striping two 6disk raidz2. I'm trying to build as much resilience as possible, hence the s3520s all around. Any better options, or should I cheap out a bit on the SSD pool and let ZFS handle the resilience? I could also do m2. PCIe Samsung 960 Pros for ZIL or l2arc (though I dont think they have super caps?)
Obviously I have some idea what's going on, but am not 100% confident. Any help is appreciated!
Last edited: