Hello and welcome to the journey :)
We will have all data backed up,
Great - this would mean that you could easily avoid a few pitfalls you're already deep into..
2x6RAIDz2: W:330MB/s - Is RAIDz2 really this slow?
The correct way to put it is; RAIDZ2 isn't really that slow, but your drives are.
The number is along the lines what to expect.
Note the first 5-GB are much faster (HD caches maybe?) It starts at about 900MB/s and slowly drops throughout the 32GB file with the big drop around 5GB mark. For instance a 4GB file did about 930MB/s, and an 8GB file is about 550MB/s
The first 5GBs are faster due to ZFS will 'collect' the data into big enough transaction group in RAM prior it being flushed out to disk. This happens to default at whatever happens first, 5 seconds of data, or X amounts of GB. (don't recall on top of my head exact size of the default setting). This is also why it is difficult to stare at a transfer speed, even from the POV of ZFS to judge the speed.
Welcome to your pitfall of measurements, first you're looking at data from a 32GB file, which gives some sort of average both speed and time. As the speed is heavily accelerated in the beginning due to the data being soaked into transaction groups prior to being flushed to disk, both the 4GB and 8GB measurements will be skewed, the smaller file = more of the "cache soak" than the larger file, thus bigger difference in numbers.
This is also why it does not make sense for you to do benchmarks other than such that PRECISELY mimics what your actual work load is.
Since you've already stated you have the data backed up, the best thing is to start testing with actual data, on actual drives.
Release yourself from this need to benchmark fictions ;)
Also, strange that write is faster than read, no?
No, due to reasons above.
Am I missing something (other than actual test of the true workload)?
Good on you! This is exactly what you are missing.
So I tried yet another layout: 3 x 4RAIDz2.
This might be the best balance. (If this is a bad idea for some reason please tell me!)
Meanwhile a Z2 solution might be fine and dandy with 20% pool utilization, it might be horrendously slow for the users when being in use for a bunch of months seeing a lot of data come and go, plus being used near the 85% mark.
I think this solution makes the least amount of sense. It is sort of taking a sports car and putting bigger tires on it, to claim it would do well off road too.
I'm for a polarizing setup. Any time performance is of interest, and "cost of hardware lost to redundancy" is acceptable - run with mirrors.
The only reason for using in this setup is due to you really need the space and cannot afford mirrors.
If Z2 is adequate performance to your needs - happy times - profit the better value of your hardware.
Rather than half-assing two things, I'm typically an advocate of dual-pool setup whenever there is both a requirement of storage, and speed.
Usally the "working set" can be fit ontop a few SSDs/nvme.
Then the storage pool can comfortably be setup for space maximization as priority rather than performance.
For example, 10wide z2.
Any info/suggestions welcomed and appreciated. Thanks.
Keep in mind, that pools performance degrades with fill rate/fragmentation. Nothing will be ever as fast as an empty pool.
Performance will be fastly different with a 30% fill rate compared to a 84% fill rate.
There are a few threasholds, generally, somewhere around 40-50% is where you have maximum performance.
Beyond 80% the decline is really steep. ZFS basically shifts logic to maximize space instead of performance.
IIRC there are threasholds aroun 85%, 90% too, but here is a territory where you don't want to go. Both due to performance, but also because if you accidentially over-fill your pool, you cannot delete files. ZFS is Copy-on-Write which means it actually needs space - to delete things.
There are some stories on the forums where people basically lost a fully healthy pool due to it being overfilled and - could not be saved.
Set quotas.
I suppose most of our files don't exceed 5-10GB
Then I suggest you don't even bother doing experiments with files larger than that, to not skew your expectations about what the system can do.
The reality is that both the 'transaction group' buffering in RAM for writes, and ARC for reading cache (more RAM the merrier!) will HEAVILY skew what your system "appears" capable of - which is the holy beauty of ZFS. It does more magic during "normal workload" than benchmarks really capture. Ie, a system that benches better than what you've seen so far, does not by any stretch mean ZFS won't be a better user experience!
Ugh! I hate commitment! ;-)
Treat the first pool or two layouts as mere one night stands and you'll be golden.
Forget expectations about future relationships - just pure fun for the moment ;)
