Just to let everyone know of my "progress" .... tested all the disks to be 100% sure that they perform on par.
Spindles
dd if=/dev/da[spindle] of=/dev/null bs=1M count=1000 ~ 160 MB/sec
dd if=/dev/zero of=/dev/da[spindle] bs=1M count=1000 ~ 165 MB/sec
Cache (MLC)
dd if=/dev/da[cache] of=/dev/null bs=1M count=1000 ~ 390 MB/sec
dd if=/dev/zero of=/dev/da[cache] bs=1M count=1000 ~ 175 MB/sec
ZIL (SLC)
dd if=/dev/da[zil] of=/dev/null bs=1M count=1000 ~ 65 MB/sec
dd if=/dev/zero of=/dev/da[zil] bs=1M count=1000 ~ 185 MB/sec
Much as expected and none of the disks drifted for more than a few percentages, so I do not foresee any glitches here.
Here is my list of ZFS values. Most of them are default values, but just to be sure that I am not doing something obviously degenerated to everybody but me ...
[root@nas] ~# sysctl -a | grep 'vfs.zfs.'
<118>vfs.zfs.l2arc_write_boost: 8388608 -> 67108864
<118>vfs.zfs.l2arc_write_max: 8388608 -> 67108864
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_lsize: 0
vfs.zfs.mfu_ghost_metadata_lsize: 0
vfs.zfs.mfu_ghost_size: 0
vfs.zfs.mfu_data_lsize: 0
vfs.zfs.mfu_metadata_lsize: 0
vfs.zfs.mfu_size: 0
vfs.zfs.mru_ghost_data_lsize: 0
vfs.zfs.mru_ghost_metadata_lsize: 0
vfs.zfs.mru_ghost_size: 0
vfs.zfs.mru_data_lsize: 0
vfs.zfs.mru_metadata_lsize: 0
vfs.zfs.mru_size: 0
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 0
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 67108864
vfs.zfs.l2arc_write_max: 134217728
vfs.zfs.arc_meta_limit: 49034384202
vfs.zfs.arc_meta_used: 0
vfs.zfs.arc_min: 24517192101
vfs.zfs.arc_max: 196137536808
vfs.zfs.dedup.prefetch: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.write_limit_override: 0
vfs.zfs.write_limit_inflated: 12882690432
vfs.zfs.write_limit_max: 536778768
vfs.zfs.write_limit_min: 33554432
vfs.zfs.write_limit_shift: 9
vfs.zfs.no_write_throttle: 0
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 4
vfs.zfs.resilver_delay: 2
vfs.zfs.top_maxinflight: 32
vfs.zfs.write_to_degraded: 0
vfs.zfs.mg_alloc_failures: 24
vfs.zfs.check_hostid: 1
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_synctime: 1000
vfs.zfs.recover: 0
vfs.zfs.txg.synctime_ms: 200
vfs.zfs.txg.timeout: 5
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 16777216
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 29
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 10
vfs.zfs.vdev.larger_ashift_minimal: 1
vfs.zfs.vdev.larger_ashift_disable: 0
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.trim_max_pending: 64
vfs.zfs.vdev.trim_max_bytes: 2147483648
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.use_uma: 0
vfs.zfs.snapshot_list_prefetch: 0
vfs.zfs.version.ioctl: 3
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.trim.enabled: 1
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
Still looking for the equivalent of vfs.zfs.stop_doing_10_sec_drops_in_write_and_for_gods_sake_please_stay_below_10ms_latency=1
