This VM has 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz cores
If I don't give you much to go on, how about linking that comprehensive list of information and how to get it I could follow?
I have shown the fio commands I use but here goes:
TrueNAS:
fio --name=random-write --ioengine=posixaio --rw=write --bs=4k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based
Run status group 0 (all jobs):
WRITE: bw=460MiB/s (482MB/s), 27.2MiB/s-29.6MiB/s (28.6MB/s-31.0MB/s), io=27.0GiB (29.0GB), run=60007-60009msec
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based
Run status group 0 (all jobs):
WRITE: bw=30.4MiB/s (31.9MB/s), 1925KiB/s-1994KiB/s (1972kB/s-2041kB/s), io=1830MiB (1918MB), run=60075-60115msec
I think what I witnessed her eis the first job doing very well up until 50%. It was going at almost 1GiB/s. Then took a knee to around 100+MiB/s. That's probably when the ZFS cache ran full.
For the second test, ZFS cache was still full, so abysmal performance from the start.
So yes, this would indicate poor performance by the P2s. However even at the 120 Mb/s you show in your test, I have two vdevs and should be able to sustain 250 MB/s over the whole pool, right?
So let's get down to basics: How do I performance test the hardware devices directly?
EDIT: What is worse about the P2s, Crucial seems to have swapped the TLC for QLC chips without changing model numbers. I might have a mix going here... God damn it, the enshittification these days. Can't trust anybody...