First,
I am not sure how well your ATTO Runs represent actual performance here, since you run them with 256MB test size which might very well fit in some cache or the other. Yes I saw the direct IO box, but not sure thats sufficient to ignore FreeNas in Memory + individual Disk write cache.
Else, I still see several question here (which I can't answer)
1. Why is even a dd test (which is as baremetal as it gets, only dependent on drivers & HW) not able to reach the maximum speed of the drives (SAS writes especially - thats 10 vdevs which means each disk only contributes 340 MB/s to the total).
From my previous tests I am sure a single vdev would reach higher values than that and the more vdevs you add the less benefit you get from each new vdev.
If you have time (and no data on the drives) you could test a 20 disk stripe to see what limit you can come up with there.
2. The next loss is of course network (and protocol). Its quite interesting to see that Windows iSCSI is performing significant better than ESXi
in bhyve locally on the FreeNas box (provided you have enough CPU) to see how that performs with NFS/iSCSI
3. How does the async performance translate to actual sync performance? Whats your slog if any? Or are you fine running async in the end?
In the end of course all that matters is whether the solution you have satisfies your requirements - if SAS3 via iSCSI is enough then its probably not worth spending even more time on this without getting answers.
I am not sure how well your ATTO Runs represent actual performance here, since you run them with 256MB test size which might very well fit in some cache or the other. Yes I saw the direct IO box, but not sure thats sufficient to ignore FreeNas in Memory + individual Disk write cache.
Else, I still see several question here (which I can't answer)
1. Why is even a dd test (which is as baremetal as it gets, only dependent on drivers & HW) not able to reach the maximum speed of the drives (SAS writes especially - thats 10 vdevs which means each disk only contributes 340 MB/s to the total).
From my previous tests I am sure a single vdev would reach higher values than that and the more vdevs you add the less benefit you get from each new vdev.
If you have time (and no data on the drives) you could test a 20 disk stripe to see what limit you can come up with there.
2. The next loss is of course network (and protocol). Its quite interesting to see that Windows iSCSI is performing significant better than ESXi
in bhyve locally on the FreeNas box (provided you have enough CPU) to see how that performs with NFS/iSCSI
3. How does the async performance translate to actual sync performance? Whats your slog if any? Or are you fine running async in the end?
In the end of course all that matters is whether the solution you have satisfies your requirements - if SAS3 via iSCSI is enough then its probably not worth spending even more time on this without getting answers.