Yorick, Do you see NVME-oF being useful where the backend storage is actually on ZFS?
I don’t have the expertise to answer that question, sorry.
From what I’ve read, the idea of NVMe-oF is to avoid the translation layer between SCSI and NVMe commands, so reducing latency, when the backend storage is addressed using NVMe.
iSER similarly seeks to reduce latency, also uses RoCE, and is targeted towards backend storage that’s addressed with SCSI, so that’s your SAS3 use case.
If run over Ethernet, both would benefit from “Converged Ethernet”, DCB, so that the overhead of TCP isn’t needed. That requires switches that support DCB.
vSphere can do either: iSER since 6.7, and NVMe-oF since 7.0.
I’m sure your performance-minded customers will let you know which they prefer: iSER towards SCSI storage, or NVMe-oF towards NVMe storage.
I’m just googling things over here.