You and me both. I've been in the networking/VMware game for quite some time, and although I mainly focus on cloud these days, I still do a fair bit of on-prem work.
I don't think I have mentioned the NICs. The ESXi hosts are using Intel 82599.
One NAS:
root@lando[~]# pciconf -lv | grep -A1 -B3 network
oce0@pci0:5:0:0: class=0x020000 card=0xe72a10df chip=0x071019a2 rev=0x03 hdr=0x00
vendor = 'Emulex Corporation'
device = 'OneConnect 10Gb NIC (be3)'
class = network
subclass = ethernet
oce1@pci0:5:0:1: class=0x020000 card=0xe72a10df chip=0x071019a2 rev=0x03 hdr=0x00
vendor = 'Emulex Corporation'
device = 'OneConnect 10Gb NIC (be3)'
class = network
subclass = ethernet
The other looks to be an Intel 82599 again:
root@truenas[~]# pciconf -lv | grep -A1 -B3 network
ix0@pci0:1:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
class = network
subclass = ethernet
ix1@pci0:1:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
class = network
subclass = ethernet
They are happy replicating on the other NIC from one to the other at a sustained 5 Gbps or so doing a replication task.
The second NAS, well, I can rebuild that and do what I want with it without any fear of data loss. I'm tempted to try some other NAS distro and see if the behaviour changes at all.
To be fair, you might be onto something. It could be some weird FreeBSD ism, with the specific type of NIC, that's why I put Scale onto the second NAS just to see. I even tried the on-board 2.5 Gbps NIC as Scale picked that up, no dice.