I believe I've solved the problem as well as I'll be able to given the limitations of my current hardware. As a quick recap, last night I modified several FreeNAS tunables and was able to bump the speed from 75 MBps to 105 MBps on an rsync between the FN server and Linux server. That's an increase of almost 40%, certainly non-trivial.
This evening I did some research into tuning 10GbE on the Linux side and used a Mellanox guide to tweak some tunables there. The document I used is here, and below is what I entered in Ubuntu:
With both sets of tunables active I was able to measure 155 MBps on an rsync between the two servers, and the Linux server was running at ~97% CPU so it's maxed out. At least this speed proves that everything is running at 10GbE as it's too fast for gigabit, so it looks like nothing is wrong with my ESXi configuration or OS drivers, hooray! Soon I'll be upgrading the CPU in the Linux server and reconfiguring it with FreeNAS to support ZFS replication, but at least now I'm comfortable knowing I'm getting everything the old unit is capable of, and the bottleneck is no longer the NIC for the time being.
This evening I did some research into tuning 10GbE on the Linux side and used a Mellanox guide to tweak some tunables there. The document I used is here, and below is what I entered in Ubuntu:
Code:
sysctl -w net.ipv4.tcp_timestamps=0 sysctl -w net.ipv4.tcp_sack=1 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.core.rmem_max=4194304 sysctl -w net.core.wmem_max=4194304 sysctl -w net.core.rmem_default=4194304 sysctl -w net.core.wmem_default=4194304 sysctl -w net.core.optmem_max=4194304 sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304" sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304" sysctl -w net.ipv4.tcp_low_latency=1
With both sets of tunables active I was able to measure 155 MBps on an rsync between the two servers, and the Linux server was running at ~97% CPU so it's maxed out. At least this speed proves that everything is running at 10GbE as it's too fast for gigabit, so it looks like nothing is wrong with my ESXi configuration or OS drivers, hooray! Soon I'll be upgrading the CPU in the Linux server and reconfiguring it with FreeNAS to support ZFS replication, but at least now I'm comfortable knowing I'm getting everything the old unit is capable of, and the bottleneck is no longer the NIC for the time being.