- Joined
- May 13, 2015
- Messages
- 2,478
EDIT (16 May 2017): Enabling jumbo frames solved the problem. I'm getting > 9 Gigabits/second with the gear described here.
I have a problem with 10GbE networking between FreeNAS servers: they seem to be constrained when receiving network data.
The servers
I'm configuring a new All-In-One server ('FALCON') and am testing 10GbE connectivity between it and my main AIO server ('FELIX'). Both hosts are running VMware ESXi v6.0U3 (build 5224934) and are equipped with Intel X520-DA1 10GbE NICs connected with Twinax cables to a Dell 5524P switch. I've set up FreeNAS and Ubuntu VMs on both ESXi hosts as follows:
FALCON: ESXi host, Supermicro X9DRi-LN4F+, dual E5-2660 CPU, 128GB RAM
FreeNAS VM BANDIT: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 48GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC
FELIX: ESXi host, Supermicro X10SL7-F, E3-1241 v3 CPU, 32GB RAM
FreenNAS VM BOOMER: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 16GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC
As I mentioned above, my problem is that FreeNAS seems to be constrained to roughly half of the available bandwidth when receiving data. Both rsync and replication between the two FreeNAS VMs barely break gigabit speeds. What's odd is that my iperf testing indicates that FreeNAS is capable of transmission speeds at the expected rate -- as long as the recipient isn't another FreeNAS server. I've attached network graphs that show this.
I believe I've eliminated ESXi networking as the source of the problem; iperf reports ~7.7Gbits/second between the two Ubuntu VMs. This is in line with what @Ericloewe and @MatthewSteinhoff both led me to expect in my "Will this gear do 10 gigabits" thread.
I've done absolutely no network tuning or customization either of ESXi or any of the virtual machines. The Ubuntu VMs get satisfactory results 'out-of-the-box'. Do I need to tune the FreeNAS VMs for better receiving performance? Do any of you network gurus have any suggestions? I'm stumped...
Testing with iperf 2.0.5
iperf server command:
iperf client command:
FreeNAS Network graphs

I have a problem with 10GbE networking between FreeNAS servers: they seem to be constrained when receiving network data.
The servers
I'm configuring a new All-In-One server ('FALCON') and am testing 10GbE connectivity between it and my main AIO server ('FELIX'). Both hosts are running VMware ESXi v6.0U3 (build 5224934) and are equipped with Intel X520-DA1 10GbE NICs connected with Twinax cables to a Dell 5524P switch. I've set up FreeNAS and Ubuntu VMs on both ESXi hosts as follows:
FALCON: ESXi host, Supermicro X9DRi-LN4F+, dual E5-2660 CPU, 128GB RAM
FreeNAS VM BANDIT: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 48GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC
FELIX: ESXi host, Supermicro X10SL7-F, E3-1241 v3 CPU, 32GB RAM
FreenNAS VM BOOMER: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 16GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC
As I mentioned above, my problem is that FreeNAS seems to be constrained to roughly half of the available bandwidth when receiving data. Both rsync and replication between the two FreeNAS VMs barely break gigabit speeds. What's odd is that my iperf testing indicates that FreeNAS is capable of transmission speeds at the expected rate -- as long as the recipient isn't another FreeNAS server. I've attached network graphs that show this.
I believe I've eliminated ESXi networking as the source of the problem; iperf reports ~7.7Gbits/second between the two Ubuntu VMs. This is in line with what @Ericloewe and @MatthewSteinhoff both led me to expect in my "Will this gear do 10 gigabits" thread.
I've done absolutely no network tuning or customization either of ESXi or any of the virtual machines. The Ubuntu VMs get satisfactory results 'out-of-the-box'. Do I need to tune the FreeNAS VMs for better receiving performance? Do any of you network gurus have any suggestions? I'm stumped...
Testing with iperf 2.0.5
iperf server command:
iperf -s -w512k -fg
iperf client command:
iperf -c [hostname] -w512k -fg -t60 -i5
Code:
iperf server iperf client Bandwidth ------------------------------------------------- ubuntu(falcon) ubuntu(felix) 7.69 Gbits/second ubuntu(felix) ubuntu(falcon) 7.64 Gbits/second ubuntu(felix) bandit(falcon) 6.75 Gbits/second bandit(falcon) ubuntu(felix) 2.83 Gbits/second ubuntu(falcon) boomer(felix) 6.73 Gbits/second boomer(felix) ubuntu(falcon) 4.60 Gbits/second bandit(falcon) boomer(felix) 2.83 Gbits/second boomer(felix) bandit(falcon) 2.77 Gbits/second
FreeNAS Network graphs


Last edited: