SOLVED 10GbE FreeNAS servers are constrained when receiving network data

Status
Not open for further replies.

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
EDIT (16 May 2017): Enabling jumbo frames solved the problem. I'm getting > 9 Gigabits/second with the gear described here.

I have a problem with 10GbE networking between FreeNAS servers: they seem to be constrained when receiving network data.

The servers

I'm configuring a new All-In-One server ('FALCON') and am testing 10GbE connectivity between it and my main AIO server ('FELIX'). Both hosts are running VMware ESXi v6.0U3 (build 5224934) and are equipped with Intel X520-DA1 10GbE NICs connected with Twinax cables to a Dell 5524P switch. I've set up FreeNAS and Ubuntu VMs on both ESXi hosts as follows:

FALCON: ESXi host, Supermicro X9DRi-LN4F+, dual E5-2660 CPU, 128GB RAM
FreeNAS VM BANDIT: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 48GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC

FELIX: ESXi host, Supermicro X10SL7-F, E3-1241 v3 CPU, 32GB RAM
FreenNAS VM BOOMER: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 16GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC

As I mentioned above, my problem is that FreeNAS seems to be constrained to roughly half of the available bandwidth when receiving data. Both rsync and replication between the two FreeNAS VMs barely break gigabit speeds. What's odd is that my iperf testing indicates that FreeNAS is capable of transmission speeds at the expected rate -- as long as the recipient isn't another FreeNAS server. I've attached network graphs that show this.

I believe I've eliminated ESXi networking as the source of the problem; iperf reports ~7.7Gbits/second between the two Ubuntu VMs. This is in line with what @Ericloewe and @MatthewSteinhoff both led me to expect in my "Will this gear do 10 gigabits" thread.

I've done absolutely no network tuning or customization either of ESXi or any of the virtual machines. The Ubuntu VMs get satisfactory results 'out-of-the-box'. Do I need to tune the FreeNAS VMs for better receiving performance? Do any of you network gurus have any suggestions? I'm stumped...

Testing with iperf 2.0.5

iperf server command: iperf -s -w512k -fg
iperf client command: iperf -c [hostname] -w512k -fg -t60 -i5
Code:
iperf server  iperf client  Bandwidth
-------------------------------------------------
ubuntu(falcon)  ubuntu(felix)  7.69 Gbits/second
ubuntu(felix)  ubuntu(falcon)  7.64 Gbits/second

ubuntu(felix)  bandit(falcon)  6.75 Gbits/second
bandit(falcon)  ubuntu(felix)  2.83 Gbits/second

ubuntu(falcon)  boomer(felix)  6.73 Gbits/second
boomer(felix)  ubuntu(falcon)  4.60 Gbits/second

bandit(falcon)  boomer(felix)  2.83 Gbits/second
boomer(felix)  bandit(falcon)  2.77 Gbits/second

FreeNAS Network graphs
bandit-network-graph.jpg boomer-network-graph.jpg
 
Last edited:

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I have a problem with 10GbE networking between FreeNAS servers: they seem to be constrained when receiving network data.

The servers

I'm configuring a new All-In-One server ('FALCON') and am testing 10GbE connectivity between it and my main AIO server ('FELIX'). Both hosts are running VMware ESXi v6.0U3 (build 5224934) and are equipped with Intel X520-DA1 10GbE NICs connected with Twinax cables to a Dell 5524P switch. I've set up FreeNAS and Ubuntu VMs on both ESXi hosts as follows:

FALCON: ESXi host, Supermicro X9DRi-LN4F+, dual E5-2660 CPU, 128GB RAM
FreeNAS VM BANDIT: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 48GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC

FELIX: ESXi host, Supermicro X10SL7-F, E3-1241 v3 CPU, 32GB RAM
FreenNAS VM BOOMER: FreeNAS 9.10.2-U2 (e1497f2), 4 vCPU, 16GB RAM, VMXNET3 vNIC
Ubuntu VM: Ubuntu server 16.04.2 LTS, 1vCPU, 2GB RAM, VMXNET3 vNIC

As I mentioned above, my problem is that FreeNAS seems to be constrained to roughly half of the available bandwidth when receiving data. Both rsync and replication between the two FreeNAS VMs barely break gigabit speeds. What's odd is that my iperf testing indicates that FreeNAS is capable of transmission speeds at the expected rate -- as long as the recipient isn't another FreeNAS server. I've attached network graphs that show this.

I believe I've eliminated ESXi networking as the source of the problem; iperf reports ~7.7Gbits/second between the two Ubuntu VMs. This is in line with what @Ericloewe and @MatthewSteinhoff both led me to expect in my "Will this gear do 10 gigabits" thread.

I've done absolutely no network tuning or customization either of ESXi or any of the virtual machines. The Ubuntu VMs get satisfactory results 'out-of-the-box'. Do I need to tune the FreeNAS VMs for better receiving performance? Do any of you network gurus have any suggestions? I'm stumped...

Testing with iperf 2.0.5

iperf server command: iperf -s -w512k -fg
iperf client command: iperf -c [hostname] -w512k -fg -t60 -i5
Code:
iperf server  iperf client  Bandwidth
-------------------------------------------------
ubuntu(falcon)  ubuntu(felix)  7.69 Gbits/second
ubuntu(felix)  ubuntu(falcon)  7.64 Gbits/second

ubuntu(felix)  bandit(falcon)  6.75 Gbits/second
bandit(falcon)  ubuntu(felix)  2.83 Gbits/second

ubuntu(falcon)  boomer(felix)  6.73 Gbits/second
boomer(felix)  ubuntu(falcon)  4.60 Gbits/second

bandit(falcon)  boomer(felix)  2.83 Gbits/second
boomer(felix)  bandit(falcon)  2.77 Gbits/second

FreeNAS Network graphs
View attachment 17903 View attachment 17904
Did you attempt the network tuning I suggested in the "will the gear do 10Gb" thread you linked? The network buffers freeNAS ships with are okay-ish for 1Gbe but need help when going beyond that.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Did you attempt the network tuning I suggested in the "will the gear do 10Gb" thread you linked? The network buffers freeNAS ships with are okay-ish for 1Gbe but need help when going beyond that.
I had it in my head that your tweaks would only apply when running FreeNAS 'on-the-metal'. But I'm going to give the tunables a try and see what happens. Thanks!
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I had it in my head that your tweaks would only apply when running FreeNAS 'on-the-metal'. But I'm going to give the tunables a try and see what happens. Thanks!
The sysctl setting are for the network socket/buffers so they aren't hardware specific. You may or may not want to change the congestion control algorithm, it'd be worth testing.

Edit: Here's some info on the H-TCP cc algorithm.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
The sysctl setting are for the network socket/buffers so they aren't hardware specific. You may or may not want to change the congestion control algorithm, it'd be worth testing.

Edit: Here's some info on the H-TCP cc algorithm.
Thanks. I've enable H-TCP and tried the tunables below, without success...

I'm beginning to think the problem may lie with the FreeBSD VMX driver. I found this interesting item in sysctl:
Code:
dev.vmx.1.rxq0.hstats.nobuffer: 0
dev.vmx.0.rxq0.hstats.nobuffer: 35087
vmx0 is the NIC we're interested in, vmx1 is on separate vSwitch storage network not related to these tests. I take 'nobuffer' to mean that the vmx0 driver ran out of buffers during my iperf tests. But I could be wrong, and documentation is pretty sparse on Google. In any case, I tried bumping up the number of Rx queues from 8 to 16 with a loader tunable -- the maximum according to the FreeBSD docs. Still no joy...

Next I'm going to dive into the ESXi side of things and see what I can find out about alleviating the problem from that side.

Here are my tunables, with the default values in the 'Comment' field:
tunables.jpg
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I think there was a guy just the other day who was having similar trouble with his virtual NICs, so a driver limitation seems likely.

Any chance you can passthrough a NIC?
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
You could try compiling the latest vmware tools freebsd vmxnet3 driver against your version of freeNAS. I have no idea how old that driver is that ships in the freeNAS kernel.

Edit: Looks like VMware pushed an updated version (10.1.5) of the freebsd tools on 02-23-2107. Here's a link on compiling the .ko from source. Should be pretty straight forward.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I think there was a guy just the other day who was having similar trouble with his virtual NICs, so a driver limitation seems likely.

Any chance you can passthrough a NIC?
I think I'll try that first, before going off on a tangent compiling the driver and dinking with the innards of FreeNAS itself... :rolleyes:
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I think I'll try that first, before going off on a tangent compiling the driver and dinking with the innards of FreeNAS itself... :rolleyes:
You're not changing the "innards" you're just adding the updated driver. You had to add the driver when running it as a VM with the vmxnet3 NIC going back to 9.3, but to each his own. Hope you can get it sorted out.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
You're not changing the "innards" you're just adding the updated driver. You had to add the driver when running it as a VM with the vmxnet3 NIC going back to 9.3, but to each his own. Hope you can get it sorted out.
I remember that link from a couple of years ago, when 9.3 didn't support VMXNET3. I compiled the driver and tried it out back then...

I thought the current version of FreeNAS (9.10.x) used the FreeBSD VMX driver -- the one I linked above -- not the one from VMware. But I'll look into it, because I've been experimenting with passing through the NIC and, and besides not being the optimal solution for my use-case, it's also abysmally slow as well.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Been diving into this problem pretty deeply, without success.

I get over 9 Gbits/second of bandwidth when I rig both servers to boot FreeNAS on the bare metal. These were new, plain-vanilla installations with no tunables whatsoever. I also get these rates between virtualized instances of Ubuntu server 16.04.02 LTS. So the problem isn't the hardware; the NICs, cables, and switch are all working. But running FreeNAS on the metal isn't an option for me. I can't dedicate a Dual E5-2660 server with 128GB of RAM to FreeNAS. That would be massive overkill. I need to utilize the server's resources for other virtual machines.

In addition to the 'inbox' driver, I've tried these additional VMware ixgbe drivers for the Intel X520-DA1/82599 NIC: 4.5.1, 4.4.1, 4.1.1.4. None made any difference.

I increased the ixgbe ring buffers on both ESXi servers to the maximum value of 4096.

I switched both FreeNAS servers over to the E1000 driver. Resulting bandwidth was much slower than when using the VMXNET3 driver.

I swapped out the NIC in the FALCON server with a Mellanox ConnectX EN and a different X520 card. Same results in both cases. But again, I'm pretty sure the problem isn't the hardware.

I passed the NIC through to the FreeNAS VM on FALCON and got results that were even worse than using the ESXi driver. This surprised me, as I expected pass-through to work as well for a NIC as it does for HBAs.

I tried setting FreeNAS tunables using this guide, but results stubbornly remained the same.

It's frustrating that both FreeNAS VMs can transmit at > 9Gbits/second to a Ubuntu VM; but not to each other!

These are the basic results I get, where (as above) BANDIT is a FreeNAS VM on ESXi host FALCON and BOOMER is a FreeNAS VM on ESXi host FELIX.
Code:
iperf server	iperf client  Bandwidth
--------------  ------------  -----------------
ubuntu(falcon)  boomer(felix)  9.37 Gbits/second
bandit(falcon)  ubuntu(felix)  3.06 Gbits/second
bandit(falcon)  boomer(felix)  2.96 Gbits/second

ubuntu(felix)  bandit(falcon)  9.35 Gbits/second
boomer(felix)  ubuntu(falcon)  7.94 Gbits/second
boomer(felix)  bandit(falcon)  3.96 Gbits/second

ubuntu(falcon)  ubuntu(felix)  9.09 Gbits/second
ubuntu(felix)  ubuntu(falcon)  9.37 Gbits/second

I found a FreeNAS bug (#15620) reporting exactly the same behavior I'm experiencing, but the dev team kicked it out as a FreeBSD problem. I think they're right and that the problem is with the FreeBSD VMX driver, so I'm going to register for a FreeBSD bug account and file a report there. I'll post details here as things develop.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
A compromise solution might be to dedicate a 10Gb NIC to the servers and pass them through. It's costly, but not as much as a whole server - and those Xeons should have plenty of PCIe connectivity available.

I get over 9 Gbits/second of bandwidth
Ooh, with X520s? Intel's new drivers look interesting, that's Chelsio territory.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I still think you should try the latest driver from vmware tools (version 10.1.5). They include a binary version if you don't want to compile one yourself. It won't hurt anything, just copy the vmxnet3.ko to your /boot/kernel/ directory and add the tunable
Code:
variable vmxnet3_load 
value YES
as a loader and reboot.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I still think you should try the latest driver from vmware tools (version 10.1.5). They include a binary version if you don't want to compile one yourself. It won't hurt anything, just copy the vmxnet3.ko to your /boot/kernel/ directory and add the tunable
Code:
variable vmxnet3_load
value YES
as a loader and reboot.
I'll give it a shot! What have I got to lose, right? :smile:
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I'll give it a shot! What have I got to lose, right? :)
Just a thought, you might need to unload the vmx kernel module so you don't end up with a driver conflict between the vmxnet3.ko and the if_vmx.ko. You can run a kldunload to drop the vmx module.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Just a thought, you might need to unload the vmx kernel module so you don't end up with a driver conflict between the vmxnet3.ko and the if_vmx.ko. You can run a kldunload to drop the vmx module.
Hmmm... there is no if_vmx.ko module. I think the FreeBSD vmx driver was built into the kernel; at least I can prevent it from loading by adding this to the /boot/device.hints file:
Code:
hint.vmx.0.disabled="1"
I dropped vmxnet3.ko into /boot/kernel and added the tunable. But it doesn't load; I end up with no network.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Hmmm... there is no if_vmx.ko module. I think the FreeBSD vmx driver was built into the kernel; at least I can prevent it from loading by adding this to the /boot/device.hints file:
Code:
hint.vmx.0.disabled="1"
I dropped vmxnet3.ko into /boot/kernel and added the tunable. But it doesn't load; I end up with no network.
Apologies, it goes in the /boot/modules directory, I re-read my post and saw I typed /boot/kernel

Once it's in the modules folder you can run a kldload vmxnet3 to get it running.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Apologies, it goes in the /boot/modules directory, I re-read my post and saw I typed /boot/kernel

Once it's in the modules folder you can run a kldload vmxnet3 to get it running.
I've tried both locations. It loads from /boot/kernel, but won't load out of /boot/modules. My understanding was that it should load from either one. FWIW, the other 'vm' modules were in /boot/kernel.

Even though it loads, I get no network to configure.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I've tried both locations. It loads from /boot/kernel, but won't load out of /boot/modules. My understanding was that it should load from either one. FWIW, the other 'vm' modules were in /boot/kernel.

Even though it loads, I get no network to configure.
Okay, so the binary isn't compatible. If you want to try the vmxnet3 you'll need to compile it from source against the version of freenas you're using. If you follow the instruction that I linked earlier in the thread you can do the compiling in a virtual box vm on your desktop and once you have the kernel specific built .ko copy it into the freeNAS VM your working on.
 
Status
Not open for further replies.
Top