Should get 10GB, getting 1GB

[francisaugusto]

Ok, this is a bit complicate, so bear with me:

I have recently installed a fiber cable linking one room to another. The cable is a 10GB multimode fiber.
This is the setup:
Room 1: Mac Mini with 10GB ethernet card, Mikrotik CRS305-1G-4S
Room 2: Supermicro MB with 10GB SFP+ port, Unifi switch with 10GB SFP+

Room 1 and room 2 are connected via the 10GB cable between the Unifi and and the Microtik switches. Both are reporting a link of 10GB. A link between the Mac and the Mikrotik, as well as the link between the supermicro and the Unifi switch, are all reported as 10GB.

On the Supermicro board, I have ESXi and TrueNAS as a VM. On ESXi I have a VLAN for storage (vlan 50). The TrueNAS has a NIC connected to the port group that has the vlan 50, and the Mac Mini is also connected to the vlan 50. I seem to be able to get traffic between the Mac and the TrueNAS without the router being involved (which would then reduce the speed, as the router is connected to a 1GB link).

But iperf is showing 1GB speeds between the Mac Mini and the TrueNAS! I have no clue on what could be impacting this, but could anyone give me a hint?

 

[jgreco]

This probably isn't a FreeNAS or TrueNAS issue. When virtualized, the networking between a FreeNAS VM and the ESXi host is virtualized and even if you were to have selected a 1Gbps ethernet interface such as an E1000 (em0), is still able to sustain significantly faster speeds.

If you were truly getting 1GB speeds (the capital-B means bytes) then you really are doing very well and getting most of your 10Gbps link.

Presumably you mean "10Gbps" when you say "10GB"; "10GB" really means 100Gbps or 10GBytes, and no commonly available boards sport 100Gbps onboard. You haven't provided any other details, such as the models, versions, or configuration of the hypervisor mainboard, the ESXi version, the type of network card, the driver being used by ESXi, what ESXi reports the link speed to be, what optics you're using, what the Mikrotik and Unifi switches report link speeds to be, what type of virtual network interface you selected, what sort of speeds you get from one VM to another VM (leaving the physical layer out of the equation entirely), etc., etc.

Please also note that you really need to be using iperf3 in parallel mode with large buffer sizes and maybe a few other tweaks to test 10Gbps link speeds.

 

[francisaugusto]

This probably isn't a FreeNAS or TrueNAS issue. When virtualized, the networking between a FreeNAS VM and the ESXi host is virtualized and even if you were to have selected a 1Gbps ethernet interface such as an E1000 (em0), is still able to sustain significantly faster speeds.

If you were truly getting 1GB speeds (the capital-B means bytes) then you really are doing very well and getting most of your 10Gbps link.

Sorry, I meant 10Gbs. I am getting 1.16 Gbits/sec between my Mac and my TrueNAS, and not 10Gbits.

Presumably you mean "10Gbps" when you say "10GB"; "10GB" really means 100Gbps or 10GBytes, and no commonly available boards sport 100Gbps onboard. You haven't provided any other details, such as the models, versions, or configuration of the hypervisor mainboard, the ESXi version, the type of network card, the driver being used by ESXi, what ESXi reports the link speed to be, what optics you're using, what the Mikrotik and Unifi switches report link speeds to be, what type of virtual network interface you selected, what sort of speeds you get from one VM to another VM (leaving the physical layer out of the equation entirely), etc., etc.

Thanks - I did however provided some info: the links between the switch and the machines are all reporting 10Gb, so are the links on ESXi. I am using a group port that has a VLAN and a dedicated NIC on each so that the machines can communicate without the packets going through a router.

ESXi 7.
TrueNAS-12.0-U1
Drivers: e1000 on ESXi (Traffic between VM's are reporting 10Gbs speeds).
Optics: SFP-10G-SR-UTPR (Multimode transceiver) and S+RJ10 (Mikrotik) Ethernet 10GbT transceiver.

Please also note that you really need to be using iperf3 in parallel mode with large buffer sizes and maybe a few other tweaks to test 10Gbps link speeds.

Ok, though, however, iperf 2.0.12 is reporting the right traffic between the VM's, ie. 10Gbs.
Thanks for your attention so far, let me know if there is anything I can add.

 

[francisaugusto]

I tested now with iperf3. No changes. But...
I tested now against another VM on the same vlan, and I got 10Gbs, so I wonder if there's anything to set on FreeNAS, or any incompatibility between the vmware nic drivers. And the drivers are actually VMXNET 3. I wonder if changing to E1000 will improve things.

 

[RichTJ99]

Can you pass the 10gb nic directly to freenas?

 

[neb50]

What network card are you using on the host?

I am using an intel x520 and VMXNET 3 type adaptor and was getting just under 2Gbps on the receive side until I added the "tso lro" options to the interface in FreeNAS.

Now I am getting around 8-10Gbps transfer rate on the receive side and the TX side has always been around that even without the options.

 

[francisaugusto]

The problem was solved. Two things were missing (in addition to setting mtu to 9000 on all hosts):

- enabling jumbo frames on the Unifi Switch
- setting the MTU to 9000 on the vSwitch on ESXi.

Now I got near 10Gbit speed on my storage vlan.