10GbE performance issue in one direction

[jims2321]

what is your MTU size? And are you seeing a lot of retries?

 

[mimesot]

Hi,
thank you for your response.

what is your MTU size?

1500. I didn't change it, because i was told i shouldn't use MTU 9000 as a silver bullet. The problem seems to be sth else. Furthermore i was told, that i should not hamper with our router.

And are you seeing a lot of retries?

Thr retries between the fast 9,3Gbit/s connections among is always about 2000 whereas the connections that run about 2Gbit/s only how up to 300 retries.

 

[Spearfoot]

Hi,
thank you for your response.


1500. I didn't change it, because i was told i shouldn't use MTU 9000 as a silver bullet. The problem seems to be sth else. Furthermore i was told, that i should not hamper with our router.



Thr retries between the fast 9,3Gbit/s connections among is always about 2000 whereas the connections that run about 2Gbit/s only how up to 300 retries.

I went through my own 10Gb 'struggle session' about three years ago:

SOLVED - 10GbE FreeNAS servers are constrained when receiving network data

EDIT (16 May 2017): Enabling jumbo frames solved the problem. I'm getting > 9 Gigabits/second with the gear described here. I have a problem with 10GbE networking between FreeNAS servers: they seem to be constrained when receiving network data. The servers I'm configuring a new All-In-One...

www.ixsystems.com

Enabling jumbo frames got me over the hurdle. I know a lot of people simply hate jumbo frames, but they work for me.

Also, have you updated the firmware on your Intel NIC?

Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Network Adapter X550 Series

This download record contains all files required to update the Non-Volatile Memory (NVM) update utility version 3.30 for Intel® Ethernet Converged Network Adapter X550 Series.

downloadcenter.intel.com

 

[mimesot]

Hi,
thank you too for your reply!

Enabling jumbo frames got me over the hurdle. I know a lot of people simply hate jumbo frames, but they work for me.

I will consider this as an option. But do you have an idea how that might come into play with my asymmetrical behaviour among my two FreeNAS devices?

Also, have you updated the firmware on your Intel NIC?

Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Network Adapter X550 Series

This download record contains all files required to update the Non-Volatile Memory (NVM) update utility version 3.30 for Intel® Ethernet Converged Network Adapter X550 Series.

downloadcenter.intel.com

No, I didn't, but that's a good idea in general in order to rule some issues out.

 

[Spearfoot]

But do you have an idea how that might come into play with my asymmetrical behaviour among my two FreeNAS devices?

I do not... FreeBSD lags behind the Linux world when it comes to device drivers for newer hardware, such as your Intel NIC. So perhaps that's the explanation.

 

[mimesot]

I do not... FreeBSD lags behind the Linux world when it comes to device drivers for newer hardware, such as your Intel NIC. So perhaps that's the explanation.

Since the cards are both X550 but of a different age this might explain it in case this means they have different firmware and linux doesn't care about that whereas freenas potentially does. Thats at least a path worth investigating. Thank you for this idea.

 

[no_connection]

should not hamper with our router

If you are routing then that is a whole other layer of issue.
If you meant just through switch then look up what MTU it would accept, most modern should be fine with 9 or 10k. So that would not matter.

Did you use VLAN or not from FN?
Have you tied to increase and decrease MTU by a 100 and see if there is any issue related to frame splitting?

And if you do a few different MTU size and the throughput is linear then you have to find the cause of packet rate limiting. Maybe change NIC to pull based instead of interrupt.

With the last few posts in mind, it might just be driver having a max interrupt rate that was changed in later hardware/driver. Who knows.

 

[mimesot]

Hi!
You bring in so many new concepts to me that i hope you excuse me if I ask a few possibly stupid quesions.

If you are routing then that is a whole other layer of issue.
If you meant just through switch then look up what MTU it would accept, most modern should be fine with 9 or 10k. So that would not matter.

I always thought I should set the MTU to the same value on all network components in order to avoid conflicts. The router is not involved in the data traffic between the servers, because they are on the same Subnet/VLAN. So, does that mean I can set the MTU to 9000 on all my servers and the core switch and it will work even though most other network users and the router will stick to the traditional 1500?

Well, I just tried it and - surprise - i got the freenas<->freenas Connection up to 6Gbit/s in the bad direction and 9.8Gbit/s in the good direction. I still wanna find out what the problem is though.

Did you use VLAN or not from FN?

Yes, I use a VLAN and I use the same VLAN on all devices. The data packages should enter and exit all involved devices in a tagged state.

Maybe change NIC to pull based instead of interrupt.

With the last few posts in mind, it might just be driver having a max interrupt rate that was changed in later hardware/driver. Who knows.

Can you tell me how to figure this out?

Greetings
mimesot

 

[no_connection]

I always thought I should set the MTU to the same value on all network components in order to avoid conflicts. The router is not involved in the data traffic between the servers, because they are on the same Subnet/VLAN. So, does that mean I can set the MTU to 9000 on all my servers and the core switch and it will work even though most other network users and the router will stick to the traditional 1500?

In short there are three things that can happen, or 4. A switch is ether going to pass or drop, I did ckeck MikroTik and most accept 9-10k so shuold be fine. So as long as the parts in the middle can pass higher then you only have to worry about the end points speaking the same size.
A router can drop, or drop and reply back with "hey it's too big you have to lower it or you shall not pass". 4th option is the tricky one and it simply split or fragment the packet, problem with that is the remaining packets don't have all header info so could get lost in the interwebs side roads. Which is why everyone should go for the drop and reply option. So probably not a problem any more.

In short whenever it wants out on the internet it will figure out how to adjust even if it happened to have a too high MTU. So you can try it and if it happens to cause problems and then go from there.

Yes, I use a VLAN and I use the same VLAN on all devices. The data packages should enter and exit all involved devices in a tagged state.

Since VLAN add to MTU it could push something too far, it might be fine but to know for sure you could test to done/up it by 100 just to see what happens.

Can you tell me how to figure this out?

I never tried it but I do remember it being a thing,

polling

www.freebsd.org

 

[FreddyAV]

Hi!

I have the exact same problem and I'm also only using iperf3 to isolate the network part of FreeNAS just as the OP does. After having tried to swap a lot of parts (cables, switches, network cards) I have just as the OP built an additional machine (not exact same hardware though) to experiment with and the default behaviour is the same. My network cards are Intel x520 (both T2 as well as SFP+ based ones), but some branded as HP.

After fiddling with loads of tuningparameters I just this morning got a jump from the default behaviour (9.4 Gigabit and approx 1.0 - 3 Gigabit) to jump up to a situation where I'm now at roughly 9.4 / 7.7 - 9.3 which is a huge improvement. I still need to fiddle with the tuning parameters but will report back once I have it working and I know which combination of tuning parameters makes it work. Will continue after I get back from work today and report back.

Cheers!

 

[FreddyAV]

Ok, as I wrote, without any tunables I have (all using iperf3) 9.4gigabit/s pushing data to the freenas box (from Linux Mint / Ubuntu 18.04) and between 1.1 and 3.3 gigabit/s reading data from the freenas box. I also get 9.4gigabit/s in both directions between two Linux boxes. Reading through this thread I noticed that the good transfers always had a Cwnd of the size of 1-2.7 Mbytes whereas the problematic transfers used a Cwnd size of approx 50-100 kbytes on the FreeNAS side. So trying to raise the Cwnd size according to this (going from no tunables):

If I set:
sysctl net.inet.tcp.abc_l_var=44
yields transfers between 3.8 and 9.2 in the problematic direction and Cwnd sizes between 100 - 700 kbytes! :)

Adding also:
sysctl net.inet.tcp.initcwnd_segments=44
and I end up with transfers between 6.4 and 9.2 (with averages over 10s being approx 8.5) so getting there! Cwnd is now in the range 350-750 kbytes.

I'm gonna try to fiddle with more tunables tomorrow and see if I can get it a bit more stable (i.e. less fluctuating).

I should also mention that through this the working direction still stays the same (9.4) with very small fluctuations.

Hope this helps someone else!

 

[FreddyAV]

Oops, just noticed something. I had added the following to my NIC:
rxcsum txcsum tso4 lro
but after removing them, the transfer speed is if anything even more stable.

 

[ianrm]

Hi folks,
a comment about performance of the 10gb interface. I moved some files around the a couple of days ago and was very disappointed with the performance. The files were form a an SSD attached to my current model mac mini.
So, had another brainstorm and decided to move large files from the mac mini internal drive. The speed of transfer was so fast the performance was not registered on the mac or the freenas. The files ranged inside from 300MB to 700MB.

Ian

 

[Spearfoot]

For what it's worth, here are the tunables I use on my systems. I get near-line-rate (~9.8Gb/s) iperf performance sending & receiving:

EDIT: I documented the system default values in the "Comment" field.

 

[FreddyAV]

Thanks Spearfoot! Some questions though:
Are your SolarFlare SFN6122F 10GbE using the ixgbe driver in FreeNAS? I couldn't find anywhere which could confirm it using an Intel chip sorry.
Are the speeds you are getting tested between FreeNAS boxes or have you also tested the speed from a Debian/Ubuntu/(any derivative) machine?

I will try your settings and see what results I get.

 

[Spearfoot]

Thanks Spearfoot! Some questions though:
Are your SolarFlare SFN6122F 10GbE using the ixgbe driver in FreeNAS? I couldn't find anywhere which could confirm it using an Intel chip sorry.
Are the speeds you are getting tested between FreeNAS boxes or have you also tested the speed from a Debian/Ubuntu/(any derivative) machine?

I will try your settings and see what results I get.

When I run FreeNAS on-the-metal, it uses the sfxge driver for the SFN6122F's. On my all-in-ones, I configure FreeNAS to use vmx. Same performance either way.

Oh, and I use jumbo frames, too. Flaming to start in 1, 2, 3...

 

[FreddyAV]

Ok, thanks, I suspected as much from the hw.vmx parameters. Unfortunatly, for my particular setup the tunables you listed does not result in any changes to my issue. The only two that seem to impact speed at all are as mentioned:
sysctl net.inet.tcp.abc_l_var=44
sysctl net.inet.tcp.initcwnd_segments=44

It still varies a bit and I\m still missing roughly 1Gpbs in speed in the problematic direction. I know a lot of people are against jumbo frames but I will look into it again tomorrow.

I can now also verify that between two FreeNAS boxes my speeds are very stable at approx 9.4Gigabit/s, irrespective of tunables. Running Ubuntu on the same hardware (on one of the servers) yields the problematic results. How can I check what two FreeNAS boxes agree on but one FreeNAS and one Ubuntu does not agree on and then results in the problematic behaviour?? Anyone? Please!?

 

[mimesot]

Ok, as I wrote, without any tunables I have (all using iperf3) 9.4gigabit/s pushing data to the freenas box (from Linux Mint / Ubuntu 18.04) and between 1.1 and 3.3 gigabit/s reading data from the freenas box. I also get 9.4gigabit/s in both directions between two Linux boxes. Reading through this thread I noticed that the good transfers always had a Cwnd of the size of 1-2.7 Mbytes whereas the problematic transfers used a Cwnd size of approx 50-100 kbytes on the FreeNAS side. So trying to raise the Cwnd size according to this (going from no tunables):

If I set:
sysctl net.inet.tcp.abc_l_var=44
yields transfers between 3.8 and 9.2 in the problematic direction and Cwnd sizes between 100 - 700 kbytes! :)

Adding also:
sysctl net.inet.tcp.initcwnd_segments=44
and I end up with transfers between 6.4 and 9.2 (with averages over 10s being approx 8.5) so getting there! Cwnd is now in the range 350-750 kbytes.

I'm gonna try to fiddle with more tunables tomorrow and see if I can get it a bit more stable (i.e. less fluctuating).

I should also mention that through this the working direction still stays the same (9.4) with very small fluctuations.

Hope this helps someone else!

Wow, thank you, I will try that as soon as I can.

For what it's worth, here are the tunables I use on my systems. I get near-line-rate (~9.8Gb/s) iperf performance sending & receiving:
View attachment 41403

EDIT: I documented the system default values in the "Comment" field.

Thank you as well

 

[FreddyAV]

I have found that FreeNAS to FreeNAS is working irrespective of the abc_l_var / initcwnd_segments settings above and also using iSCSI on a Windows 10 desktop I get almost full speed out of my network. It does then seem like it is the Linux side that is problematic.

To this end I have started to try a few variants of Linux Mint & Ubuntu on my Linux desktop machine (in posts above running Linux Mint 19.2 Cinnamon amd64 with kernel: 5.0.0-31-generic #33~18.04.1-Ubuntu SMP Tue Oct 1 10:20:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux) and I have found that Linux Mint 17 Cinnamon amd64 (booting from the install-ISO) gives way better performance out of the box. From that I do get about 7 gigabit/s slowly rising to full speed with standard setting for abc_l_var / initcwnd_segments. Setting them to 44/44 I get a stable full 10Gig speed from iperf3. I have also verified this by copying some 40-odd-gigabyte files from my FreeNAS box through NFS. For my normal install (Mint 19.2) I get about 200-300 Mb/s while copying the files into /dev/null but from the Linux Mint 17 ISO I do get 500-700 Mb/s doing the same operation from the same pool. I should also mention that Linux 20 performs poorly (on par with Mint 19.2).

I have dumped the sysctl settings into files I'm comparing but there are several other possible sources of the problem/change, e.g. kernel, drivers, etc. I have also found some old reports saying that one should probably disable lro on the Linux side but apparently that means rebuilding the kernel and drivers. Will look into this later.

I guess this means that this turns into a Linux issue rather than a FreeNAS issue but I'm personally happy to report back here for anyone interested. I have to quit now though and won't be back until the end of the week due to work.

Cheers!

 

[FreddyAV]

For anyone interested: I have now narrowed it down to being a problem somewhere after kernel 4.15 (from the original Mint 19.2 install) because this one works as intended (if I use 44/44 for the two cwnd settings from above). I ran out of time this morning and had to be off to work but I will compare sysctl dumps for this kernel and the 5.0.0 which is problematic on the same machine to see if there is anything there of interest. Otherwise I will try more kernels. I guess this means it will either be a problem report to mint or ubuntu for regarding settings or a problem report on the kernel/driver.

Anyways, what I'm saying is that for an install of Linux Mint 19.2 and with both abc_l_var / initcwnd_segments set to 44 on the FreeNAS side kernel 4.15 gives the expected speed on the network whereas kernel 5.0.0 does not (approx 1/3 of expected speed). Then there is the obvious problem for Linux that e.g. Windows 10 can get the expected speeds without setting abc_l_var / initcwnd_segments to anything than the default 2 and 10 but that is an other issue, or from my point of view just a matter of "optimization".