Network slow down to 1GB NIC Speed / IntelX550

[Viandoriel]

Hello!

since upgrade from TrueNAS 12.0-U8.1 to TrueNAS 13.0-U2 the network speed slow's down to only 1GBit/s instead of 2,5GBit/s.
With Version 12 this was working fine. Nothing else has changed from my side, and I can exclude the other (Client) Side. There is no Switch connected. It's a direct connection to one single PC which got cross checked already. Network drivers on the Client side are up to date (original Intel driver, not via Windows Update).

Hardware is an IntelX550 on board on a ASRock Rack E3C246D4I-2T Mainboard.
Mainboard: https://www.asrockrack.com/general/productdetail.asp?Model=E3C246D4I-2T
Screenshot:

Can anybody help or have an Idea?
I would assume that there is some driver issue on the TrueNAS side. But I have no Idea how I can verify this.

Regards,
V.

 

[Viandoriel]

After there came no reply from anybody, I've just opened a bug report.
Just for Information if anybody else is facing this issue.

 

[MrGuvernment]

Have you tried manually setting the speed on TrueNAS and Windows to 2.5? vs using auto-negotiate just to see?

 

[Viandoriel]

On the Windows side yes. But in TrueNAS I cannot change anything. There is no option at all in the web interface.

Of course my first thoughts about the problem resolution where on the Client side.
But after cross checking the components, I can confirm that its from the Server (TrueNAS) side. Because it happened after the upgrade to TrueNAS 13.0-U2

Meanwhile I've opened already a bug report, as I wrote in the thread before already.
But it was closed without any solution... and to be honest, I also dont even understand the answer.

Here it is:

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

William Gryzbowski
18. Oktober 2022 um 10:40


Hello,
Unfortunately that seems to be an upstream issue we have no resources to investigate.
We will monitor and pull any upcoming upstream fixes.
Meanwhile you might want to consider giving TrueNAS SCALE a shot.

I did not even had the chance to comment this, or to ask the question what exactly they mean with "an upstream issue" ?!
Do you maybe have an idea what this could mean?

 

[spuky]

That it is a Problem in Freebsd or the Intel freebsd drivers...

 

[Viandoriel]

That it is a Problem in Freebsd or the Intel freebsd drivers...

okay, but how can I solve it? :)
Of course I tried already... but all the time I search for "how to install the intel driver on FreeBSD" it comes up to be a very bad idea to do this on TrueNAS. I dont want to destroy something or lock myself out so so.

To be honest, I'm not really a Linux/BSD guy. But I understand the basics.

 

[spuky]

unless you are into driver development you might not be able to...it's more a job for intel and freebsd developers...

looking at the specs of your board... the intel nic is not advertised as 2.5G it is
- 10000/1000 /100 /10 Mbps

the newer steps 2500 Mbps or 5000 Mbps are not that well supported...
and that is on the PC and the Server side.. so a Switch that supports 10G and 2.5gb could get you out of the situation so a 10Gb connection to the switch and a 2.5Gb to the PC

or a 10GB Nic for the PC

or try out SCALE since linux has more usage and with that better driver support..

and did you try u3 wich also updates Freebsd version..

 

[Viandoriel]

Hi Spuky,

ok I got your point - seems like I have overlooked it.
It's just strange that it was working fine, with TrueNAS Version 12.
And I found out that there is a specific driver from Intel available for FreeBSD:

Intel® Ethernet Converged Network Adapter X550 - Prod Info, Docs,...

View products for Intel® Ethernet Converged Network Adapter X550. Find product specifications, technical documentation, downloads and support and more from Intel.

www.intel.com

My personal problem is just that I don't have the skills, how to safely install this driver here. I'm affraid of the risk to "destroy" the system.

and that is on the PC and the Server side.. so a Switch that supports 10G and 2.5gb could get you out of the situation so a 10Gb connection to the switch and a 2.5Gb to the PC

or a 10GB Nic for the PC

Good point. I din't thought about this as a workaround. But in my configuration, I need something like a SAN. For Security reasons there is a point-to-point connection needed. Also Energy saving is a huge thing here. That's why the NAS is allflash. But this has nothing to do with the Network issue... I know... also will not help here... I know ;) That's the thing I have to deal with. :(

or try out SCALE since linux has more usage and with that better driver support..

The guy from Jira said this already. But I think this will be no option currently.

did you try u3 wich also updates Freebsd version..

I will wait until the Software Status for Enterprise is on "Conservative" Status:

Software Status - TrueNAS Roadmap - Open Source NAS Software

Get up-to-date insight on the TrueNAS roadmap. Explore the TrueNAS project timeline and stay informed with the latest updates.

www.truenas.com

The benefit of a fast connection versus the risk of "othger bugs" is simply too high here.


by the way: thanks a lot for your thoughts and effort, I really appreciate that! :)

 

[Viandoriel]

Small update here, but not helpful yet:

1. Updating to "TrueNAS-13.0-U3" does not change anything (I was expecting this, as no network driver thing is in the changelog)

2. Tried the "Realtek NIC known issue" solution from under: https://www.truenas.com/docs/core/corereleasenotes/#known-issues
--> no change. (I know, it's another Vendor, but I was hoping that its influencing it maybe...)


Just in case somebody has the same issue. Until today there seems to be no solution.
Only the Workaround via Switch.

 

[souporman]

2.5Gbps and 5Gbps seem like they should be avoided altogether whenever possible. I don't even understand why the industry went 1Gbps, 10Gbps....long pause....2.5Gbps, 5Gbps.

 

[spuky]

src - FreeBSD source tree

cgit.freebsd.org

is the patch that added support to advertise them via advertise_speed sysctl

This change enables the user to control 2.5G and 5G autonegotiation speeds via advertise_speed sysctl for X550T devices. Due to reported interoperability issues with switches, 2.5G and 5G speeds will not be advertised by default.

sysctl -d dev.ix.0.advertise_speed

should show them as possible values...

so it should be possible to enable them with a tunable...

 

[Viandoriel]

src - FreeBSD source tree

cgit.freebsd.org

is the patch that added support to advertise them via advertise_speed sysctl

This change enables the user to control 2.5G and 5G autonegotiation speeds via advertise_speed sysctl for X550T devices. Due to reported interoperability issues with switches, 2.5G and 5G speeds will not be advertised by default.

sysctl -d dev.ix.0.advertise_speed

should show them as possible values...

so it should be possible to enable them with a tunable...

oooooooohhhhhh
Seems like we're getting closer to a solution :D
You're right, its listed here:

But I have no Idea how I can get it enabled right now.
Maybe you? :/

 

[spuky]

from the hex values in the patch: 37 / 0x00000037 should be the value to enable 2.5 and 5 in addition to 7 ( 100M/1G/10G)

so 0x07
+ 0x10
+ 0x20
__________
= 0x37

sysctl dev.ix.0.advertise_speed=37

should be the value to go with but my system with intel x540 nics gives me invalid argument...
on the console that should go away after a reboot (so trying it out should not harm)

to keep it permanent you'd if it works
you would have to set an entry under system tunables...

variable: dev.ix.0.advertise_speed
value 37
type sysctl

maybe try setting it for the 2nd x550 nic

sysctl dev.ix.1.advertise_speed=37

and see if it works...by plugging in a cable there

And I am not responsible for you locked out of your system..!!

 

[souporman]

And I am not responsible for you locked out of your system..!!

OP, If possible, pop a 2nd 1Gbps line in and create a second way of accessing the system, if this is your only connection.

 

[Viandoriel]

from the hex values in the patch: 37 / 0x00000037 should be the value to enable 2.5 and 5 in addition to 7 ( 100M/1G/10G)

so 0x07
+ 0x10
+ 0x20
__________
= 0x37

sysctl dev.ix.0.advertise_speed=37

should be the value to go with but my system with intel x540 nics gives me invalid argument...
on the console that should go away after a reboot (so trying it out should not harm)

to keep it permanent you'd if it works
you would have to set an entry under system tunables...

variable: dev.ix.0.advertise_speed
value 37
type sysctl

maybe try setting it for the 2nd x550 nic

sysctl dev.ix.1.advertise_speed=37

and see if it works...by plugging in a cable there

And I am not responsible for you locked out of your system..!!

Spuky, I have to say THANK YOU SO MUCH! You are my hero Today! :)

Your Input was 99% right.
sysctl dev.ix.0.advertise_speed=37
Changed the Link Speed to down to 100 MBit/s

So I tried a little bit around...
Of course on the second Channel (ix1), as this is the Channel where I need this speed.
The first (ix0) one is only Admin Console ... Security reasons, as I mentioned already ;)

The Solution here is:
sysctl dev.ix.0.advertise_speed=20

Additionally I activated Jumbo Frames (9k) and moved some files. Looks good! (just as before)





So Man, I'm really Thankfull for your help!
How can I buy you a Coffee now? :)

 

[spuky]

would think that 0x17 would be the value to add 2.5 but maybe you would need to provide it as hex

sysctl dev.ix.0.advertise_speed=0x17

a decimal 20 also flips the same highest bit so in makes sense that it works...

So Man, I'm really Thankfull for your help!
How can I buy you a Coffee now? :)

from your screenshot i see that you at least speak german..

Schön dir helfen zu können... gib das Geld für den Kaffee Leute die es brauchen ;-)

 

[Viandoriel]

would think that 0x17 would be the value to add 2.5 but maybe you would need to provide it as hex

sysctl dev.ix.0.advertise_speed=0x17

a decimal 20 also flips the same highest bit so in makes sense that it works...

Tested and verified it right now: 0x17 also works! :)