Enabling 2.5gbit link speeds on Intel 10gbit controller

[directhex]

I have a server board with dual 10gbit interfaces, via an Intel X550 chip, driven by the ixgbe.ko driver. Out of the box, the driver offers 100mbit, 1GbE, and 10GbE speeds - due to historical compatibility problems with the supported 2.5GbE and 5GbE speeds. I've tested my network equipment, and I know 2.5GbE works at both ends - but to get it working requires sending magic data to the driver. Specially, as per the documentation:

ethtool -s eth? advertise 0x1800000001028

Except TrueNAS seems to not ship with ethtool. Any bright ideas?

 

[directhex]

OK, easier than I expected, ethtool *is* present, but /usr/sbin isn't in the regular user $PATH so I didn't notice. I just added a couple of pre-init scripts to enable 2.5GbE, and it seems happy now.

 

[flashdrive]

OK, easier than I expected, ethtool *is* present, but /usr/sbin isn't in the regular user $PATH so I didn't notice. I just added a couple of pre-init scripts to enable 2.5GbE, and it seems happy now.

Hello @directhex

thank you for sharing in your very first posts this very interesting news.

Could you please document what exactly you did? This would be extremely useful to me, since I want to enable the 2,5 GBit mode on a couple of T540s in my LAN.

Best,

 

[directhex]

Hello @directhex

thank you for sharing in your very first posts this very interesting news.

Could you please document what exactly you did? This would be extremely useful to me, since I want to enable the 2,5 GBit mode on a couple of T540s in my LAN.

Best,

It varies per driver, so this applies only to the "ixgbe" driver, as per https://www.kernel.org/doc/html/lat...net/intel/ixgbe.html#identifying-your-adapter

I just added one pre-init script per interface, like so:

I don't know what you need for other drivers. Maybe you shouldn't, for some - e.g. the Intel I225-V in my desktop is regarded as particularly buggy in NBASE-T

 

[NickF]

Before the mob comes swarming in...whatever this work around is...it will not work on X520/X540 or any other older intel 10 gig adapters :)
But I am also interested in what you did

 

[directhex]

Before the mob comes swarming in...whatever this work around is...it will not work on X520/X540 or any other older intel 10 gig adapters :)
But I am also interested in what you did

The "on some devices" in the docs doing a lot of heavy lifting

 

[flashdrive]

Before the mob comes swarming in...whatever this work around is...it will not work on X520/X540 or any other older intel 10 gig adapters :)
But I am also interested in what you did

@NickF

Why does it not work?

Did you try?

 

[NickF]

They do use the same ixbe driver. I’d even bet you can maybe make it report 2.5Gb. But I’d also guarantee that the silicon, which was created in the PCI-e Gen2 era, does not have support.

That being said making it “work” and making Linux think it will work are two different things

Watch out for packet loss and misbehavior if you try to do this on older silicon

Also 802.3bz was not even ratified when the X540 launched

2.5GBASE-T and 5GBASE-T - Wikipedia

en.m.wikipedia.org

Intel® Ethernet Controller X540-AT2 Product Specifications

Intel® Ethernet Controller X540-AT2 quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.

ark.intel.com

EDIT: To be clear here.. the command, and specifically the string on the end, 0x1800000001028 is a hexadecimal representation of a bitmask. Converted to decimal:

Code:

000110000000000000000000000000000000000100001000

The 1's represent something being enabled/supported... I cant find anything in the official docs which demystify this into human speak. I askeed ChatGPT for the lulz. It says:

Code:

Bit 0 (...1000 at the end): 10baseT/Half
Bit 1 (...0100 right before the last 4 bits): 10baseT/Full
Bit 5 (...0001000000 towards the end): 100baseT/Half
Bit 13 (...0001000000000000 towards the middle): 1000baseT/Half
Bit 14 (...0010000000000000): 1000baseT/Full

It does not know what the left most bits represent, 00011, which I think we are assuming is what is enabling 2.5 and maybe 5 Gb.

So really, all the command that's being run is doing is saying "HEY, ADVERTISE THESE SPEEDS AS AVAILABLE" but does not take into account the ACTUAL hardware. Just because the driver allowed the change doesn't mean the hardware necessarily will support it.

It's a bit different than like overclocking your monitor. You can maybe run a panel rated at 60Hz up at 75Hz or a panel at 120Hz at 144Hz...but in the case of a network adapter the risks are greater and the problems are a bit more difficult to observe without some inherent networking knowledge.

 

[flashdrive]

@NickF
thank you for the extensive explanation

I will try to enable it and get back, this will take some weeks...

 

[NickF]

Sure. Again, my post is all assuming that it even LETS you commit the change...and I do not recommend trying because I'm fairly confident it won't work as expected if it does allow you to do it.

 

[Lectra]

Ok, I am gonna have to look over the documentation for executing preinit scripts, cause im super new to Truenas, but I was able to accomplish the same thing you did with the following command

ethtool -s enp5s0 advertise 2500baseT/Full on and the same can be done for the 5Gbit as well by switching the 2500 to 5000.

However, its not permanent and does revert back on reboot. Your the first ive seen talking about this problem,

The issue I draw with Truenas is I get that its unstable, or has had reliable issues, but why disable it by default and not provide any documentation for it. --atleast any I could find via google.

FOR THE MOB!!!!! THIS ONLY APPLIES TO X550-t1/t2 and the X225v

 

[Lectra]

I didn't even think to look at the Linux documentation... :(

 

[flashdrive]

@directhex
@NickF
@Lectra

TrueNAS-SCALE-23.10.0

With an Intel X540-T2 DualPort 10G Ethernet Adapter I do get a fail.

How do I check for the used driver?

Is it because of Intel(R) Ethernet Controller X540 versus my X540-T2 ?

linux/Documentation/networking/device_drivers/ethernet/intel/ixgbe.rst at master · torvalds/linux

Linux kernel source tree. Contribute to torvalds/linux development by creating an account on GitHub.

github.com

Use ETHTOOL_xLINKSETTINGS or ethtool_netlink instead of deprecated ETHTOOL_xSET to allow autoneg for 2500baseT Full Duplex (#686) · Issues · NetworkManager / NetworkManager · GitLab

NetworkManager uses the deprecated ETHTOOL_GSET/SSET API to change link settings. The advertising member of its data structure is limited to 32bits which hides and disables all...

gitlab.freedesktop.org

ethtool -s enp1s0f1 advertise 2500baseT/Full on

netlink error: link settings update failed
netlink error: Invalid argument

ethtool -s enp1s0f1 advertise 0x1800000001028

netlink error: link settings update failed
netlink error: Invalid argument



ethtool enp1s0f1
Settings for enp1s0f1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes


ifconfig

eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether --- txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xf7e00000-f7e20000

enp1s0f0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether --- txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet --- netmask 255.255.255.0 broadcast ---
inet6 --- prefixlen 64 scopeid 0x20<link>
ether --- txqueuelen 1000 (Ethernet)
RX packets 144541 bytes 98099269 (93.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 178431 bytes 238192791 (227.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1943 bytes 843741 (823.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1943 bytes 843741 (823.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

 

[directhex]

X540 doesn't support 2.5/5gbit modes, as per Intel

 

[NickF]

Yeah this is expecteD:

Code:

Settings for enp1s0f1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full