2.5Gbps and 5Gbps not included in "advertised link modes" on my X550-T2

[tsm37]

Hello, My NIC is Intel x550-T2 and I'm on Truenas Scale TrueNAS-SCALE-22.12.1. Switch is a Unifi Flex 10Gbps.

Please see the screenshot attached. I'm not having issue with establishing a 10Gbps connection. My question is why 2.5Gbps and 5.0Gbps link speed full duplex are not shown in advertised link modes? Do I need to use ethtool command to tell the NIC to include them in the advertised link modes?

Please shed some light. Thanks

 

[NugentS]

err Why?
If its working at 10Gb, why would you care about 2.5 & 5?

 

[tsm37]

err Why?
If its working at 10Gb, why would you care about 2.5 & 5?

This is because if for whatever reasons, the 10 Gbps connection craps out or the auto negotiation with the switch for 10 Gbps fails, then I guess my NIC will fall to 5 Gbps and not 1 Gbps??

 

[jgreco]

Hello, My NIC is Intel x550-T2 and I'm on Truenas Scale TrueNAS-SCALE-22.12.1. Switch is a Unifi Flex 10Gbps.

Please see the screenshot attached. I'm not having issue with establishing a 10Gbps connection. My question is why 2.5Gbps and 5.0Gbps link speed full duplex are not shown in advertised link modes? Do I need to use ethtool command to tell the NIC to include them in the advertised link modes?

Please shed some light. Thanks

The X550 predates the 2.5G and 5G idiocy, and it is quite possible, likely even, that the driver version being used doesn't support it. It is worth noting that Intel doesn't seem to have any intention to support 2.5G and 5G for the X550 on anything other than Linux at this time. iXsystems may be using the upstream Debian driver, in which case you'll need to wait for Debian to include newer drivers, or iXsystems may have a particular reason (such as the common use of X550 on various mainboards that they may be using in their products) for using a well-tested driver for this chipset that nevertheless doesn't include the gimpy 2.5G and 5G stuff. Don't know for sure, but this is the likely set of options.

 

[tsm37]

Understood. If it's not broken, then don't fix it. :) I will move on.

 

[AMiGAmann]

So is my understanding correct, I am not able to use 2.5GbE with this card in TN Scale?

I tried to set it manually to 2.5GbE with ethtool:

ethtool -s enp45s0f1 speed 2500 duplex full autoneg off

The command is accepted, but nothing happens. When checking with "ethtool enp45s0f1" it shows the current speed is still 10000Mb/s.

I wanted to check, if and how much the power consumption is reduced with that lower speed. The card consumes approx. 15W even if there is no link.

 

[tsm37]

So is my understanding correct, I am not able to use 2.5GbE with this card in TN Scale?

I tried to set it manually to 2.5GbE with ethtool:


The command is accepted, but nothing happens. When checking with "ethtool enp45s0f1" it shows the current speed is still 10000Mb/s.

I wanted to check, if and how much the power consumption is reduced with that lower speed. The card consumes approx. 15W even if there is no link.

I'm using both 10 Gbps ports on my x550-t2 and heatsink is hot. I ended up putting a 120mm pc fan blowing directly onto the heatsink.

 

[AMiGAmann]

Yeah, the heatsink is also very hot in my system.

Because 5G would probably be enough for my current system I hoped to reduce the power consumption and heat by reducing the speed. But unluckily it seems not to be supported so I am unable to test.

 

[jgreco]

I hoped to reduce the power consumption and heat by reducing the speed.

If these are important characteristics for you, then why are you using -Tx chipsets? I believe the X710-DA4's we're running are like 6.6 watts for the entire card running full tilt with four 10Gbps SR links; if you go to twinax you can get that down to under 4 watts.

 

[AMiGAmann]

If these are important characteristics for you, then why are you using -Tx chipsets?

Because I did not know before about the differences.

The X550-T2 is quite affordable when buying used. The X710-T2L/DA4 is much more expensive.

When comparing the power consumption it will take me about 4-5 years until the X710 will get cheaper. So yes, I should have bought the X710, it would have been the better choice.

But before changing the existing hardware I wanted to try if I can reduce costs by software changes/reducing speed.

 

[jgreco]

The X550-T2 is quite affordable when buying used. The X710-T2L/DA4 is much more expensive.

The credible X550-T2's on eBay seem to be north of $150. I've been buying AOC-STG-I4S (XL710-AM1, the Fortville part) for $200. The X710-T2L is a MUCH newer part and is expected to be expensive (~$350) because of that, plus it burns more watts (up to 14.2w). Although thankfully maybe only a bit more than twice the watts of the X710 SFP+ stuff.

 

[AMiGAmann]

Here in Germany the X550-T2 costs also around $150, but the AOC-STG-I4S costs around $340 (used). And you need additional GBICs or a DAC cable, don't you?

 

[jgreco]

GBIC's are mostly for early gear. They were replaced with the much better SFP and SFP+ technology. The money that SFP modules or DAC cabling costs is generally massively offset by the much lower cost of the switchgear and power consumption savings on both ends, plus the better performance you get.

You have to shop around for the AOC-STG-I4S. There are other variations on this card depending on what exactly you need, or maybe you do not even need an X710 based card. A lot of folks have had great success with cards like the Solarflare SFN6122F which goes for about $30 (can sometimes be found WITH SR SFP+ optics for that price) and has a driver authored by Solarflare. It isn't as good as an X710, but it is still pretty good.

The point here is that if you're worried about power consumption, running 2.5G or 5G from your server to your switch is probably a bad choice unless you literally have no other option (such as a switch with no 10G SFP+ uplink). Not only will the SFP+ be lower power, but it will also allow simultaneous clients to have more speed at a lower cost (assuming you can buy used). Obviously this may or may not work for you, but I feel terrible when people forge down the awful path of Category cable just because they didn't understand the SFP+ stuff.

 

[AMiGAmann]

Oh, my knowledge seems to have some more gaps, sorry for that. I thought all those modules were called GBICs. I did not know that the newer ones are called SFP/SFP+ modules.

The AOC-STG-I4 is too expensive here. The best solution for me would probably be using Intel X710-DA2 cards, which are below $200.

But I will have to install new (fibre optics) cables in that case. I checked the Intel compatibility matrix and it seems the X710-DA2 does not support SFP+ copper modules like the E10GSFPT module. In that case I could have used the existing cabling.