2.5 gigabit connectivity

[Megalodon]

Hello

I've seen several threads on the options for 2.5 gigabit connectivity. It seems the Realtek 8125 might be better supported going forward, but my experience has been that I have not been overwhelmed by the quality of Realtek driver on FreeBSD, and there's been reliability problems.

Something that appears to now be available is the Mikrotik S+RJ10 v2 transceiver, which supports NBASE-T. A lot of previous transceivers claiming support reported 10G to the cage and hence ran into a lot of problems when the link speed disagrees with the reported speed. This transceiver is the first one I'm aware of other than AQS-107 that deals with this satisfactorily, and is quite a bit cheaper.

I've tested it on several switches and my TrueNAS host and it appears to provide 2+ gigs in both directions as expected. With other transceivers the throughput will often drop to <1 gigs due to overrunning the transceiver's buffer, and the rate limiting setups needed to deal with this are non-trivial.

A 10 gig NIC with a 2.5 gig transceiver is a roundabout way to get 2.5 support, but it may be the simplest way in some scenarios. Now that there's cheap unmanaged NBASE-T switches I believe this is quite well suited to a lot of small/home office setups, reusing old cable runs, etc. The main caveat is that I have not been able to force the link to 2.5G, so you may have a situation where the transceiver auto-negotiates to 10G over cabling that can't handle it reliably if you have one of these on either end.

Posting this mostly so people trying the same searches I was doing last year can find it.

 

[Arwen]

One of the issues with a 2.5Gbps SFP+ copper transceiver is that it requires, (as far as I know), the host port NIC to support multi-gig. Early on the SFP+ ports supported 1Gbps or 10Gbps fiber. Not all host ports with SFP+ support the in-between speeds.

Further, their may be power constraints.

But, hey, I could be wrong.

 

[Megalodon]

One of the issues with a 2.5Gbps SFP+ copper transceiver is that it requires, (as far as I know), the host port NIC to support multi-gig. Early on the SFP+ ports supported 1Gbps or 10Gbps fiber. Not all host ports with SFP+ support the in-between speeds.

Further, their may be power constraints.

But, hey, I could be wrong.

Yes it's a bit of a mess, that's why I found the satisfactory performance with this transceiver noteworthy. It's a Mikrotik transceiver and initially I only intended to use it on a Mikrotik switch. In that scenario there's no way to know what non-standard behavior they've implemented. But once I had the transceiver in hand there was no reason not to test it on another switch, where it worked fine, and on my TrueNAS host, where it worked fine.

The Mikrotik switch reported the interface speed as 2.5 correctly, but the other switch and the TrueNAS host both reported 10, yet still provided good performance. My best guess is that the transceiver is reporting the link speed in some non-standard way, a naive interface will see 10 and will attempt to send at 10. How the transceiver avoids having its buffer overrun is not clear to me, it's possible it sends a pause frame to the cage. I'm not sure how to confirm that though because my understanding is that the NIC consumes those without passing them up to the OS, hence tcpdump etc is pointless (tried anyway, didn't see anything).

The only other transceiver I'm aware of that does this is the AQS-107, and it does it by basically implementing a layer 2 bridge internally. That's why it's quite expensive. The S+RJ10r2 release notes state the previous version did not support jumbo frames and that r2 does, which suggests it's a layer 2 store-and-forward device. That's circumstantial though, not sure how to go about getting confirmation.

Either way my testing suggests the S+RJ10r2 satisfactorily supports 2.5 even in a naive NIC or switch with no support for multi-gig. Hence it is likely the easiest way to connect a 10G host to a 2.5G network in a lot of scenarios. The only alternative I was aware of prior to this was to use a switch with both 2.5 and 10 ports to manage the handoff between speeds, or to switch to SCALE.

 

[Arwen]

@Megalodon - Very interesting. Yes, it is probable that the S+RJ10r2 is using pause frames.

Some of us are still waiting more, (and cheaper), higher than 1Gbps networking gear. My new NAS board has 2 x multigig copper ports, (and 2 x 1Gpbs ports), but I will be stuck at 1Gbps for a while.

 

[rvassar]

If I remember correctly the SFP transceiver is actually a 2 port switch in itself, and presents as 10GbE to the SFP+ and either 5GbE or 2.5GbE to the copper. There are some reasonably priced 8 port 2.5GbE "home segment" switches on the market now and 802.3bz supports 2.5GbE on Cat5e up to 100m. The problem I have is I already bumped my home office to 10GbE for fun, and while my house is wired for Cat 5e in every room, there's not one single 2.5GbE host in the rest of the house. The only case I could make for it is if my Internet provider pushed above 1GbE. It may happen some day, but I'll probably be retired.

 

[Morris]

Arwen is correct in her first answer. There are lots of compatibility issues with SOHO 10-Gb and mult gig. It is best to stay to same brand switch and SFP+ or dedicated coper NICs or switch ports. Also look out for flow control as it can bite you. Some applications want it on and others off.