SFP+ help

[drsassafras]

First time using SFP+ and have hit a roadblock. Wanted to get some feedback.

I have:

* Chelsio 10GB Dual Port PCIe SFP+ Network Card CC2-S320E-SR 110-1114-30
* Dell PowerConnect 6224 24 Port Gigabit Switch
* QSFPTEK 10G SFP+ DAC Cable, 0.5m (2ft) Passive Direct Attach Copper Twinax Cable for Cisco SFP-H10GB-CU50CM, Ubiquiti, D-Link, Netgear, Mikrotik, Open Switch Devices

When I place the Chelsio into the truenas box, the card was recognized right away and I was able to see the two SFP+ ports appear on my dashboard. After ordering a usb to serial converter, was able to get the firmware updated on the switch. Now Its time to connect them. No lights. Switch port combinations, no lights. Not recognized on either the switch or truenas as plugged in.

So I plugged the same cable into 2 ports on the switch. Lights! So the router appears to be working. Did the same on the trunas box. No lights.

What am I missing? The card appears to be supported. The router seems like it's working fine but the truenas refuses to establish a connection.

Any thoughts?

 

[Samuel Tai]

Did you configure the Chelsio interface under Network->Interfaces? The default state of interfaces on Unix systems is down, unless told otherwise.

 

[drsassafras]

Hmm what would I do? I have an intel 4x 1gb card. When I plug an rj45 into them the lights turn on green and the status in TrueNAS goes from "DOWN" to "UP". I thought the up and down was an indicator of a connection rather than a configuration option....

 

[Samuel Tai]

Try setting an IP address and mask on the Chelsio cxgb0 interface.

 

[danb35]

It's possible that the Chelsio card just doesn't like the DAC--I seem to recall having that problem with one of my cards. Most of my 10G stuff is done using compatible optics from fs.com and fiber patch cables, partially for this reason.

 

[drsassafras]

Nope. I decided to plug my extra cable in, so both cxgb0 and cxgb1 were occupied. Then applied an ip to both. No luck. No lights on router or TrueNAS

 

[drsassafras]

It's possible that the Chelsio card just doesn't like the DAC--I seem to recall having that problem with one of my cards. Most of my 10G stuff is done using compatible optics from fs.com and fiber patch cables, partially for this reason.

That may be what is going on. Drat. Right now I am using that .5m cable. Can you make a suggestion as to what might work? How will I know what works and does not?

 

[jgreco]

Connect one port of the Chelsio to the other with the DAC cable.

At a prompt, type

ifconfig cxgb0 up
ifconfig cxgb1 up
ifconfig cxgb0

And see what the reported media status is.

The reason I specifically discourage the use of DAC cables in the 10 Gig Networking Primer is because it is a minefield of vendor-coded compatibility issues, and they are rookie-hostile in various other ways. Not to mention that they're fixed length.

 

[danb35]

How will I know what works and does not?

The "official" answer is to buy Chelsio-branded optics--quite expensive new, but probably available used for reasonable prices. Another possibility is to buy claimed-compatible third-party optics from eBay--probably the cheapest route. My answer is to buy new compatible optics from fs.com:

Cisco SFP-10G-SR Compatible 10GBASE-SR SFP+ Transceiver Module - FS.com

FS Cisco SFP-10G-SR compatible SFP+ optical transceiver supports up to 300/400m link lengths over OM3/OM4 MMF 850nm via an LC duplex connector.

www.fs.com

Chelsio-compatible is a custom order, but there's no extra charge for that. Corresponding optics for the Dell end, of course. And then patch cables like these:

1m (3ft) LC-LC OM4 Uniboot BIF Fiber Patch Cable - FS.com

Buy 1m (3ft) LC-LC UPC Duplex 2.0mm OM4 Multimode Uniboot BIF Fiber Patch Cable, PVC(OFNR) from reliable fiber patch cable supplier-FS.COM!

www.fs.com

Naturally, in your desired length. There are lots of other options, but I like the uniboot design.

 

[jgreco]

The 6224 should work with generics, I believe. Some of the newer switches, in particular Force10 based ones, went from supporting generics to vendor-coded during a major software version upgrade some years back.

 

[drsassafras]

No change in link status via the interface. Not sure what all the stuff the terminal spit out means.

 

[danb35]

I think the "status: no carrier" is the relevant piece.

 

[Samuel Tai]

Yes, you've confirmed these DAC cables don't work with your Chelsio.

 

[jgreco]

It's vaguely interesting that it detects it as twinax but can't link, but drivers can be weird about these things.

 

[drsassafras]

If the 6224 works with generics, does that mean that any transceiver will work? I found AFBR-709SMZ for $10 on eBay. It says it is Avago computable on fs.com

Or should I select "generic" and use that, or "dell" and use that?

Just trying to figure out how much leeway I have and what kind of deals I can get.

I tried, seemingly unsuccessfully to put this hardware together following some guide on the TrueNAS website. Well I guess it got me this far, which is more than I could have done on my own. The PowerConnect 6224 is great. Loud, but am ordering new fans for it. Only thing that failed me were the patch cables. I read somewhere on TrueNAS hardware recommendations as long as the distance was not great they could be used. While guess that one was a flop.

While I suppose its just a matter of getting the right hardware now.

 

[jgreco]

I don't recall the 6224 being mentioned in the 10 Gig Networking Primer, which is the only "guide" around here I'm aware of.

Compatibility of generic SFP+'s is not guaranteed. The way this works is that there's maybe ten major "models" of common 10G SFP+s, such as the Finisar FTLX8571D3BCL. When Chelsio sells this, it gets a vendor code and becomes the Chelsio FTLX8571D3BCL-CL (note the suffix), but the actual electronics of the SFP+ are exactly the same as the Finisar, or the Intel FTLX8571D3BCL-IT, or the Force10 FTLX8571D3BCL-FC. For a device that accepts "generic" FTLX8571D3BCL, it's VERY likely to take every one of these. However, if you have a card checking the vendor ID, it will only take that specific suffixed part. The only difference between all of them is an integer (IIRC) stored in the configuration EEPROM (IIRC) and a paper label.

However, the FTLX8571D3BCL is not the only 10G optic out there. There are others, and they will have different capabilities. The firmware on your device needs to know how to properly "interface" with the capabilities of the SFP+, so if you stick a different and incompatible kind of SFP+ into a device that only knows how to talk to the FTLX8571D3BCL, it probably won't work. Similarly, a DAC cable is made to look like a dumb-ish SFP+, but they can be vendor-coded, and a device might decide that it doesn't know how to talk to an "unknown" type of SFP+ (which is your DAC end) or might resent it not being coded for the proper device manufacturer (Chelsio in your case). I don't know what your specific problem is.

In any case, used legitimate vendor SFP+'s are generally fairly cheap on eBay. Expect to pay $10-$20. If you prefer new, you've already been referred to fs.com above, which is fine too.

 

[drsassafras]

The 6344 may not have been mentioned in the guide, but I defanalty heard about it somewhere here. Maybe the forums? I spent a lot of time reading about this hardware.

Thanks guys. This gives me everything I need to know. I can find and order the right hardware now and won't make this mistake again.

You guys are the best. I mean really amazing. Will let you know how it turns out after I get the hardware I need.