intel 520 10-Gig and usupported SFP+ on 9.3

[briffle]

I have an Intel 520 that was working great in my server, with some generic SFP+ twinax cables. Sent to remote datacenter, and they weren't long enough, so they used some cisco SFP+'s and fiber. I am now getting that its an "Unsupported SFP+ module" and I don't see my 10G nic's listed. (I have a iDrac card I am accessing it through remotely)

From what I have read in the freebsd forums, I need to try adding this to /boot/loader.conf
hw.ixgbe.unsupported_sfp=1
Many people seem to have good luck with it (you know.. Cisco and Nexus are a rather common combination!)

However, in the past, I would use "mount -rw /" to then modify the file. with 9.3, they changed things. now I get:
mount: /: unknown special file or file system

how can I get that setting into the loader.conf in 9.3? is there another file that will inject it at boot?

 

[dlavigne]

Yup, as you have noticed, 9.3 is no longer mounted read-only.

However, what you're looking for in the 9.3 docs is how to add a Tunable.

 

[briffle]

The fun with the documentation is everything in it is about using the gui.. Which you can't do when your network cards aren't detected, and your at the console. (In general, I would love to see some common CLI commands that could be used.. it would be much, much faster for doing things like setting up 25 NFS shares, etc)

 

[dlavigne]

You really don't want to use the CLI for config stuff as it won't be written to the config database.

For your issue though, if you need to set a tunable in 9.3 in order to successfully boot or access the system, you will need to modify it in the grub editor. Interrupt the grub menu, press e to enter the editor, and make your changes there. Note that this will only work for that boot. Once the system is accessible, make it permanent in Tunables.

 

[jgreco]

And people laugh at me for running -SR fiber everywhere and using the vendor-specified optics.

It is more expensive, yes... but it nearly eliminates this sort of thing.

 

[Dave Genton]

And people laugh at me for running -SR fiber everywhere and using the vendor-specified optics.

It is more expensive, yes... but it nearly eliminates this sort of thing.

You do have a point here. While Cisco likes to play that game also and hide the services in IOS for unsupported SFP+ transceivers, and have no such service available at all in Nexus (NX-OS) the Twin-Ax cables and their respective SFP's bound together into one physical segment are not meant to be fully SFP+ compliant. They are fully capable of doing what required over very short distances however the logic in the chipset of those transceivers is greatly reduced compared to a standard "SFP+" Fiber Optic Transceiver. Price alone however should tell you this is a fact when two of these including the cable in the middle can be purchased for under $200 while the next step up, short reach LED based SFP+ optical transceiver has a list price in the thousands, for ONE SINGLE SFP !

I am curious as to the outcome here however. In my home lab I more often use twin-ax vs optics and SR-SFP's for price reasons but have yet to have an issue with them. Currently I do not have them installed in either of my FreeNAS 9.3 boxes but did have early in the 9.2.1.x release train without issues. Please let me know if the sysctl edit allowed you to boot your freenas 9.3, and if so does it consistently booting error free with the tuneable in place ??

 

[jgreco]

Price alone however should tell you this is a fact when two of these including the cable in the middle can be purchased for under $200 while the next step up, short reach LED based SFP+ optical transceiver has a list price in the thousands, for ONE SINGLE SFP !

Yeah, hilarious. Dell seems to want $511 for the branded SR optic but these go for $20-$60 on eBay. Networking and server hardware is a fascinating thing to buy used on eBay; if you understand the dynamics, then you can get some real deals. Otherwise you might be getting some Shenzhen knockoffs. We're at a point now where some networks are actually refreshing their networks and are having to upgrade from SFP+ to copper as many of the newer servers no longer include SFP+, and some networks also seem to be refreshing towards newer 10G hardware. Some people are horrified by the idea of buying on eBay, but especially for something like optics, where the price differential is so high, I can afford to buy several of any item and literally discard ones that don't work and STILL be cheaper off - and that never actually seems to happen.

I am curious as to the outcome here however. In my home lab I more often use twin-ax vs optics and SR-SFP's for price reasons but have yet to have an issue with them. Currently I do not have them installed in either of my FreeNAS 9.3 boxes but did have early in the 9.2.1.x release train without issues. Please let me know if the sysctl edit allowed you to boot your freenas 9.3, and if so does it consistently booting error free with the tuneable in place ??

The twinax is expected to work. The OP had an issue with DC dumbhands installing unsupported optics.

 

[briffle]

I had a person that was going to the datacenter for another reason throw a cat5 into one of the gigabit ports on the server, and was able to get into it, and start working a bit.. I'm not sure if it was that tunable, or a more recent update I applied, but I am able to use those optics.

 

[Borja Marcos]

And people laugh at me for running -SR fiber everywhere and using the vendor-specified optics.

It is more expensive, yes... but it nearly eliminates this sort of thing.

Speaking of this, be aware of a regression problem with the ixgbe drivers. It affects both recent versions of Linux and FreeBSD.

At some point in the past Intel broke the media detection code for SFPs. I guess that the driver author wasn't aware that a SFP module announcing a 10 Gb capability of "Infiniband passive" is a valid 10 GbE passive twinax, and the media detection code doesn't configure the interface properly. It works, but without a valid media identification and it's not marked as full duplex. As a result, if you try to use LACP with them, it will refuse to initialize the interfaces because the standard requires full-duplex interfaces.

I found the problem and I've updated a bug report for FreeBSD. The solution is simple, a mere three lines of code.

Apart from this particular issue, it should work as a charm without the added expense of fiber SFPs. The cables we are using are made by Prolabs and in theory they use Intel electronics, so they should be fully supported.

 

[jgreco]

If you wanted an opportunity for a faster fix to appear in FreeNAS, you can file a bug report with FreeNAS and the devs might roll that into TrueOS early.

 

[Borja Marcos]

If you wanted an opportunity for a faster fix to appear in FreeNAS, you can file a bug report with FreeNAS and the devs might roll that into TrueOS early.

Actually I'm not running FreeNAS with those cards, but I thought you should know.

I imagine the bugfix will be rolled soon, because sbruno has been following the issue contibuting some suggestions for printf's.

I imagine that once he sees the fix it will be applied to FreeBSD. It's a stupid fix, three lines of code.