Help getting SFP+ connection between Truenas and Mikrotik Switch

[os-mark]

Need some help connecting my truenas server with 10GB intel x520 NIC with my Mikrotik CRS310 series switch.

I've tried removing DHCP from the default NIC and rebooting, then disabling the oboard NIC and booting up with only the intel card. Then assigned DHCP to that interface but no IP is ever assigned, i've rebooted, tried both ports etc. I cant seem to get any lights on the switch or NIC no matter what I do.

The same hardware works fine in my windows machine so the NIC, transceivers and cable all seem fine.

Truenas is working fine on the network with onboard 1GB NIC, but I have installed INTEL x520-DA2 card, and while this card is detected and visible in truenas I am unable to get a link to the switch. I have tested the card and transceivers on my windows system and confirmed they are functional, tried genuine intel transceiver as well but same result... no ACT or 10G lights.

My windows PC is connecting to the same switch with the same hardware and 10GB is working as expected.

Any ideas how I might troubleshoot or try to configure this?

 

[Constantin]

Have you tried turning off auto negotiate and only allowing a given Ethernet connection speed / flow control / etc setting? I remember one of my mikrotiks struggling with also.

 

[jgreco]

Are you using an Intel SFP+ or Intel coded generic optic in the x520? Snap a picture or provide the model number of the SFP+. See more discussion in the 10 Gig Networking Primer in the Resources section for the comprehensive answers.

 

[os-mark]

Have you tried turning off auto negotiate and only allowing a given connection speed?

yes but only on the mikrotik side, tried setting to 1G and 10G. Is there a way to do this in Truenas as well?

 

[Constantin]

I don’t have my TrueNAS console in front of me but I’m pretty sure it’ll tell you what it’s trying to connect at.

For me, the main issue was that auto negotiate failed and had to be turned off while a specific speed and setting had to be turned on. Especially if using SFP optics in a SFP+ cage? Not there ATM, sorry.

 

[os-mark]

Are you using an Intel SFP+ or Intel coded generic optic in the x520? Snap a picture or provide the model number of the SFP+. See more discussion in the 10 Gig Networking Primer in the Resources section for the comprehensive answers.

Genuine Intel card, came with 2 of these cisco transceivers but I also bought this genuine intel https://www.amazon.com/dp/B016YK9CPI?psc=1&ref=ppx_yo2ov_dt_b_product_details

 

[os-mark]

I don’t have my TrueNAS console in front of me but I’m pretty sure it’ll tell you what it’s trying to connect at.

For me, the main issue was that auto negotiate failed and had to be turned off while a specific speed and setting had to be turned on. Especially if using SFP optics in a SFP+ cage? Not there ATM, sorry.

Yes I had to do this with the windows machine, In mikrotik I set it to 1G, then back to 10G and everything lit up. But just cant get any communication going with truenas over SFP+.

 

[jgreco]

Genuine Intel card, came with 2 of these cisco transceivers but I also bought this genuine intel https://www.amazon.com/dp/B016YK9CPI?psc=1&ref=ppx_yo2ov_dt_b_product_details

I wouldn't expect the Ciscos to work. They'll be vendor coded for Cisco and usually there is not a lot of flexibility there.

The "Genuine Intel" you listed is from "Maravi Canada" and is likely to be a fake with fake Intel label. New Intel optics are somewhat difficult to come by; everyone buys the $13.90 generics from 10GTek (SKU AXS85-192-M3-INT-10Gtek) or the $20 generics from FS.COM (P/N SFP-10GSR-85). As far as I know, Intel stopped manufacturing these awhile ago, and more recently they sold their optics business to Jabil. I fear that your "Genuine Intel" seller may be someone who bought a tray of cheap (maybe not even Intel coded) SFP+ and had some labels printed.

The card itself could also be a fake, so I suggest you start by validating the Yottamark on the back of the card. If that passes, then assume something is fishy with the SFP+'s. Fortunately the drivers for both FreeBSD and Linux have the ability to override the non-Intel SFP+ check, see the 10 Gig Networking Primer resource for details.

 

[os-mark]

I wouldn't expect the Ciscos to work. They'll be vendor coded for Cisco and usually there is not a lot of flexibility there.

The "Genuine Intel" you listed is from "Maravi Canada" and is likely to be a fake with fake Intel label. New Intel optics are somewhat difficult to come by; everyone buys the $13.90 generics from 10GTek (SKU AXS85-192-M3-INT-10Gtek) or the $20 generics from FS.COM (P/N SFP-10GSR-85). As far as I know, Intel stopped manufacturing these awhile ago, and more recently they sold their optics business to Jabil. I fear that your "Genuine Intel" seller may be someone who bought a tray of cheap (maybe not even Intel coded) SFP+ and had some labels printed.

The card itself could also be a fake, so I suggest you start by validating the Yottamark on the back of the card. If that passes, then assume something is fishy with the SFP+'s. Fortunately the drivers for both FreeBSD and Linux have the ability to override the non-Intel SFP+ check, see the 10 Gig Networking Primer resource for details.

yes the card appears legit, it has the yottamark hologram on the back. I will order another SFP+ module to add to my collection and see how it goes :) Thanks for the feedback, what a pain.

 

[DigitalMinimalist]

Why are you not using DAC cables?
I bought mine used for cheap and would only use optics for >15m…

As a side note (not related to your x520 issue)
I had to update my x710-DA2 firmware to get 10GBit with my no-name DAC cables (only 1Gbit connection before)

 

[os-mark]

Why are you not using DAC cables?
I bought mine used for cheap and would only use optics for >15m…

As a side note (not related to your x520 issue)
I had to update my x710-DA2 firmware to get 10GBit with my no-name DAC cables (only 1Gbit connection before)

I ordered this one as well last week, but it doesn’t arrive until Jan 9 or so. I was able to get transceivers 1-2 days.

https://www.amazon.com/gp/product/B08YDD2RHL/ref=ppx_yo_dt_b_asin_title_o07_s00?ie=UTF8&th=1

 

[chuck32]

I can confirm the finisar FTLX8571D3BCVIT1 working with the x520. The HP transceivers that came with the card did not work.

 

[Constantin]

Vendor locks are so annoying!

So I’d start there - set up the MikroTik not to negotiate re speed / flow control / etc and then try some new transceivers coded for Intel. I had a very good experience re: getting genuine used gear from Unix surplus, who’ll sell the genuine 520 and transceiver pair for $41 - or transceivers / cards separately.

 

[jgreco]

Why are you not using DAC cables?

Because DAC cables suck when they're a meter too short (or too long). Fiber is cheap and can be ordered to the inch if you want. DAC usually only comes in by-the-meter lengths. There's lots of reasons to just buy the optics and be future-proof. Also, some SFP+ implementations do vendor lock on DAC cables, so going with optics is really the only foolproof solution.

 

[os-mark]

Vendor locks are so annoying!

So I’d start there - set up the MikroTik not to negotiate re speed / flow control / etc and then try some new transceivers coded for Intel. I had a very good experience re: getting genuine used gear from Unix surplus, who’ll sell the genuine 520 and transceiver pair for $41 - or transceivers / cards separately.

That is exactly where I bought my card and transceivers, both card confirmed working. I was unable to get the included cisco transceivers working on either truenas or my windows machine but I bought an intel one and got windows working, now I am on my third transceiver (cisco, intel and now GTek) and still unable to get any lights from the truenas box. Trunas sees the interface just cannot get any Tx/Rx activity, always 0bps. Frustrating.

 

[os-mark]

I really feel like I am just missing something with the interface configuration within truenas, having multiple network cards seems to confusing the matter.

 

[Constantin]

I am so sorry that you are experiencing this. I went through similar issues with my TrueNAS and MikroTik switch. Here is how I configured my switch:

and here is what ifconfig in the shell reports regarding my interface - note, for example that I hard-coded the IP address for this interface on the TrueNAS. Dropping into the shell and running ifconfig on the relevant interface, I get the following reply.

# ifconfig ix0
ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Fast0
options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether [redacted MAC address]
inet [redacted fixed IP address] netmask 0xffffff00 broadcast [redacted network range]
media: Ethernet autoselect (1000baseSX <full-duplex,rxpause,txpause>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
[~]#

You can set the IP address in the Network -> Interfaces Tab. I left everything default in there except I set a fixed IP address at the bottom of the page.

IIRC, I made a point of putting in 1GbE Transceivers since this specific Mikrotik switch is only good for SFP, not SFP+ like the TrueNAS SFP+ interface. While there are some 10/1 GbE transceivers out there, most are fixed speed, IIRC. I hope that helps.

 

[os-mark]

Thanks to everyone for the feedback and help. After switching cables, transceivers, rebooting the switch and truenas a few times I've finally got my connection going. Awesome!

 

[MrGuvernment]

Will note 10GTek off amazon, DACs all, for me have all worked flawless between my Intel SFP+ NICs and Brocade switches. Seems like a safe bet as noted above.