Scale ixgbe external card not powering on

[Scepterus]

So then the obvious question is does it work with pfSense on both sides?

You mean from pfSense to pfSense? Why would I loop it?

I would expect a Silicom card to already be unlocked.

It is, at least on the hardware side, as I mentioned, same machine running Windows PE nothing else changed, it works perfectly. When I boot up TrueNAS, the link goes dark. So it is not a hardware lock, or else the WinPE would also have the same issue.

 

[jgreco]

You mean from pfSense to pfSense? Why would I loop it?


It is, at least on the hardware side, as I mentioned, same machine running Windows PE nothing else changed, it works perfectly. When I boot up TrueNAS, the link goes dark. So it is not a hardware lock, or else the WinPE would also have the same issue.

As explained above, these things involve multiple bits. If we want to ascertain where the problem lies, it is best to find a way to characterize the problem. One way to do this is to swap in different things, as you have just done with Windows PE, to see if behaviour changes. Your Windows PE test hints at this possibly being a driver-enforced lock (rather than a firmware-enforced lock).

Since it should be easy to change the OS on your TrueNAS host for testing, then, we have a few more interesting things that we can try, which provide additional insight. Loading pfSense is super-easy as it is an appliance. It is FreeBSD-based, which is something I am very familiar with, and MUCH easier for me to think about than Scale/Debian. If we were to load pfSense on it and get a link, that is very significant to me in the same way your Windows PE test is to you. Plus I understand the driver's mechanism for coping with unsupported SFP's because I have the source code. But we still have to remember that it is entirely possible that the driver itself might not be ABLE to work with the copper SFP's, so we are still moving down a road of needing to experiment with each part to characterize the problem. So we have at least two outcomes we need to explore, driver SFP vendor lock, or overall driver shortcoming where it cannot cope with the copper SFP even though the Windows PE one could.

 

[Scepterus]

If we were to load pfSense on it and get a link

That's a lot more steps than booting to a live Debian USB and see if we get a link. Every time I test these things, it's downtime for my NAS, and all its functions. So I'd rather keep that to a minimum. I'll try to boot to debian asap.

 

[jgreco]

That's a lot more steps than booting to a live Debian USB and see if we get a link. Every time I test these things, it's downtime for my NAS, and all its functions. So I'd rather keep that to a minimum. I'll try to boot to debian asap.

Yeah, but that's not what I wanted. I suggested FreeBSD-based pfSense because I know how to twiddle the driver on that, and you can also live boot the installer CD for that. The idea was to take slow, methodical steps to eliminate variables. You can try jumping all the way to the end if you want, though.

 

[Scepterus]

Yeah, but that's not what I wanted. I suggested FreeBSD-based pfSense because I know how to twiddle the driver on that, and you can also live boot the installer CD for that. The idea was to take slow, methodical steps to eliminate variables. You can try jumping all the way to the end if you want, though.

Well, just booted to Debian live CD, and without doing anything, the 10GbE card works and I even disabled the 1G cable then pinged out, and everything works. So, does TrueNAS use a different driver store or database or something? Because it's the same kernel and OS basically.

 

[Ericloewe]

Which version, specifically? This is sounding like a recent driver change on one side or the other.

Worst case, the SFP was recently blacklisted for poor performance. Best case, a driver bug was fixed that broke 10GBase-T adapters.

 

[Scepterus]

Which version, specifically?

Of what? debian? debian-live-11.5.0-amd64-cinnamon, that's the one I tested.

 

[Ericloewe]

I thought I'd have a look into the Linux kernel commits that reference the ixgbe driver. Holy crap, it's scary how much work is still being put into it after all these years.
One thing that stood out to me is that at least the Linux man page claims that SFP modules must be "known to the driver", even if unsupported modules are enabled. This means you could fairly easily end up with a slightly outdated driver and slightly too-new 10GBase-T adapter.
At this point, it's best to throw this over to the devs. I'd recommend posting your findings in a bug ticket. In addition, I'll take the liberty of summoning the forums' resident dev @mav@, who also has some familiarity with these NICs.

 

[Scepterus]

So the driver on TrueNAS may be outdated? Can't I just update it myself to the latest one on Debian?

 

[jgreco]

So the driver on TrueNAS may be outdated? Can't I just update it myself to the latest one on Debian?

The driver on TrueNAS may be whatever it is, you can check the version numbers.

No, you cannot update it yourself. TrueNAS is an appliance and you're not supposed to be tinkering inside it.

 

[Ericloewe]

So the driver on TrueNAS may be outdated?

Or newer than the one in your live CD. This is one of those things that take 30 seconds to check if you know how to and forever if you don't.

 

[Scepterus]

TrueNAS is an appliance

It's also a Linux machine, and if I can solve this that way, I'd rather do it myself than wait for a patch.

Or newer than the one in your live CD

Code:

 ethtool -i enp3s0f1
driver: ixgbe
version: 5.10.142+truenas

Apparently it's a custom driver.

And the latest driver from intel is:

Code:

Description: Includes Linux*-based drivers version 5.16.5 for Intel® 10 Gigabit Ethernet Network Connections with PCI Express*. View download options.

Version: 5.16.5

So yeah, outdated.

 

[freqlabs]

Code:

 ethtool -i enp3s0f1
driver: ixgbe
version: 5.10.142+truenas

Apparently it's a custom driver.

We use the in-tree ixgbe driver that comes with the kernel. That version is the kernel version. Bluefin has a newer 5.15 kernel, if you want to try that.

And the latest driver from intel is:

Code:

Description: Includes Linux*-based drivers version 5.16.5 for Intel® 10 Gigabit Ethernet Network Connections with PCI Express*. View download options.

Version: 5.16.5

So yeah, outdated.

That is also the kernel version.

 

[Scepterus]

Bluefin has a newer 5.15 kernel, if you want to try that.

Already am, same issue, the ticket I opened is being investigated, hopefully they'll find the root cause.