SOLVED Trouble with new Chelsio T520-SO-CR NIC

[B166ER]

Hi,

i recently bought a Chelsio T520-SO-CR (not from ebay). My new switch hasn't arrived yet but i noticed that the card isn't properly initialized. Dmesg shows some errors:

Code:

t5nex0: <Chelsio T520-SO> mem 0xdd300000-0xdd37ffff,0xdc000000-0xdcffffff,0xdd88
4000-0xdd885fff irq 16 at device 0.4 on pci1                                
t5nex0: failed to prepare adapter: 22.                                      
device_attach: t5nex0 attach returned 22

It seems weird that MSI-X isn't used. When i issue an

Code:

sysctl dev.t5nex.0.%desc

in the terminal, the card is not recognized. However, the NIC appears in the Bios as well as lspci + pciconf but not in ifconfig. Should i be concerned that i got a defective card or could this be a bios related issue? I couldn't find anything related to the message "failed to prepare adapter: 22".
Any help or hint is appreciated, thx!

 

[B166ER]

Unfortunately, a bios update from R2.1a to the current R2.4 did not solve the issue. According to the Chelsio FreeBSD User Guide, the output should look like this:

Code:

t5nex0: <ChelsioT520-LL-CR> mem 0xe5300000-0xe537ffff,0xe4000000-0xe4ffffff,0xe5884000-0xe5885fff irq 16 at device 0.4 on pci1t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq

I specifically bought the T520 because it was recommended to work best in FreeNAS. I'm really at a loss here...

 

[Elliot Dierksen]

Chelsio cards do work great with FreeNAS. Unfortunately it sounds like you may have hardware issues with that card. I forget if the card BIOS has an option to reset to default, but that might be worth trying.

 

[B166ER]

Chelsio cards do work great with FreeNAS. Unfortunately it sounds like you may have hardware issues with that card. I forget if the card BIOS has an option to reset to default, but that might be worth trying.

Thx for the tip! I'll look into it. Could it be that it's not the card that is defective but rather one of the optical transceivers? I have a red LED going on the card itself. So yes, it doesn't look too good.

 

[Elliot Dierksen]

Could it be that it's not the card that is defective but rather one of the optical transceivers?

It is certainly possible. It is harder to troubleshoot things when you only have one or two of them. I ran into this when I was enabling the 40G segment in my network. I wasn't able to fully confirm that one of my 40G hosts had a network card that wasn't working quite right until I had a 3rd host with which to test.

 

[B166ER]

I took out all the transceivers, rebooted but got the same error message. The NIC also gets pretty toasty, the transceivers were quite warm to the touch despite the card not having any traffic at all. I now removed the card from the server and will try to install it in my Windows Workstation. But i think that NIC was DOA.

 

[B166ER]

Now it's getting interesting. As it seems, the card installed and initialized just fine under Windows 10 Prof on an ASUS ROG Crosshair VIII Hero. The red LED just indicates that the card has power. After the installation, no LED is active and the control panel shows the device. So it's either a hardware compatibility issue with my mainboard or it has something to do with FreeNAS-11.1-U7 that i'm still using. Now, i'm going to flash the firmware of the card to the latest version under Windows and may try again to install it in FreeNAS (or get another card and leave it in there). I'm a bit reluctant to upgrade to 11.2 because i'm not sure if the upgrade will break anything and i need the system for work.

 

[Elliot Dierksen]

That Chelsio card has been supported in FreeNAS for a long time. I am using a T580 now, but I used a T520 under 11.1 and it worked fine. It may be an issue with the card, the slot in the main board, or slot compatibility issues in the main board.

 

[B166ER]

Good to know. I'm inclined to think it has something to do with the mainboard/slot compatibility issues. Unfortunately, my cpu cooler blocks the first pcie x8 slot of the board, so i used the second pcie x16 slot (which is a x8 as well). Both slots are directly connected to the cpu. I don't see a bios setting that could be of relevance for the issue. I may contact Supermicro and ask for help. I find it weird that such a common card would cause a compatibility issue that wasn't resolved yet.

 

[B166ER]

For everyone with a X11SSM-F: I contacted Supermicro Support. They checked with the LAB and the LAB has not tested this Chelsio NIC, they cannot guarantee it's compatibility. The only listed SFP+ 2-port NIC is the AOC-STGN-i2S.
So in general they only recommend what is supported on their online compatibility list and that is not a lot. Seems like i have to buy a new system. I'd like to give the AsRock Rack X570D4U-2L2T a chance.

According to this list, FreeBSD 11.1 isn't officially supported either:
https://www.supermicro.com/support/resources/OS/C236.cfm

 

[c77dk]

For everyone with a X11SSM-F: I contacted Supermicro Support. They checked with the LAB and the LAB has not tested this Chelsio NIC, they cannot guarantee it's compatibility. The only listed SFP+ 2-port NIC is the AOC-STGN-i2S.

Hmmm, this sounds strange - I'm running on a X11SSM-F with a T520-CR (not the -SO- but close) and have had no issues at all. Getting a similar card running under Ubuntu on my desktop is quite hit/miss, but FreeNAS and now TrueNAS has been running without issues here.

 

[B166ER]

Hmmm, this sounds strange - I'm running on a X11SSM-F with a T520-CR (not the -SO- but close) and have had no issues at all. Getting a similar card running under Ubuntu on my desktop is quite hit/miss, but FreeNAS and now TrueNAS has been running without issues here.

I ordered a second card and will test as soon as it arrives. But the first one works fine in my Windows PC, now that i got my new switch. I'm very surprised about the issue either, didn't expect it at all. Maybe i can rearrange the CPU cooler and try SLOT7 instead of SLOT6. But if it doesn't work i'll have to get another board. And Supermicro boards were always particularly picky about hardware, imho. But yeah, i'm at a loss here.

 

[B166ER]

The second card arrived. I tried SLOT4 and SLOT6 again but no luck, it's clearly not working. Either a compatibility issue or the driver in 11.1-U7. The only thing i could try now, is upgrading to 11.2 and see if it fixes things. This sucks pretty bad.
I'm pretty sure this has something to do with the cxgbe driver. The card itself appears in lspci so it's not that it's not been seen by FreeNAS. It just doesn't get initialized for whatever reason by the driver. The error message comes from the cxgbe driver.

 

[B166ER]

I upgraded to FreeNAS 11.2-U8 but as expected still have the same issue. But now there is an additional error:

Code:

 
t5nex0: fw init failed: 12.
t5nex0: error during attach, adapter is now in recovery mode.

Is there something missing in my loader.conf or rc.conf that is necessary for the cxgbe driver to initialize the card properly? I thought it should work ootb.

 

[Samuel Tai]

According to the cxgbe man page, you may need to add loader tunables t5fw_cfg_load="YES" and if_cxgbe_load="YES" to load the T5 firmware and driver on boot. Note, don't try to add these directly to loader.conf; set them via a System->Tunables entries.

 

[B166ER]

According to the cxgbe man page, you may need to add a loader tunable t5fw_cfg_load="YES" to load the T5 firmware on boot. Note, don't try to add this directly to loader.conf; set this via a System->Tunables entry.

Thx, I tried that over the GUI and it seems that it loads the firmware on boot but i still get the same error. And i was under the impression that this stuff is compiled into the kernel. At least 'kldstat -v | grep t5' shows me all the modules.

 

[Samuel Tai]

Sorry, I edited my post to add if_cxgbe_load="YES". This is actually a separate module in /boot/kernel, so I don't think it's pre-compiled.

 

[B166ER]

We're getting closer now i think. I only get one error message now and no "failed to prepare adapter: 22". And that one error message is still the "t5nex0: fw init failed: 12. t5nex0: error during attach, adapter is now in recovery mode.". I try to add the other stuff too, maybe it's necessary. My mistake before was to edit the loader.conf directly instead of using the GUI.

 

[B166ER]

I added t4fw_cfg_load and t6fw_cfg_load but still this one error:

Code:

t5nex0: fw init failed: 12.
t5nex0: error during attach, adapter is now in recovery mode.

 

[Samuel Tai]

Does the card provide any BIOS messages during boot? It could be the actual firmware is too old for the t5fw_cfg module to deal with. According to https://doc.dpdk.org/guides/nics/cxgbe.html, the firmware for the cxgbe driver has to be at 1.24.17.0 or higher.