Mellanox ConnectX-3 SFP+ Nic not being detected in TrueNAS/Supermicro X8DTN+

[Swillyums]

I recently upgraded my network to 10gb, but I can't get my TrueNAS rig to recognize it.

This is the ebay listing for the product I bought: https://www.ebay.ca/itm/2PCS-MCX341...3?pageci=80ea2655-f689-46eb-adcd-183105ef37f2

Both cards are tested and working in my PC (Windows) and my Dell R710 (ESXI).

My motherboard is a Supermicro X8DTN+.

I have tried the card in several PCIE slots, with and without a riser. It fails to show up in pciconf -lv or anywhere that I have checked. It lights up and flashes when I connect a DAC cable.

I would love to hear anyone's suggestions of things I could try. As far as I've seen, everyone says that it should just work. They card is supported with FreeNAS; the person I bought it to said that he's never had any issues with SFP+ cards in the Supermicro servers, etc.

I'm not really sure what to try, or what the problem may be.

Please forgive me as I would put my knowledge at beginner to intermediate.

Thanks so much for the help.

 

[cojoe1]

Hello My problem is similar. I have two identical rigs except one has the Mellanox ConnectX 3 and the other the Finisar FTLX8571D3BCL. The specs on both rigs have the Supermicro X9SCM-F, Xeon E3 1230V2, 32GB 1600, DDR3, ECC Ram. both have been working fine for years until I upgraded to TrueNAS 12. immediately the SFP+ modules refused to show up, I lost connection (obviously). I am a lost as to what to do. I removed the boot drive, reinstalled FreeNAS11 and recovered everything. I then tried to upgrade again to TrueNAS12 and the same thing happened, I even tried a fresh install as opposed to an upgrade, all to no avail. The indicator lights on on the card does not even show up when the modules are plugged in. Does anyone know why this is happening? I believe that TrueNAS does not have the drivers for these devices. Additionally, I have a total of 22 of these devices deployed in various homes, churches and small businesses in my community and I am a little nervous with this situation. Can someone please help? thank you.

 

[Samuel Tai]

Although the mlx4en driver module for the ConnectX-3 is compiled into the kernel (so the driver does exist in TrueNAS 12.x), it looks like a code commit in July 2020 changed the method the driver used to initialize onto the PCI bus. I recommend you revert your systems back to 11.3-U5, which used the older PCI initialize routine, and submit a bug report that the newer initialization routine doesn't work on your hardware.

 

[cojoe1]

Thank you Samuel. Will do!

 

[ZiggyGT]

Thank you Samuel. Will do!

Had this initialization been changed? I am using Mellanox cards in 11.3 and was thinking of moving to Truenas.

 

[Belphegor]

Had this initialization been changed? I am using Mellanox cards in 11.3 and was thinking of moving to Truenas.

I have two servers running with a Mellanox MCX311A-XCAT NIC in TrueNAS Core 12.0 and both systems are stable with no initialisation problems. I believe the original issue mentioned above was closed and marked as "could not reproduce" at the time.

The best way to test this with your hardware is to migrate to TrueNAS and rollback to FreeNAS 11.3 should there be any unwanted behaviour. Be sure not to upgrade your pools until things run smoothly on TrueNAS Core.

 

[ZiggyGT]

Thanks. I will give TRUENAS a try. This was my backup system and a cascade of errors on my part led to the destruction of the pools. I will have to rebuild from scratch. at one time I had FREENAS running as 10GB bridge to connect all my 10gb clients. In the 10gb primer there is no mention of the Mellanox SPF+ cards. They used to be popular, i was hoping they were still supported in OS.

 

[ZiggyGT]

Thanks. I will give TRUENAS a try. This was my backup system and a cascade of errors on my part led to the destruction of the pools. I will have to rebuild from scratch. at one time I had FREENAS running as 10GB bridge to connect all my 10gb clients. In the 10gb primer there is no mention of the Mellanox SPF+ cards. They used to be popular, i was hoping they were still supported in OS.

I made the plunge updated the drives and moved to TRUENAS. Everything seems to OK. Turns out I changed the 10Gb card to an Intel. To stress it a little I copied a couple huge directories from my FREENAS Server to my TRUENAS server to the other using a Win10 machine to set it up. It has a Solar Flare network 10Gb card. They are connected to an Aruba 2500 switch (has 4x10GB ports). FREENAS is not using the 10Gb Mellanox card but maxing out the 1Gb card on an HP Z400 MB. Why is the FREENAS using the 1Gb card vs the 10Gb card? I did a mount on the Win 10 system to the IP address of the 10Gb card. Here is a screen capture that shows that the mounted drive is 192.164.0.60 on the Mellanox card. The 192.164.0.61 is the FREENAS servers Intel IP address. What am I doing wrong?

 

[NugentS]

do you have multiple network cards on the same IP range?

 

[rvassar]

Just adding some observation/commentary... I picked up a Mellanox ConnectX-3 EN thinking it was a safe & cheap option, but it appears to not be the case. It was intended for an ESX 6.7u3 home lab machine, and I ran into some interesting firmware issues that I chose not to work thru at the moment. Short version, it didn't play well in ESXi 6.7u3 at all... Windows 10, it works just fine. Reading the various release notes, I'm led to believe it would work just fine if I updated the firmware, but I haven't had the time to try it. YMMV... I've had pretty good luck with Mellanox HW over the years, but... Watching and waiting now...

 

[ZiggyGT]

do you have multiple network cards on the same IP range?

Yes I use 192.168.0.51 for 1Gb card and 192.168.0.61 for 10Gb card or TRUENAS and 192.168.0.50 on 1Gb and 192.168.0.60 on 10Gb card

 

[ZiggyGT]

Just adding some observation/commentary... I picked up a Mellanox ConnectX-3 EN thinking it was a safe & cheap option, but it appears to not be the case. It was intended for an ESX 6.7u3 home lab machine, and I ran into some interesting firmware issues that I chose not to work thru at the moment. Short version, it didn't play well in ESXi 6.7u3 at all... Windows 10, it works just fine. Reading the various release notes, I'm led to believe it would work just fine if I updated the firmware, but I haven't had the time to try it. YMMV... I've had pretty good luck with Mellanox HW over the years, but... Watching and waiting now...

I have several dual and single ConnectX-2s that have worked well in FREENAS (they do not run that hot). The ConnectX-3 I have will not work in my HP Z400 server. I also use SOLAR Flare cards for Win 10 clients (when I bought them there was no FREENAS discussion about them but they have good win10 support and were very inexpensive) Recently I obtained an Intel card. It is a pain because I use mostly optical SPF+ cables and the intel cards are fussy about the optical transceivers. (that was a unpleasant $$ surprise) I forgot which card was where when I first posted in this thread. I think the 10GB section could include all the options and caveats.

 

[NugentS]

Yes I use 192.168.0.51 for 1Gb card and 192.168.0.61 for 10Gb card or TRUENAS and 192.168.0.50 on 1Gb and 192.168.0.60 on 10Gb card

Which explains your problem. You cannot have two NIC's on the same network with different IP's. Actually you can - it just leads to some strange issues like you are having where the wrong card is used.

Actually I am surprised I got to answer this as usually our resident grinch is all over this issue as it seems to be a particular "white elephant" of his (for good reason)

I knew there was a resource written on this subject
FreeNAS/TrueNAS Resource

 

[ZiggyGT]

Which explains your problem. You cannot have two NIC's on the same network with different IP's. Actually you can - it just leads to some strange issues like you are having where the wrong card is used.

Actually I am surprised I got to answer this as usually our resident grinch is all over this issue as it seems to be a particular "white elephant" of his (for good reason)

I knew there was a resource written on this subject
FreeNAS/TrueNAS Resource

Very interesting article. Yes I have done this on Windows, and expected this to work similarly. Now I know what I cannot do.
Is this what I need to do?:
To access my servers I should then put them on a separate subnet, say 192.168.1.51 for 10Gb and 192.168.0.51 for the 1Gb card. If I do the same thing for the second server 192.168.1.50 10Gb / 192.168.0.50 1Gb. To access the high speed subnet from my windows machines I will need to add a second IP address on the 192.168.1.X subnet to each of the clients? Some of the Windows clients have 10Gb cards as well, so those get IP addresses on the highspeed subnet. The DHCP server is on 192.168.0.1. Who provides the DHCP on the highspeed subnet? I realize if I go with static IP addresses I don't need, but trying to understand the implications of the separate subnet. Can I access the highspeed subnet from a wireless device if it is only on the slower speed subnet? I have never done anything that was not on the same subnet so please help. In Windows when I try to assign two IP addresses to a single card they both have to be static IP addresses?

 

[NugentS]

why not just use the 10Gb network and not use the 1Gb. NAS on 10Gb connected to a 10Gb switch. Clients attached to the same switch can be either 10Gb or 1Gb (or 10/100, or 40Gb.........) and it will all just work

 

[ZiggyGT]

I see what you are saying. I guess I was too tentative that the 10Gb stuff will work. I wanted another method to reach the clients. I'll just uplug the servers from 1Gb network and give it a whirl. Thanks for being kind.

I disconnected the 1Gb cables and setup the FREENAS/TRUENAS servers only on the 10Gb. I found that the 10Gb cable that I was relying on was plugged into the wrong port and not configured properly. That is why I was not getting it to work without the 1Gb connection. Once I fixed that, everything is working. Performance is not much better than 1Gb for little jpg files. But much better for large movie files. Now I can benchmark and figure out how to boost performance. I'll search around for some advice. Thanks again.

Note: all 10TB from the old system are on the new system. Rate were up to 1.6Gb/sec. That's a copy though the 10Gb Win 10 Client so I guess that is not too bad.