Melanox ConnectX-3

[Miguel Nunes]

Hi all,

I have aquired a Melanox ConnectX-3 infiniband card that I want to setup on a freeNAS build.

I am using a HP Microserver for which the PCIe version is 2.0. The card is 3.0.

The driver loads at startup, but at a certain point the system crashes.

I have decided to try other opensource OS to figure out if there was any incompatibility and so far it is running well.

Could someone help me figuring out how to make the adapter work?

Thanks in advance.

 

[melloa]

I have decided to try other opensource OS

What are you running that it working out-of-the-box?

By the way, FreeNAS is based on FreeBSD. Up to couple months I had to manually enable to NIC drivers to use it (Mellanox and Chelsio) on FreeBSD. Now I'm installing FreeNAS with Chelsio cards without any modifications. Just FYI.

Also post full specs of your server and FreeNAS version you are trying to install.

 

[indy]

I am using a ConnectX-3 MCX311A-XCAT with FreeNAS 11.2.
Worked out of the box - the only persisting problem is a (seemingly) random disconnect every couple of days.

 

[melloa]

Worked out of the box - the only persisting problem is a (seemingly) random disconnect every couple of days.

Did you look at the logs? Maybe is a good idea to chat with the Devs.

 

[Miguel Nunes]

What are you running that it working out-of-the-box?

By the way, FreeNAS is based on FreeBSD. Up to couple months I had to manually enable to NIC drivers to use it (Mellanox and Chelsio) on FreeBSD. Now I'm installing FreeNAS with Chelsio cards without any modifications. Just FYI.

Also post full specs of your server and FreeNAS version you are trying to install.

I tried Ubuntu Studio and the I was able to use the card.

My Setup is:

HP Microserver Gen 7
16GB of ECC Ram
HP BMC board
4 x 500GB disks with 2 redundant discs.
Melanox ConnectX-3 infiniband card.

 

[Miguel Nunes]

I tried Ubuntu Studio and the I was able to use the card.

My Setup is:

HP Microserver Gen 7
16GB of ECC Ram
HP BMC board
4 x 500GB disks with 2 redundant discs.
Melanox ConnectX-3 infiniband card.

System Information

Hostname freenas.local

Build FreeNAS-11.1-U6

Platform AMD Turion(tm) II Neo N54L Dual-Core Processor

Memory 16313MB

 

[Miguel Nunes]

I have upgraded to the new version of FreeNAS and the system doesn't crash anymore. The card doesn't work though.

I will contact Mellanox with the issues.

Will keep you guys updated.

 

[melloa]

The card doesn't work though.

What does dmesg says about the card? Can it be seeing? It's loading the driver?

Try this from the shell:

Code:

dmesg | grep Mellanox

Below my output with a Chelsio card listed:

Code:

root@nas[~]# dmesg | grep Chelsio
cxgbc0: <Chelsio T320, 2 ports> mem 0xfb781000-0xfb781fff,0xfb780000-0xfb780fff irq 32 at device 0.0 on pci2

For Connectx-3 it should be loading mlx4en driver.

For FreeBSD simply adding to the loader.conf:

Code:

mlx4en_load="YES"

Would do the trick, but with FreeNAS I'm sure things will be different. Hopefully some one from the Dev team can help you with the correct way.

Again, check your dmesg to see if, at least, is been recognized.

 

[ronclark]

I am running Connectx-3 HP branded (649281-B21) Firmware version 5.50.14643.0 duel ports and thy works great. i did a firmware upgrade before i install them.
All three cards thy all came with different versions of firmware.

 

[Miguel Nunes]

Hi there,

Update to my adventure.

I am able too run a connectX card on freeNAS but I have to load manually the folowing modules:

mlx4ib
ipoib

When I switch to a connectX-3 the first modules gives an error.

I can't use the ConnectX cause the bracket if High Profile. I temporalely removed it for testing. I can't use if in a production env.

I have ordered two ConnectX-2 that I am going to test on FreeNAS.

I think there might be a combination problem between card and MotherBoard.

I will keep you guys update. Thanks for all help.