Proxmox 5.1 Mellanox Connect-X2

[bumann]

Hi,

i want to build a Mellanox IP Conenction between my Freenas and Proxmox Server.

Both Servers have dual Port MHQH29-XTC Cards inside.



I try:

modprobe mlx4_en
echo eth > /sys/bus/pci/devices/0000\:0d\:00.0/mlx4_port1

echo eth > /sys/bus/pci/devices/0000\:0d\:00.0/mlx4_port2

set a ip Adress

ifup the interface

On Freeenas i see the interface is active, but i can not ping the other server.

 

[Nick2253]

I don't actually see a question here. Is everything working? Not working? Are you not getting full speeds?

 

[bumann]

sorry, it is not working. I have build another test setup, with two freenas server, it dind´t work, too.

 

[Nick2253]

You should just be able to connect the controller in FreeNAS, configure a static IP address, do the same in ProxMox, and they should be able to communicate.

Are you able to see the hardware in FreeNAS? Does the GUI detect the NICs?

 

[bumann]

Yes, the gui shows the interface and i setup static ip and netmask. The gui shows the nic port as active.

 

[Nick2253]

That would indicate to me that it is working. What makes you thing it's not working? What errors/behavior are you getting?

 

[bumann]

I can not ping to the other host.

 

[Nick2253]

Are you also seeing the NIC in your ProxMox server?

Are you connecting with fiber? If so, are you sure that your LC cables are oriented properly? Do you see lights in your SFP modules?

 

[bumann]

I connecting with QSFP Cable. i have setup a other test with two freenas systems an these cards. setup the fixed ip but i can not ping, too.
I see the lights on the interface blinking when i do ping.

 

[Nick2253]

On the off chance this is the problem, are you sure both systems are configured to respond to pings over the Mellanox interfaces?

This should Just Work™. I would suggest putting one of the cards in your desktop, and try using that to troubleshoot. That would at least confirm that the cards are working, and FreeNAS is working correctly. At this point, it's difficult to say what exactly is going on: are you sure both cards are working? That the cable is good? Is there a configuration issue? Is there a driver issue?

 

[bumann]

The two freenas test systems are fresh installed with the newest freenas. i have flashed the newest firmware to the mellanox cards too.
Do you think the default setting from freenas is to block ping on mellanox interface?
I think on the gigabit ports it is allowed.

 

[Nick2253]

Do you think the default setting from freenas is to block ping on mellanox interface?

I wouldn't think so. However, I've done that before with troubleshooting network equipment, so I thought I'd ask.

At this point, it sounds like you're in need of a way to test that the cards and cable are good. At the very least, if you could get the card installed on your desktop, you'd be able to test using Wireshark or similar tools to see if maybe you're only getting one way traffic. If you can source another cable locally, that would be a good check.