Need some help with Infiniband connectivity

[cyberlink01]

Ill start by saying this is my home network so I can mod on the fly.

My FreeNas server is

• FreeNAS-11.2-U3
  • dual quad core E5405 CPU'S
  • 32g of ECC ram
  • 18Tb of drive space
  • Internal Intel Ethernet card
  • Mellanox MT26418 (40Gb)

The Switch is an 8 port Mellanox 40Gb switch

I have 4x 40Gb ports on my blade server plugged into the switch and configured with IPoIB with one server setup with OpenSM. Those servers are Centos7 and communicate over the IPoIB just fine. At this time I have the default Infiniband partition configured to 10Gb. It was at 40Gb but was reconfigured for testing.

When I boot the freenas server mlx4 gives an error I am trying to figure out.

Code:

mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 00:02:c9:5a:65:9a
mlx4_en: mlx4_core0: Port 1: Using 8 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 8 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
mlx4_core0: command INIT_PORT (0x9) failed: in_param=0x0, in_mod=0x1, op_mod=0x0, fw status = 0x8
mlx4_core0: Required capability exceeded device limits
mlx4_en: mlxen0: Failed Initializing port

It does appear to recognize the card.

Code:

mlx4_core0@pci0:3:0:0:    class=0x0c0600 card=0x002215b3 chip=0x673215b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26418 [ConnectX VPI PCIe 2.0 5GT/s - IB DDR / 10GigE]'
    class      = serial bus

I also noted that the card is configured for ether

Code:

sys.device.mlx4_core0.mlx4_port2: eth
sys.device.mlx4_core0.mlx4_port1: eth
sys.device.mlx4_core0.board_id: MT_0F90120008
sys.device.mlx4_core0.hca: MT26418
sys.device.mlx4_core0.fw_ver: 2.9.1810

Im assuming the issue is the entry that states "Required capability exceeded device limits" Im just not sure what compatibility it is referring to.

Any help would be appreciated.

 

[Mlovelace]

If you want the card to run in ib mode you need to set sys.device.mlx4_core0.mlx4_port to auto. sys.device.mlx4_core0.mlx4_port1: auto (ib)

Edit: I suspect the interfaces won't function in freeNAS in IB mode. IIRC the Mellanox interfaces didn't load unless they were configured for eth in previous forum posts.

 

[cyberlink01]

Mlovelace, Thank you for the response. I tried putting them in IB mode but get the response of

Code:

mlx4_core0: command SET_PORT (0xc) failed: in_param=0x72b801000, in_mod=0x1, op_mod=0x0, fw status = 0x40
mlx4_core0: command SET_PORT (0xc) failed: in_param=0x72b801000, in_mod=0x2, op_mod=0x0, fw status = 0x40

Setting them to auto does not appear to do anything to it other than say it is in auto

Code:

sys.device.mlx4_core0.mlx4_port1: eth -> auto (eth)

Switching the card from IB back to Eth initializes it but I get the same error as above.

 

[Mlovelace]

There isn't much that I could find about that error. You could see if the card's firmware is up-to-date. http://www.mellanox.com/page/firmware_download

 

[cyberlink01]

Yep, I have done the fireware check. lol

That is the reason I asked, Im not finding a lot on that error so was hoping someone may know something or maybe a developer may have something to say.

I started with a qlogic card but turns out it does not run under FreeNas or FreeBSD. Maybe Ill order a Mellanox ConnectX-3 and see if it does any better.

 

[Mlovelace]

Yep, I have done the fireware check. lol

That is the reason I asked, Im not finding a lot on that error so was hoping someone may know something or maybe a developer may have something to say.

I started with a qlogic card but turns out it does not run under FreeNas or FreeBSD. Maybe Ill order a Mellanox ConnectX-3 and see if it does any better.

The developers aren't on the forums for the most part. You can submit a bug report about the error if you want the devs to see it. Mellanox cards are not supported well in freeNAS. If you want solid 10Gbe get a chelsio card. IX ships the TrueNAS servers with the T520-SO-CR card, so you're going to get the best results with that. If you want 40Gbe the T580 is the 40Gbe version, though I've not seen anyone running it so I can't speak it it's performance.