Ok, so I just tested, and everything works with both cards in loopback on the FreeNAS box. I installed both cards in the FreeNAS box, cabled them together, and when it booted up, I got green "Physical Link" lights, and even yellow activity lights. So, apparently, the issue is not FreeNAS recognizing the Infiniband cards, nor failing to start up openSM. All that is apparently working.
I already tested the Windows box in loop back, and it worked too. So, that means that both cards work, the cable works, the Windows drivers are good, the FreeNAS drivers are good, and openSM is working. So, the problem has to be something wrong with Windows trying to talk to FreeBSD/FreeNAS. When I connect from FreeNAS to Windows, I can't even get the green "Physical Link" light. Everything acts as though it's unplugged [Windows] "Network cable unplugged", [FreeNAS] "No Carrier" / "Giving up" "Status Changed to DOWN".
You mentioned that your Windows driver was version 5.35.12970.0; mine is 5.35.12978.0, virtually identical, so I would wager that that is not the problem.
On my FreeNAS box, pressing 9 on the console menu to go into the shell, running:
Code:
root@freenas:~ # dmesg | grep Mellanox
shows that the driver version is:
Code:
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Mar 14 2018)
Could you please check what version your FreeNAS box is using?
As for the firmware:
Code:
Windows 10 > Device Manager > Network Adapters Mellanox ConnectX-2 IPoIB Adapter > [right click] > properties
Mellanox Properties Dialog > Information Tab
This shows my firmware version as 2.9.1200. If I recall correctly, the firmware version on the other card is 2.9.1000, just slightly lower. I would say that that could be a problem, except for the fact that both loopback configs work, so that would seem to indicate that the firmware combination is fine.
P.S. As I said, I'm a FreeNAS noob, but from that console menu that FreeNAS boots into, pressing 9 to get to the shell, that is a normal root terminal right? Or is there something special/restricted about it? Also, it doesn't say anywhere on those screens how to return from the shell back to the menu, but I noticed that typing the command "exit" works. Just want to make sure that that is the proper way, and I'm not accidentally doing a force close or something.
P.P.S. Forgive me if I'm being overly simplistic about explaining my steps, but (1) I want to make sure that I'm doing them correctly, and not making any stupid assumptions, and (2) I'm sure I won't be the last person to have this issue, and if someone else should find this post, I'm hoping they can use my troubleshooting as a guide, if we ultimately get this working.