Mellanox Connectx-2 with FreeNAS 11

Status
Not open for further replies.
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
I just got in a Mellanox Connectx-2 card and put it into my FreeNAS server.
I have been unsuccessful at getting it to work.
FreeNAS is able to see the card and i am able to see it in the command line network configuration of FreeNAS.
When I set it as the default nic and give it a static ip address it does not come through.
any ideas.
 

Attachments

  • Capture.JPG
    Capture.JPG
    31.1 KB · Views: 1,571
Last edited by a moderator:
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
dlavigne said:
Post the output of ifconfig within code tags.
Click to expand...
Code:
bce0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500																	
		options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>						
		ether e4:1f:13:b7:71:3c																									
		nd6 options=9<PERFORMNUD,IFDISABLED>																						
		media: Ethernet autoselect																								
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500												
		options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>									
		ether e4:1f:13:b7:71:3e																									
		inet 10.1.10.20 netmask 0xffffff00 broadcast 10.1.10.255																	
		nd6 options=9<PERFORMNUD,IFDISABLED>																						
		media: Ethernet autoselect (1000baseT <full-duplex>)																		
		status: active																											
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384																  
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>																	  
		inet6 ::1 prefixlen 128																									
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3																				
		inet 127.0.0.1 netmask 0xff000000																						  
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>																				  
		groups: lo																												
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500														
		options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,
RXCSUM_IPV6,TXCSUM_IPV6>																											
		ether e4:1d:2d:68:ff:3e																									
		nd6 options=9<PERFORMNUD,IFDISABLED>																						
		media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)													  
		status: active																											
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500																	
		ether e6:1f:13:aa:71:3f																									
		nd6 options=9<PERFORMNUD,IFDISABLED>																						
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500													  
		ether 02:8a:2f:ed:33:00																									
		nd6 options=1<PERFORMNUD>																								  
		groups: bridge																											
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15																
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200																	
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0																	
		member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>															  
				ifmaxaddr 0 port 7 priority 128 path cost 2000																	
		member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>																
				ifmaxaddr 0 port 2 priority 128 path cost 20000																	
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500											  
		options=8<VLAN_MTU>																										
		ether 02:ff:20:00:07:0a																									
		nd6 options=1<PERFORMNUD>																								  
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)																		
		status: active																											
		groups: epair
 
Last edited by a moderator:
DrKK

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
I'm not sure what motherboard you're using, but I can tell since it has a BCE0 on it, none of us will approve. :)

Anyway:

Looks like bce1 is apparently active, and connected. With something plugged in. Looks like mlxen0 is plugged in, and connected, but not active (NB: I am using "active" in a more colloquial sense, not the way it is meant in "ifconfig"). I suggest you physically go to this console (which of course, this is not a console, this is some non-server rig), but whatever, physically go to the box, with local keyboard and monitor, and DISABLE the bce1 in the console menu. *THEN* enable the mellanox as the default NIC with static IP. 98% sure this is your problem.

You see, I don't think if the system is already configured to use NIC X, then you can just enable NIC Y and it will cycle off NIC X and go to NIC Y, which is what I think you are assuming. You must first explicitly release NIC X, which you cannot do unless you are physically at the computer (you can't release the NIC you're remotely using to talk to the server, right?).
 
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
DrKK said:
I'm not sure what motherboard you're using, but I can tell since it has a BCE0 on it, none of us will approve. :)

Anyway:

Looks like bce1 is apparently active, and connected. With something plugged in. Looks like mlxen0 is plugged in, and connected, but not active (NB: I am using "active" in a more colloquial sense, not the way it is meant in "ifconfig"). I suggest you physically go to this console (which of course, this is not a console, this is some non-server rig), but whatever, physically go to the box, with local keyboard and monitor, and DISABLE the bce1 in the console menu. *THEN* enable the mellanox as the default NIC with static IP. 98% sure this is your problem.

You see, I don't think if the system is already configured to use NIC X, then you can just enable NIC Y and it will cycle off NIC X and go to NIC Y, which is what I think you are assuming. You must first explicitly release NIC X, which you cannot do unless you are physically at the computer (you can't release the NIC you're remotely using to talk to the server, right?).
Click to expand...

The server is a IBM x3650 m3 with dual 16 core xeons 48Gb ram and 16 900Gb WD XE gen 2 sas drives. The switch i am using is a ubiquiti US‑16‑XG.
I went into the IBM IMM and started a KVM client and rest the bce1 from the menu and set up the mellanox with a static..
if i try to set up the mellanox with dhcp it does not get an ip address either.

what should i do in this menu.
Thanks
 

Attachments

  • Capture.JPG
    Capture.JPG
    126.1 KB · Views: 1,687
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
at this point i have tried multiple card in different pcie ports... the server sees the card and i am able to give it a static ip and configure it.. i have noticed on the switch it sees that something is plugged in put does not see any data. maybe a driver issue. maybe need to compile a driver for the card?
 
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
Problem solved. Turnes out one of the 10gb DAC cables is bad. Swapped out and working fine
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
I just got in a Mellanox Connectx-2 card and put it into my FreeNAS server which is seen by the Freenas server.
I have been unsuccessful to a connection on the 10gbe.
I can ping my windows machine in a peer to peer configuration and vice versa.
The reason I got the Mellanox 10gbe adapters was after watching a Youtube video. (https://youtu.be/8f5cuzcNrLo)
Admittedly the video shows the adapter working Freenas Corral.
I went the HCL and to my horror, my current NIC isn't listed.
IPs used are 192.168.240 for the onboard 1Gbe NIC and 192.168.200.1 both on subnet 255.255.255.0
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
Code:
FreeBSD 11.0-STABLE (FreeNAS.amd64) #0 r313908+d7d07647f69(freenas/11.0-stable): Thu Jul 20 19:01:05 UTC 2017

  FreeNAS (c) 2009-2017, The FreeNAS Development Team
  All rights reserved.
  FreeNAS is released under the modified BSD license.

  For more information, documentation, help or support, go here:
  http://freenas.org
Welcome to FreeNAS
root@freenas:~ # ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
  ether 0c:c4:7a:6b:60:7c
  inet 192.168.1.240 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 0c:c4:7a:6b:60:7d
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect
  status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  groups: lo
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 00:02:c9:52:b8:50
  inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
  status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
  ether 02:96:bb:d5:4b:00
  nd6 options=9<PERFORMNUD,IFDISABLED>
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 11 priority 128 path cost 2000
  member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 10 priority 128 path cost 2000
  member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 9 priority 128 path cost 2000
  member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 8 priority 128 path cost 2000
  member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 7 priority 128 path cost 2000
  member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 6 priority 128 path cost 2000
  member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:06:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:07:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:08:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:09:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:0a:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
epair5a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=8<VLAN_MTU>
  ether 02:ff:e0:00:0b:0a
  nd6 options=1<PERFORMNUD>
  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
  status: active
  groups: epair
root@freenas:~ #
 
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
Oldtimer said:
I just got in a Mellanox Connectx-2 card and put it into my FreeNAS server which is seen by the Freenas server.
I have been unsuccessful to a connection on the 10gbe.
I can ping my windows machine in a peer to peer configuration and vice versa.
The reason I got the Mellanox 10gbe adapters was after watching a Youtube video. (https://youtu.be/8f5cuzcNrLo)
Admittedly the video shows the adapter working Freenas Corral.
I went the HCL and to my horror, my current NIC isn't listed.
IPs used are 192.168.240 for the onboard 1Gbe NIC and 192.168.200.1 both on subnet 255.255.255.0
Click to expand...
what switch and sfp+ transmitters are you using?
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
ibmguy said:
what switch and sfp+ transmitters are you using?
Click to expand...
No Switch ibmguy.
This is on a peer to peer setup. using ip address 192.168.200.1 on the NAS and ip address 192.168.200.2 on the pc,
using a 3M SFP+ cable as the link.
I have done so many changes I am at a total lost regarding this setup.
What are your steps in assiging 10gbe adapter to the nic in which I can transfer files through?
 
I

ibmguy

Dabbler
Joined
Jul 6, 2017
Messages
20
Oldtimer said:
No Switch ibmguy.
This is on a peer to peer setup. using ip address 192.168.200.1 on the NAS and ip address 192.168.200.2 on the pc,
using a 3M SFP+ cable as the link.
I have done so many changes I am at a total lost regarding this setup.
What are your steps in assiging 10gbe adapter to the nic in which I can transfer files through?
Click to expand...
Unfortunately i am not failure with the peer to peer set up with the connect x2...
I had much luck using the ubiquiti US‑16‑XG... besides my cable being bad.

are you using sfp+ dac or sfp+ fiber transmitters?
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
I am using sfp+ dac.
Your setup makes me drool
I am trying to locate a pair of Chelsio adapters. My search for the answers has been futile thus far.
 
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Oldtimer said:
I am using sfp+ dac.
Your setup makes me drool
I am trying to locate a pair of Chelsio adapters. My search for the answers has been futile thus far.
Click to expand...
You said you can successfully ping the FreeNAS server IP address from the PC and vice versa... have you bound the FreeNAS SMB service to the 10G IP address? It may only be bound to your 1G connection.
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
Spearfoot said:
You said you can successfully ping the FreeNAS server IP address from the PC and vice versa... have you bound the FreeNAS SMB service to the 10G IP address? It may only be bound to your 1G connection.
Click to expand...
Firstly I am transfering data over the peer-to peer link.
My transfers speeds are slightly over the 1G rate.
The C drive a Corsair SSD. 240Gb, the other two CDM images are the Freenas shares 1Gb(X:/) and 10Gb.(Y:/)
My main pc is so mediocre. CPU FX6300,
Asus M5A97 R2.0 Motherboard (AMD 970/SB950, 4x DDR3, S-ATA 600, ATX, PCI-Express 2.0, USB 3.0, USB BIOS Flashback,
8Gb of Corsair Ram and older Sata 2 drives.
 

Attachments

  • CDM2.png
    CDM2.png
    241.3 KB · Views: 1,284
Last edited:
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Oldtimer said:
Firstly I am transfering data over the peer-to peer link.
My transfers speeds are slightly over the 1G rate.
The C drive a Corsair SSD. 240Gb, the other two CDM images are the Freenas shares 1Gb(X:/) and 10Gb.(Y:/)
My main pc is so mediocre. CPU FX6300, MB Asus m97 R2.0 8Gb of Corsair Ram and older Sata 2 drives.
Click to expand...
So, it's working and you no longer have a problem?
 
O

Oldtimer

Dabbler
Joined
Feb 19, 2015
Messages
10
Status
Not open for further replies.

Similar threads

Neek
  • Locked
Mellanox ConnectX-2 IB not coming up in FreeNAS 11
Replies
5
Views
5K
JustinClift
JustinClift
P
  • Locked
Mellanox ConnectX-2
Replies
4
Views
2K
Johnnie Black
Johnnie Black
JohnDonaldson
Mellanox MNPA19-XTR not detected after reboot...
Replies
5
Views
4K
dougs
D
M
SOLVED 10Gbe performance issue in FreeNAS 11
Replies
11
Views
11K
msbxa
M
R
SOLVED Mellanox Connectx2 does not connect at startup FreeNAS-11.3-U1 needs a command
Replies
5
Views
2K
RacerX
R
Top