SOLVED TNC13U2 MLX CX3Pro VLAN issues (kind of solved, at least working for now)

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Hey guys,

I know that MLX cards are not the most ideally suited for TNC but they are all I have fitting a x8 slot.
I want to setup a low(ish) power secondary NFS Box for ESXi so I revived an older SM X11SSL-F system which only has x8 and x4 slots. I added a HBA, an Optane and a CX3 Pro I had left over.
My problem is that the card works fine when running no VLANs, but I can't get it to pass traffic with VLANs configured.
I have used another TNC Box (12.8 with a CX5) as blueprint, they have identical config at the switch (access ports on VL10/11) and OS level (different IPs obviously).

The CX5 has been working fine forever on 12.x but the CX3Pro is acting up on 13. Everything seems to be jolly, ports are up but its just not working...

Will drop in a non-Pro next, but am kind of lost what the issue might be except some incompatibility issue ... which is why I thought i should ask if anything is known regarding this ;)

Thanks

Edit: I am not trying to set up multiple VLANs on a single interface (which obviously would need a trunk on the switch) but only access the physical NIC with a VLAN. Normally I wouldnt even need to tell the OS that the NIC is running a vlan since the switch sets it, but thats what i did on the working box so i did it here too. Tried all variants o/c(vlan interface on top, vlan interface on trunked port etc pp) , nothing

Edit2 - added config pics -tn13/ ip 31 is not working, fn11, ip29 is fine
 

Attachments

  • tn13_p1_details.JPG
    tn13_p1_details.JPG
    42.1 KB · Views: 76
  • fn11_p1_details.JPG
    fn11_p1_details.JPG
    45.3 KB · Views: 76
Last edited:

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Alright... realized that I had a physical x16 slot available (x8 electrical) so went ahead and tested some more

1- replaced the CX3Pro with a CX5... showed up in TN13, but could not link up.
2- reinstalled TN12 on the same box (with CX5) - link up at first try... ip, vlan, go. Working as expected
3- replaced the CX5 with the CX3Pro... acted up a bit, i think due to all interfaces having DHCP on automatically... sorted that out, ip, vlan, go. Working as expected.

So, conclusion - TN13 does not like Mellanox cards - in a sneaky, unfriendly way.
Boy am I glad that I have not updated my main box yet.
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Ok, things are not as simple it seems...

Finished configuring the 12.8 box network, shares etc... tried mounting from esxi - not working...
Tried ping locally again ... not working...
Took me while but it turns out, as long as the secondary interface of the card is configured (presumably with another vlan) it wont work with the CX3Pro... configure just one interface with vlan... all good.

will swap the CX5 back in.
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
So after i swapped it back in it didnt come up either... reinstalled 3 times, no change.

Then i saw that the switch says UP on these port, just the TNC (12) Gui does not....
Adding up to the option for the interface resolved that...
So, 12.8U1 is now working fine with the CX5 (25G card) with both ports configured...

Now upgrading to 13 again...
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Works too. Same config as before. I dont get it.
But I am not going to touch it any more, lets hope it continues working.
 
Top