Do Mellanox ConnectX-2 work with FreeNAS 11?

[Supercookie]

I know the work with FreeBSD 10.1 and up and don't with FreeNAS 9.x but is FreeNAS 11.1 based on FreeBSD 10.1?

 

[Johnnie Black]

They work on FreeNAS 11.0, they also did on 9.10, didn't upgrade my main FreeNAS server to 11.1 yet but expect they will still work.

 

[jgreco]

They "work", but maybe not as well as the Chelsio cards, which are hands down first choice for FreeNAS.

 

[wblock]

FreeNAS 11 is based on FreeBSD 11.

 

[Supercookie]

They "work", but maybe not as well as the Chelsio cards, which are hands down first choice for FreeNAS.

what do you mean "work"?

 

[jgreco]

There's lots of things that "work", which means, they probably work, most of the time, possibly even all of the time, and your experience with and willingness to be a guinea pig with a given hardware item that has seen nowhere near the extensive mileage of some other more proven solution is of interest.

I realize that the cost differential between a $12 Mellanox ConnectX-2 and a $325 Chelsio T520-CR on eBay makes them very attractive for hobbyist and home users, and I certainly don't mind that, just try to remember that you're getting a low-end adapter with relatively immature support. It's perfectly possible it'll be just dandy. It's also perfectly possible it'll not be. I just thought it worth pointing that out. "work" != WORK.

 

[Supercookie]

There's lots of things that "work", which means, they probably work, most of the time, possibly even all of the time, and your experience with and willingness to be a guinea pig with a given hardware item that has seen nowhere near the extensive mileage of some other more proven solution is of interest.

I realize that the cost differential between a $12 Mellanox ConnectX-2 and a $325 Chelsio T520-CR on eBay makes them very attractive for hobbyist and home users, and I certainly don't mind that, just try to remember that you're getting a low-end adapter with relatively immature support. It's perfectly possible it'll be just dandy. It's also perfectly possible it'll not be. I just thought it worth pointing that out. "work" != WORK.

i thought you meant that it would work at less than full capacity from experience.

 

[jgreco]

It probably does work at "less than full capacity". It's an early generation PCIe 2 card, which means it's less efficient and running at a slower speed. But the flip side of that is that even if were only to run at HALF the speed of a Chelsio, and I expect it'll be significantly better than that, it's running many times faster than gigE, so does it matter?

 

[nightshade00013]

I agree they may not be as fast as some of the other ones but for a lot of home users who want a faster connection it's a whole lot better than GbE.

The one thing you have to watch out for is there are some that are different than the others and will not work. Can't remember exactly what the difference is but I think one is a FiberChanel and one is FiberEthernet. The Fiber Ethernet ones work just fine as far as I can tell, yes I have a couple. I know that someone on the forum not long ago had the other and they are scrap bin material.

You can run a direct connection between a desktop and the FreeNAS fairly easily or grab a switch. I was actually able to bridge two ConnectX-2's with an intel ethernet NIC in a OpnSense router but it would not stay stable long term and eventually there was a problem where the free memory would start dropping and it would crash the router. The direct connection just requires you to set a static IP address on both and remember to use that IP to connect if using SMB or you will drop to GbE speeds. Windows 7 however does not seem to play nice with the latest Mellanox drivers I ended up using the 4.6 version and that installed and has worked fine however. Use in a desktop also requires a system with a lot of airflow or adding in a fan to cool the NIC.

Transferring from the FreeNAS to my Desktop has been a ton faster especially when reading from ARC to a RAM drive. Right now I am getting about 300 MB/s average with peaks to around 330MB/s using TeraCopy. That shows to be around 25% of the adapter's max speed. Using the windows built in copy feature it goes up to around 450MB/s or around 35% of the max speed. So is it better than an old GbE, absolutely and for the cost I am more than happy to get some more hopefully to soon add a switch to the mix soon.

 

[Johnnie Black]

This is what I get with mine:

Pretty happy with it, especially considering the cost.

 

[tvsjr]

$325 Chelsio T520-CR on eBay

It's even better than that. I just picked up three fully-functional T520-CRs for a total of $307.15 shipped, or $102.38/card. Plugged them in and they Just Work(tm). Even got Chelsio to sell me brackets!

 

[jgreco]

Ok well I'm not keeping up with eBay pricing, but fact remains that $100/card is going to be a show stopper for some people. Has me remembering the days when 10 *MEGA*bit cards were lots more than that. Still, I think it is worth it for the problem-free happiness of "it just works."

 

[Johnnie Black]

I forgot to mention, I do have an "issue" with these NICs and FreeNAS, if I enable jumbo frames I get a little better performance but also start getting some timeouts during transfers, mostly if there is traffic both ways, I don't get these with the same NIC in a linux server, so this is maybe a consequence of being less supported like @jgreco mentioned, still happy, and for my home use can't really justify the Chelsio NICs cost.

 

[nightshade00013]

I had an issue where I believe the cards were defaulting to an odd MTU. Windows wanted to run something like 1524 but FreeNAS was 1500. Not sure what the router went to but it crashed more when everything was bridged together than after I set it to 1500. Initially it would only be stable for about 30 minutes of traffic. After it was good for almost two days.

No plan to run higher than that for a speed boost anyway. Jumbo frames would make it run faster but I am happy with what I have going for now and since I will be eventually using the NIC for internet access as well as LAN 1500 will work fine.