10Gb is getting more confusing...

[SmoothRunnings]

After looking through the threads her on the topics of 10Gb NICs for FreeNAS the choice of what I should use keeps getting increasingly more confusing.

I am in the process of replacing my IBM DS3524 iSCI SAN with FreeNAS that will be running on a Dell R510, 2 x 6C CPUs, 64GB of RAM, 14 x 500GB SATA 7200RPM drives. My IBM SAN only provides space to my two ESXi 6.5 servers I don't use it for anything else. I also currently use about 2TB of the 8TB on the IBM of drive space. So I would like to get 10Gb NICs for it, one two port full height, and two 2 port half height cards. I need something faster than 8 x 1GB ports on my IBM (not sure if it really uses all of them).

I am open to suggestions. I would like to get all the parts together before I start the next part of my project.

Thanks!

PS. where it's confusing is that everyone has a different version and opinion of what card to use...

 

[c32767a]

After looking through the threads her on the topics of 10Gb NICs for FreeNAS the choice of what I should use keeps getting increasingly more confusing.

PS. where it's confusing is that everyone has a different version and opinion of what card to use...

Right out of the gate, you can use the same cards iXsystems ships in the commercial version of FreeNAS.. that's probably your best choice.

After that, like any other open DIY project, you can use pretty much anything that has a driver.. Cost usually drives people to try other options than Chelsio. I think if you read all the posts, there are cards that are known to work and there are cards with reputations of problems in general, or cards where a particular driver version had a bug for a while (hi Intel) and perceptions depend on who was affected by the bug and how badly.

FWIW, my preferred cards these days are Intel and Chelsio. In the past we've used Solarflare. Pick one you like and test,test,test.

 

[SmoothRunnings]

Cost is always a factor. LOL

Does FreeNAS support the infiniband cards? I have seen posts of people asking this and very little information afterwards. So I don't know if people have given up or have found FreeNAS does support it dropped updating the thread!?

As for the Intel cards, whats the status when it comes to these cards, the 520 when they are coming from China, I see that people talk about the Yotta sticker but is the general consensus still that if they don't have the yotta sticker they are fake?

I noticed that these cards call in all sorts of flavors, the earlier ones are cheaper than the newer tech, but am I really going to noticed a big difference between the Intel 520 and 540 cards? Or the cheaper $75 card vs the $400 card?

Also what is this card that ships with iXsystems FreeNAS? I checked their website, I can build a NAS with a 10Gb card but it doesn't say what brand of card they use in the build options.

Thanks,

 

[BigDave]

I see that people talk about the Yotta sticker but is the general consensus still that if they don't have the yotta sticker they are fake?

When buying Intel cards from eBay, I only buy from vendors that display the YottaMark sticker photos in the aution. You need to verify the number using
this link.
http://verify.yottamark.com/
Make sure and communicate with the seller, that the picture is of the actual card being sold. If you receive it and it's not genuine, eBay will help you get
your money back

 

[c32767a]

Cost is always a factor. LOL

Does FreeNAS support the infiniband cards? I have seen posts of people asking this and very little information afterwards. So I don't know if people have given up or have found FreeNAS does support it dropped updating the thread!?

As for the Intel cards, whats the status when it comes to these cards, the 520 when they are coming from China, I see that people talk about the Yotta sticker but is the general consensus still that if they don't have the yotta sticker they are fake?

I noticed that these cards call in all sorts of flavors, the earlier ones are cheaper than the newer tech, but am I really going to noticed a big difference between the Intel 520 and 540 cards? Or the cheaper $75 card vs the $400 card?

Also what is this card that ships with iXsystems FreeNAS? I checked their website, I can build a NAS with a 10Gb card but it doesn't say what brand of card they use in the build options.

Thanks,

My perspective is maybe a little different because at $dayjob, we buy our hardware from a supply chain we trust. If we get a bad card, it goes back to the vendor no questions asked and is replaced. I can't comment on buying through ebay, etc.. :)
(my personal home NAS has X520s in it that I got from Amazon and they work fine, fwiw)

I don't know that iX ever posted the hardware BOM for one of their TrueNAS boxes, but they are a partner of Chelsio. So it seems likely if you open the lid, you'd find some variant of the T4xx or T5xx card. Plus, various people who probably do know have posted comments in threads that imply this.. ;)

My understanding is that Infiniband support is not in the FreeNAS kernel. I know a couple posters have made noises about inserting kernel modules to support it, but I don't know that I've ever heard of success. Given that Inifiniband is somewhat of a niche networking technology, I don't know that I would want to have a hack running on a NAS where uptime is important.

We used to use the X520s, now we use X540s.. They work well enough for us since the driver issues were fixed. We haven't had any issues that have required us to spend a lot of time analyzing them. They pretty much just work with the current driver.

 

[SmoothRunnings]

We used to use the X520s, now we use X540s.. They work well enough for us since the driver issues were fixed. We haven't had any issues that have required us to spend a lot of time analyzing them. They pretty much just work with the current driver.

If I was to buy a X520-DA2 for my FreeNAS machine should I get it with the 10Gb (copper? they are tick black) cables or should I buy the transceivers and get some fiber? After looking on ebay I might source a X520-DA2 there as well as X520-DA1's with the low profile back plate for my ESXi servers.

Thanks!

 

[SmoothRunnings]

When buying Intel cards from eBay, I only buy from vendors that display the YottaMark sticker photos in the aution.

What's the risk if the card doesn't have a YottaMark sticker on it? Does it mean it's fake, or a refurbished card? I haven't seen anything that says what people have found with cards that didn't have the sticker.

Thanks,

 

[c32767a]

If I was to buy a X520-DA2 for my FreeNAS machine should I get it with the 10Gb (copper? they are tick black) cables or should I buy the transceivers and get some fiber? After looking on ebay I might source a X520-DA2 there as well as X520-DA1's with the low profile back plate for my ESXi servers.

Thanks!

So, There's RJ-45 Copper, and SFP+ based (usually called Direct Attach, "DAC" or "Twinax") Copper. RJ-45 used to be uncommon in datacenters.. It is still uncommon in datacenters I frequent. (I see many more server things with 10GbaseT ports on them for sale, so maybe that's changing.) DAC cables are very common in the DCs I'm in, $dayjob uses it, and probably 95% of the less than 5 meter runs I see in DCs are copper DAC cables.

The only time Fiber appears is when you have to go more than 5 meters or a switch or NIC requires a specific vendor SFP+ module and won't link with a copper DAC cable.

My short answer is: Use DAC/Twinax. Get the cable that's labeled as working with your switch vendor. $dayjob uses Cisco and DellForce10 switches and DAC cables labeled to work with those switches from a 3rd party like fiberstore.com and things work fine. My personal NAS' X520 was connected to a Cisco 2960X 10G port for a while and is now connected to a Ubiquti edgeswitch 16XG with 10Gtek cables from Amazon. I've also direct connected my X520 to a SolarFlare card and Chelsio T300 card in another server with no problems.

 

[SmoothRunnings]

So, There's RJ-45 Copper, and SFP+ based (usually called Direct Attach, "DAC" or "Twinax") Copper. RJ-45 used to be uncommon in datacenters.. It is still uncommon in datacenters I frequent. (I see many more server things with 10GbaseT ports on them for sale, so maybe that's changing.) DAC cables are very common in the DCs I'm in, $dayjob uses it, and probably 95% of the less than 5 meter runs I see in DCs are copper DAC cables.

The only time Fiber appears is when you have to go more than 5 meters or a switch or NIC requires a specific vendor SFP+ module and won't link with a copper DAC cable.

My short answer is: Use DAC/Twinax. Get the cable that's labeled as working with your switch vendor. $dayjob uses Cisco and DellForce10 switches and DAC cables labeled to work with those switches from a 3rd party like fiberstore.com and things work fine. My personal NAS' X520 was connected to a Cisco 2960X 10G port for a while and is now connected to a Ubiquti edgeswitch 16XG with 10Gtek cables from Amazon. I've also direct connected my X520 to a SolarFlare card and Chelsio T300 card in another server with no problems.

This is what meant in my post as an example. Not sure if the cables here are copper or fiber or what? Is there any benefit to using these cables over fiber or vice versa or will I get the same result in the end?

https://www.ebay.com/itm/INTEL-X520...e=STRK:MEBIDX:IT&_trksid=p2060353.m1438.l2649

 

[c32767a]

This is what meant in my post as an example. Not sure if the cables here are copper or fiber or what? Is there any benefit to using these cables over fiber or vice versa or will I get the same result in the end?

https://www.ebay.com/itm/INTEL-X520-DA2-E10G42BTDA-Dual-Port-Ethernet-Server-Adapter-W-2-3M-10G-Cable/252653751859?ssPageName=STRK:MEBIDX:IT&_trksid=p2060353.m1438.l2649

The card in that link is an SFP+ card and those 2 cables would be considered DAC aka Twinax.
If I was shopping for 10G for your use case, if the cable length was OK, that's what I'd get.

It used to be that 10GbaseT was a power hog. Also, Twinax is limited to about 7 meters. There are fiber variants that can go 70Km.

But they all move 10Gb/s of data from one end to the other.. :)

 

[SmoothRunnings]

The card in that link is an SFP+ card and those 2 cables would be considered DAC aka Twinax.
If I was shopping for 10G for your use case, if the cable length was OK, that's what I'd get.

It used to be that 10GbaseT was a power hog. Also, Twinax is limited to about 7 meters. There are fiber variants that can go 70Km.

But they all move 10Gb/s of data from one end to the other.. :)

I only plan to stick the FreeNAS into the same rack as the two ESXi. I don't think I'll ever need 7 meters of cable. lol :P

I was thinking of getting one of those and two X520-DA1's for my ESXi hosts.

Thanks,