10GbE NIC for Scale?

[indivision]

Can anyone recommend a specific 10GbE NIC card or controller I should look for to be compatible with TrueNAS Scale?

This is a great resource: https://www.truenas.com/community/resources/10-gig-networking-primer.42/

But, it appears to be focused on FreeBSD. I'm assuming there is some variation in Scale?

 

[dracozny]

Harder to find pure cat6 based cards if that is what you are asking. Typically any SFP+ compatible card from Intel or Matrox will get you going and then get the modules for Cat6 to plug in. I'm sure there are others that will work just fine.

 

[indivision]

Sorry. Yes. I should have clarified that. I'm wanting something for Cat6 as the house is already wired for that.

It looks like there are quite a few Cat6 based 10GbE cards on the market that say "Linux compatible". But, is there more I need to watch out for for TrueNAS Scale specifically?

As a random example: https://www.amazon.com/TP-Link-TX40...s=10gbe+nic+matrix+rj45&qid=1627712726&sr=8-8

The ones with Intel chipsets tend to be a bit more expensive. So, hoping there are some other options people have used successfully?

 

[dracozny]

Intel always is more expensive but they cater to enterprise level which is why you will always find support for their hardware in the kernel. with that said I don't think there really is any NIC not supported in the Linux Kernel unless the Devs did something unusual and culled unlikely used drivers out for their port from Deb 11

https://www.debian.org/releases/stretch/i386/ch02s01.html.en
2.1.6. Network Connectivity Hardware

Almost any network interface card (NIC) supported by the Linux kernel should also be supported by the installation system; drivers should normally be loaded automatically. This includes most PCI/PCI-Express cards as well as PCMCIA/Express Cards on laptops. Many older ISA cards are supported as well.


ISDN is supported, but not during the installation.

 

[danb35]

Keep in mind, "has a driver" does not mean "works well"--see all the issues with Realtek garbage. I'd expect Intel and Chelsio to be at least as robust under Linux as under FreeBSD (and they're both quite solid there).

 

[dracozny]

I'll have to look up all these problems with RealTek you refer to, the only I have had usually indicated a broader scope of problems with onboard NICs and cascading MOBO faults. with that said I think my biggest issue with the TP link products as a whole not necessarily the card the Op linked to in their last post has been overall lifespan of TP Link's products.

 

[indivision]

Thank you for this information! It kind of sounds like you have to just try different things out (and make sure you have a good return policy in place).

The next problem I'm seeing is that it sounds like the better switches are very loud (noisy fans). Since my wiring all leads back to an office, not sure how that will work...

 

[crkinard]

Just make sure to go Intel or don't bother.

 

[Constantin]

I wonder if Chelsio T520-BT and like cards are an option? Chelsio tends to be well-supported by ixSystems on the SFP+ side.

 

[Patrick M. Hausen]

The next problem I'm seeing is that it sounds like the better switches are very loud (noisy fans)

Are you planning to build an all 10G network or do you just want to connect the servers with 10G and "fan out" with 1G?

For the latter I can recommend the Cisco 2960-L series. WS-C2960L-24TQ-LL for example. 24x 1G, 4x SFP+, completely fanless.

 

[Constantin]

Smaller instances include the 4-port mikrotik CRS305 SFP+ switch, which is fanless also.

 

[melloa]

Smaller instances include the 4-port mikrotik CRS305 SFP+ switch, which is fanless also.

Solid performance and working here for over an year.

 

[indivision]

Are you planning to build an all 10G network or do you just want to connect the servers with 10G and "fan out" with 1G?

For the latter I can recommend the Cisco 2960-L series. WS-C2960L-24TQ-LL for example. 24x 1G, 4x SFP+, completely fanless.

Thank you. I'm just looking to have the server and 3 other machines on the network running 10G. Everything else can be 1G.

That looks like I nice router. But, more than I need. How do they pull off that many ports with no fans and no heat issues?

Smaller instances include the 4-port mikrotik CRS305 SFP+ switch, which is fanless also.

Thank you. I saw that one and calculated out a budget with it. But, wasn't sure how it would hold up over time. Saw a few ominous comments about it. But, might have just been internet noise...

Solid performance and working here for over an year.

Ok. Thank you. That's really good to know. Maybe I will go for this one after all.

Are you using fiber lines? Or modules for Cat6? If Cat6, did you pair the modules so that they were the same on both sides of the wire in every direction?

I had wanted to use direct Cat6 connection cards where I could and just put modules into the switch. But, then I read a comment that they don't work well unless you pair them.

 

[Patrick M. Hausen]

That looks like I nice router.

It's strictly layer 2, no routing.

 

[Constantin]

I use fiber only transceivers. Cheaper, cooler, less power draw. Twinax DAC is another option.

I’d stay away from 10GbE Copper with a SFP+ switch as much as possible since 10GbE copper SFP+ transceivers still run hot - mikrotik has specific guides how many can be used in each of their switches, it’s 1-2 for the CRS-305, IIRC. Other OEMs also have limits.

Transceivers do not have to paired with each other but rather they may need to be paired with/to the OEM of the equipment they are plugged into - ie a “Intel” transceiver into a “Intel” SFP+ PCIe card, and so on. This is called vendor locking. Not all OEMs try this sort of rent collection but some do.

with fiber, keep in mind to use the same type of transceiver (example: 850nm multi-mode) and the right fiber also (fiber type and connector, usually 850nm multi-mode LC for 10GbE SOHO networks)

 

[indivision]

I use fiber only transceivers. Cheaper, cooler, less power draw. Twinax DAC is another option.

I’d stay away from 10GbE Copper with a SFP+ switch as much as possible since 10GbE copper SFP+ transceivers still run hot - mikrotik has specific guides how many can be used in each of their switches, it’s 1-2 for the CRS-305, IIRC. Other OEMs also have limits.

Transceivers do not have to paired with each other but rather they may need to be paired with/to the OEM of the equipment they are plugged into - ie a “Intel” transceiver into a “Intel” SFP+ PCIe card, and so on. This is called vendor locking. Not all OEMs try this sort of rent collection but some do.

with fiber, keep in mind to use the same type of transceiver (example: 850nm multi-mode) and the right fiber also (fiber type and connector, usually 850nm multi-mode LC for 10GbE SOHO networks)

Thank you. Fiber isn't a likely option in this case because it would be a big over-haul to add it to the house (which already has Cat6 throughout). I thought I was being forward thinking by having the whole house wired. But, the electrician just fed the wires through drilled holes. I should have had him use conduit for every drop!

So, one step forward, two back. :)

Going to double-check to see if I can find what the copper limit is on the CRS-305...

 

[indivision]

Hm. If there is a copper limit due to heat on the CRS-305 I couldn't find it through their on-line documents. Would love to know for sure one way or another.

 

[Richj]

I'm using Intel Ethernet Converged Network Adapter X540-T2 and it seems to be working fine.

 

[Constantin]

Oh it’s there. I’ve looked it up before. I believe the way to get to it via Mikrotik is to select the copper 10GbE transceiver, then look up the compatible list, then read through the fine print. If I looked at it right, the answer is 2.

 

[crkinard]

I'm using Intel Ethernet Converged Network Adapter X540-T2 and it seems to be working fine.

X550-T2 there and working fine.
Intel® Ethernet Converged Network Adapter X550-T2