10GbE Setup Help

[Fastline]

Hello Guys,

I've been using the AQC107 Ethernet Adapter but the connection saturates. Everything has been done prior to switching to SFP+ for better speeds. I'm having Mikrotik CRS312-4C+8XG-RM and using Base-T for now but seriously, I'm not happy. I've been through the Network Primer post before. But I'm still new to the SFP+ setup. From what I understand, I need to choose the proper Fiber Optic and the SFP+ Module and the switch does not have a Vendor lock. Now the question is how do I confirm, that the SFP+ Module and switch have a Vendor lock or not? Moreover, what SFP+ should I choose? My budget is $50-100 for the Card and $50-100 for the SFP+ Module and the Fiber Optic.

I'm looking to buy Chelsio T520-SO-CR or Chelsio T520-CR. Not sure, if these two are the latest model from Chelsio for 10GbE. Any confirmation regarding this?

My Current Configuration:

Motherboard: GIGABYTE B460M D3H
CPU: Intel Core i5 10400
RAM: Corsair 4x16GB DDR4 3200MHz
Storage: Seagate 6x16TB EXOS (SATA 6Gb/s onboard), RAID-Z1
Network: ASUS 10GbE Base-T and the same adapter on the client computer (Connected via CAT6 Cable)
Switch: MikroTik Cloud Router Switch (CRS312-4C+8XG-RM)
PSU: Antec 850W Gold

In short, please let me know about the following:

1. Switch - if it's good enough for the 10GbE loads, without saturation/drop-offs.
2. SFP+ NIC - Recommendations. Review on Chelsio T520-SO-CR or Chelsio T520-CR.
3. SFP+ Module - What Module should i buy?
4. Fiber Optic Cable - What kind of Fiber Optic do I need to buy?

 

[Etorix]

1. It should be.
2. "Server Offload" means that the NIC offsets some work to the CPU. So if you have a choice, the non-SO cards are better.
3. For short lengths, a DAC will do, for less money.
4. Otherwise, the primer recommends OM3/OM4 patch cable with duplex LC. And then matching SR (Short Range) modules.

 

[danb35]

3. SFP+ Module - What Module should i buy?

I like the compatible optics from fs.com. Inexpensive and they work well, just tell them what brand of device you want them compatible with. As Etorix says, you want SR optics unless you need hundreds of meters of distance.

4. Fiber Optic Cable - What kind of Fiber Optic do I need to buy?

Again from fs.com, I like the uniboot cables like this:

Customized LC-LC OM4 Uniboot BIF Fiber Patch Cable - FS.com

Buy customized LC UPC to LC UPC Duplex 2.0mm OM4 Multimode Uniboot BIF Fiber Patch Cable from reliable fiber patch cable supplier-FS.COM!

www.fs.com

Small bend radius, small (2 mm, IIRC) diameter, relatively robust. A little pricier than a typical duplex cable, but more convenient to use.

 

[Fastline]

1. It should be.

Good to hear!

2. "Server Offload" means that the NIC offsets some work to the CPU. So if you have a choice, the non-SO cards are better.

Oh, wow. I wasn't aware of it at all. Thanks a bunch for letting me know.

3. For short lengths, a DAC will do, for less money.

I get it. But what about the Vendor lock thingy?

4. Otherwise, the primer recommends OM3/OM4 patch cable with duplex LC. And then matching SR (Short Range) modules.

Okay, what about the pricing and the Vendor lock thingy?

 

[Fastline]

I like the compatible optics from fs.com. Inexpensive and they work well, just tell them what brand of device you want them compatible with. As Etorix says, you want SR optics unless you need hundreds of meters of distance.

I would say like 5 meters to my other computer.

 

[ChrisRJ]

I've been using the AQC107 Ethernet Adapter but the connection saturates.

What do you mean with this?

Storage: Seagate 6x16TB EXOS (SATA 6Gb/s onboard), RAID-Z1

RAIDZ1 is discouraged for drives of that size.

Network: ASUS 10GbE Base-T and the same adapter on the client computer (Connected via CAT6 Cable)

This NIC is not well supported by FreeBSD

 

[Fastline]

What do you mean with this?

That's the Ethernet Chipset.

RAIDZ1 is discouraged for drives of that size.

Oh, i see. Then does it mean RAIDZ1 should not be used? Or should not be used with larger capacity?

This NIC is not well supported by FreeBSD

I think so. That's why planning to switch to SFP+

 

[danb35]

But what about the Vendor lock thingy?

DACs often work, but that's why I usually use compatible optics and a fiber patch cable.

 

[somethingweird]

Oh, i see. Then does it mean RAIDZ1 should not be used? Or should not be used with larger capacity?

Both! if a drive fails (RaidZ1) and your rebuilding (larger drives take longer to rebuild & stress the other drives) and another one fails, your toast!

Of course you make sure you have backups :)

 

[ChrisRJ]

That's the Ethernet Chipset.

You probably meant the model of the NIC, right?

But that was not what I meant. What I do not understand is you point of saturation. What behavior did you observe that made you write this? How did you trigger this behavior? How was this behavior different from what you had expected?

 

[Fastline]

DACs often work, but that's why I usually use compatible optics and a fiber patch cable.

Gotcha

Both! if a drive fails (RaidZ1) and your rebuilding (larger drives take longer to rebuild & stress the other drives) and another one fails, your toast!

Of course you make sure you have backups :)

Makes sense

You probably meant the model of the NIC, right?

Yes, correct, the Ethernet chipset.

But that was not what I meant. What I do not understand is you point of saturation. What behavior did you observe that made you write this? How did you trigger this behavior? How was this behavior different from what you had expected?

This is actually my first NAS, having 5x16TB X16 EXOS Models and my max speed is like 250MB/s instead of 1GB/s. I'm not sure if it is the less number of vdevs or the NIC itself.

 

[Fastline]

Hello @danb35 @Etorix

So, I decided to buy the Chelsio T520-CR. Now, what options do I have for the connectivity?

As far as I understand, I can go for any of these:

1. DAC (usually for short length). 1M, 3M, 5M, and 7M.
2. Optic (short length)
- SR SFP+ Optic Module (SM10G-SR)
- SR SFP+ Fiber Optics Multi Mode Cable
3. Optic (long length)
- LR SFP+ Optic Module (SM10G-LR)
- LR SFP+ Fiber Optics Multi Mode Cable '

Is that correct?

Now, I'm quite confused about the color code. From my understanding, the Auqa with a grey hook is the common one and is probably laser-optimized. Is that correct?

Lastly, for the Mikrotik CSS610 8G 2S Switch, does it have a vendor lock? For example, SM10G-SR is definitely compatible with the NIC, but is it compatible with my Switch?

Although, it is always recommended to use the Optic Module of the same vendor/model at both endpoints, having two different modules from two different manufacturers can be vendor locked - is this only applicable for DAC or the SFP+ Fiber setup too?

 

[Patrick M. Hausen]

I would get vendor matched SFP+ optics from fs.com.

 

[danb35]

I agree with Patrick, though used optics from eBay might save a few bucks. I don't believe Mikrotik has a vendor lock, but fs.com can provide them too so it doesn't much matter AFAICT. Short-range optics are fine unless you need over 300 meters or so in cable length, and OM3 or OM4 fiber would be fine. But I see I've already said all this up-thread.

 

[Fastline]

I agree with Patrick, though used optics from eBay might save a few bucks. I don't believe Mikrotik has a vendor lock, but fs.com can provide them too so it doesn't much matter AFAICT. Short-range optics are fine unless you need over 300 meters or so in cable length, and OM3 or OM4 fiber would be fine. But I see I've already said all this up-thread.

Yes, will update this thread on how it goes once I get the parts. BTW, is there any way to check the vendor lock?

 

[Fastline]

Guys, which one is best out of these three?

https://www.fs.com/products/69091.html

https://www.fs.com/products/40180.html

https://www.fs.com/products/141727.html

And are these compatible with SM10G-SR?

Moreover, is this card a counterfeit?

Chelsio T520 CR T520-CR 10GbE 2-Port PCIe Unified Wire Adapter Card 110-1160-50 | eBay

1x T520-CR Adapter Card. We keep our promises that we will provide you with the best solution within 24 hours. Sorry for the inconvenience.

www.ebay.com

The design is different than on the official Chelsio site

https://www.chelsio.com/wp-content/uploads/2013/10/T520-CR-PB.pdf

 

[Etorix]

1. DAC (usually for short length). 1M, 3M, 5M, and 7M.

At 5 m, fibre may make sense over a DAC.

2. Optic (short length)
- SR SFP+ Optic Module (SM10G-SR)
- SR SFP+ Fiber Optics Multi Mode Cable

Unless you get a deal-one-cannot-refuse on LR optics, SR is most appropriate.
Multi-mode modules are cheaper; single-mode is primarily intended for long distances where multi-mode does not work.
For 10 or 25 GbE, multi-mode OM3/OM4 fibre with LC connectors, and the matching optics, is the easy go-to choice.

Now, I'm quite confused about the color code.

Guide to Indoor Fiber Optic Cable Color Coding

If fiber optic cable coding makes your head spin, we have a guide to the common colors and some suggestions for STH readers on what to use

www.servethehome.com

The FAQ section of fs.com is also worth browsing.

Although, it is always recommended to use the Optic Module of the same vendor/model at both endpoints, having two different modules from two different manufacturers can be vendor locked - is this only applicable for DAC or the SFP+ Fiber setup too?

Match module with vendor at each endpoint. It doesn't matter is each end is a different vendor—that's the benefit of modules over a fixed DAC. If fs.com does not have specific modules for your vendor go for 'Generic'.

 

[Fastline]

At 5 m, fibre may make sense over a DAC.

Going with Fiber!

Unless you get a deal-one-cannot-refuse on LR optics, SR is most appropriate.

Understood!

Multi-mode modules are cheaper; single-mode is primarily intended for long distances where multi-mode does not work.
For 10 or 25 GbE, multi-mode OM3/OM4 fibre with LC connectors, and the matching optics, is the easy go-to choice.

Yes, going with OM4. OM5 is also available at fs.com. Should i go with OM5 then? Would it be compatible? Moreover, will it have any advantage in my workflow?

Match module with vendor at each endpoint. It doesn't matter is each end is a different vendor—that's the benefit of modules over a fixed DAC. If fs.com does not have specific modules for your vendor go for 'Generic'.

Gotcha

Thank you so much for clearing it all up!

 

[Etorix]

Moreover, is this card a counterfeit?

The design is different than on the official Chelsio site

I've not yet heard of counterfeit Chelsio cards. The design matches those I have; the card in the leaflet must be a different model.

 

[danb35]

https://www.fs.com/products/69091.html https://www.fs.com/products/40180.html https://www.fs.com/products/141727.html

I wouldn't go with any of these; I'd instead use the Uniboot OM4 I linked up-thread. I'm not aware of any reason OM5 wouldn't work, but I also don't know that it would.