40GbE Switch Suggestion

[Fastline]

Hello guys,

I currently own Mikrotik CRS312-4C+8XG-RM that supports 10GbE Base-T and SFP+. As I've been building 4 NAS which will be at work, each having 40GbE network, I'm looking for a switch that supports 4x 40GbE ports or i can split out of 100GbE. Initially, my plan was to get CRS518-16XS-2XQ-RM but the 2x100GbE do not support 2x40GbE speed on a 100GbE port on this switch. I planned for Mikrotik as it does not involve vendor locking and is easy to use.

Do you guys have any idea how i can connect four 40GbE systems?

Thanks

 

[danb35]

10/40GbE are pretty much EOL technologies--which is why they're relatively cheap. For four ports of 40 GbE (as well as eight more of 10 GbE, and 48 of GbE), the Brocade ICX6610 fits the bill nicely--I believe all of the QSFP ports can be used for 40 GbE. For more information, including how to flash to the latest firmware, license all features, and configure, see:

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

NOTE #1: do not PM me with switch questions, they will be ignored - post them in this public thread, where hundreds of other members can also answer, and the answer will be public for future users NOTE #2 06-22-2023: Yes, this post is still up to date and nothing has changed: in fact judging by...

forums.servethehome.com

A step up would be the ICX6650.

 

[Fastline]

10/40GbE are pretty much EOL technologies--which is why they're relatively cheap. For four ports of 40 GbE (as well as eight more of 10 GbE, and 48 of GbE), the Brocade ICX6610 fits the bill nicely--I believe all of the QSFP ports can be used for 40 GbE. For more information, including how to flash to the latest firmware, license all features, and configure, see:

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

NOTE #1: do not PM me with switch questions, they will be ignored - post them in this public thread, where hundreds of other members can also answer, and the answer will be public for future users NOTE #2 06-22-2023: Yes, this post is still up to date and nothing has changed: in fact judging by...

forums.servethehome.com

A step up would be the ICX6650.

I think you're right. That's why mostly 25GbE and 100GbE nowadays. Thank you for the insights. Will have a look into it.

BTW, is it possible to combine like two different switches?

For example, one switch has 10GbE ports and other is 25GbE and clients are connected to the 10GbE switch and the 25GbE connected to the NAS. Can this be done?

 

[danb35]

BTW, is it possible to combine like two different switches?

It's always possible to chain switches together. Some switches support stacking, which will let you combine more than one switch in a way that acts like a single switch. The 6610 does, but then you're using the 40 GbE ports for stacking rather than for client machines.

 

[Etorix]

Initially, my plan was to get CRS518-16XS-2XQ-RM but the 2x100GbE do not support 2x40GbE speed on a 100GbE port on this switch.

Have you positively found the information that this is not supported, or is this an inference from not seeing this to be advertised?
I would generally expect all SFP28 ports to support 10 Gb/s (SFP+) and QSFP28 ports to support 40 Gb/s (QSFP+), but I would not expect vendors to put much emphasis on backward compatibility with QSFP+. SFP+ clients are going to stay for some time alongside SFP28, but QSFP+ is just going to be thrown out when upgrading network insfrastructure to 100 Gb/s and beyond.

 

[Fastline]

Have you positively found the information that this is not supported, or is this an inference from not seeing this to be advertised?
I would generally expect all SFP28 ports to support 10 Gb/s (SFP+) and QSFP28 ports to support 40 Gb/s (QSFP+), but I would not expect vendors to put much emphasis on backward compatibility with QSFP+. SFP+ clients are going to stay for some time alongside SFP28, but QSFP+ is just going to be thrown out when upgrading network insfrastructure to 100 Gb/s and beyond.

Well, yeah. That's what their manual says. On the 100GbE port, it supports the following:
1x 100G; 1x 40G; 4x 25G; 4x 10G; 4x 1G

But not:
2x 50G; 1x 50G, 2x 40G

On the 25GbE ports, it supports the following:
1x 25G; 1x 10G; 1x 1G

But not:
1x 5G; 1x 2.5G

So, I need to connect like 4 systems that has 40GbE NIC but with this switch, i would be just able to connect to two of them as 100GbE port don't support 2x40G.

 

[Etorix]

Ah, I had understood "2x40G" as "two ports at 40G". Squeezing two 40G links in a single 100G port is not going to work with any vendor.
To connect four 40G clients you'd need a CRS504-4XQ-IN.

 

[Fastline]

Ah, I had understood "2x40G" as "two ports at 40G". Squeezing two 40G links in a single 100G port is not going to work with any vendor.

Are you sure about that?

Not sure which Switch i saw but that had 2x40G support on the 100G ports. I might be wrong though.

The sad part is this switch does not have independent 40GbE ports.

To connect four 40G clients you'd need a CRS504-4XQ-IN.

That's my last choice.

Would it be possible to stack CRS518-16XS-2XQ-RM and CRS504-4XQ-IN?

 

[danb35]

Are you sure about that?

I'm 99+% sure of that. "40 GbE" is four lanes of 10 GbE in a single logical and physical connection. 100 GbE is four lanes of 25 GbE in a single logical and physical connection. So, while 2 x 40 GbE is less total bandwidth than 1 x 100 GbE, you won't be able to divide up a 100 GbE connection that way.

 

[Fastline]

I'm 99+% sure of that. "40 GbE" is four lanes of 10 GbE in a single logical and physical connection. 100 GbE is four lanes of 25 GbE in a single logical and physical connection. So, while 2 x 40 GbE is less total bandwidth than 1 x 100 GbE, you won't be able to divide up a 100 GbE connection that way.

I think you're right. This is why all of their switches do not support 2x40GbE. Makes sense. Thank you for the rock solid answer :)

Any idea on stacking so that it can act as a single switch?

 

[danb35]

Any idea on stacking

None at all. Check the docs for the switches in question.

 

[kspare]

32 Port 40GbE QSFP+ Cisco Nexus Enterprise Layer 3 VXLAN Switch Dual PSU | eBay

Find many great new & used options and get the best deals for 32 Port 40GbE QSFP+ Cisco Nexus Enterprise Layer 3 VXLAN Switch Dual PSU at the best online prices at eBay! Free shipping for many products!

www.ebay.ca

Check ebay for cisco nexus 3132 or 3164 switches. they are check and they work great. They are all 40gb ports, but you can get fan cables that break out a 40gb port to 4 10gb ports easily.