10Gb forum?

[melloa]

I know there are lots of people testing 10Gb connection with their FreeNAS servers, workstations, etc.

Why not create a sub-forum dedicated to 10Gb networking? That could give birth to resources users can use. a list of tested hardware, etc.

I know that @Chris Moore is probably upset with me at this point, with all my questions, but with his help I've done some research with VyOS and worked. Could not test it on a server hardware as I didn't have one available, but it is an option for people that want to have more than the, usually, four 10Gb uplink ports available in most switches.

So, suggestion:

- Sub-forum for 10Gb tests discussion
- Resource with list of known good switches, with how-to get it working

To get us started, for your review, the VyOS switch setup notes from my test (*) -- attached.

(*) Disclaimers:
1 - English is not my first language, so lots of grammar errors
2 - I'm a hobbyist, so don't expect anything technical

 

[m0nkey_]

10GbE has usually been covered by the Networking section of the forum. Plenty of 10GbE content there.

 

[melloa]

Plenty of 10GbE content there.

Still, could be consolidated.

 

[cobrakiller58]

I'm going to agree with @m0nkey_ it isn't a bad idea but in my mind it seems unnecessary. There is even a resource in regards to the topic with some tested equipment listed in it, adding to the list would be a great idea.

 

[melloa]

I'm going to agree with @m0nkey_ it isn't a bad idea but in my mind it seems unnecessary. There is even a resource in regards to the topic with some tested equipment listed in it, adding to the list would be a great idea.

If you mean: https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/, great resource. Give you great information on cards and suggests couple 10GBASE-T switches.

If you know of a resource for sfp+ tested switches/cards/cables/transceivers and configuration, I'd be interested. Right now I'm playing with Chelsio/Melanox/DAC, VyOS switch, and the Aruba S2500. I'm learning as I go, but wouldn't be nice if we all collected our experiences in one place? Just my thoughts.

 

[kdragon75]

LOL how about this... im using
HPE InfiniBand 4X DDR Conn-X PCI-E G2 Dual Port HCA (483514-B21)

I've ran into so many things to get these cards working... albeit dual port and win10, freenas and esxi 6.7 all see dual ports, only the top port is usable... so im still in the middle of fudging with these... even ripped the HP firmware out and put Mellanox oem on..due to HP stopped at 2.8 and I found 2.9.1000 from Mellanox. what an interesting journey. in addition, I don't know what switch can be used with these as they are not sfp+ or the others but a proprietary port... I would love to know of a switch that would work with these so I can use the top port to the switch and go from there as for now.. my esxi will have a few cards and pier to pier from a few machines. havent got that far yet. Just did part...
I suck at videos so have fun and bash away... but still informative...

https://youtu.be/boDBhL2RR4U

The little trapezoid with a clip on either side? Thats a CX4 connector. Its common with infiniband hardware and older 40gbe switches. Most things are moving ot fiber now though.

 

[kdragon75]

If you mean: https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/, great resource. Give you great information on cards and suggests couple 10GBASE-T switches.

If you know of a resource for sfp+ tested switches/cards/cables/transceivers and configuration, I'd be interested. Right now I'm playing with Chelsio/Melanox/DAC, VyOS switch, and the Aruba S2500. I'm learning as I go, but wouldn't be nice if we all collected our experiences in one place? Just my thoughts.

I'm with the others. We have some great 10gbe resources but dont need a sub forum for it. For SFP compatibility, refer to the manufacturer and others experience.

 

[JustinClift]

well I've got all the hardware.. tons of cards and cables.... do you know of a switch that does ethernet?
looking at this on ebay HP ProCurve Switch 6400cl J8433A 6 Ports 10 Gbe CX4 Ports

Yep, that'll work. I've used the HP Procurve 3400cl switches before with Mellanox ConnectX and ConnectX-2 (CX4 connector) cards.

The 3400cl ones are 24x and 48x port 1GbE models, with an optional 2x 10GbE add on module (ports on the back). The add on module has a couple of models, with one having CX4 connectors. It works fine with the above mentioned Mellanox cards in 10GbE mode. :)

Examples (looking quickly on Ebay):

* https://www.ebay.com/itm/J8434A-HP-PROCURVE-SWITCH-cl-10-GBE-CX4-COPPER-MODULE/264154972330
* https://www.ebay.com/itm/HP-ProCurv...00CL-48G-w-J8443AA-10-GbE-Module/312424720477
* https://www.ebay.com/itm/J4906A-HP-...tch-w-J8434A-10GbE-Copper-Module/232639930324

From memory, the 6400cl is the same generation stuff, but is 10GbE only instead of having a bunch of 1GbE ports.

On the plus side, the power draw for CX4 connector cards is lower than 10GBASE-T.
On the negative side... the power draw for the HP Procurve 3400cl switches is around 70W, measured at the wall with a decent meter, with the 10GbE module installed. Pretty much equivalent to leaving a desktop computer on 24/7.

 

[jgreco]

Yep, that'll work. I've used the HP Procurve 3400cl switches before with Mellanox ConnectX and ConnectX-2 (CX4 connector) cards.

The 3400cl ones are 24x and 48x port 1GbE models, with an optional 2x 10GbE add on modules (ports on the back). The add on module has a couple of models, with one having CX4 connectors. It works fine with the above mentioned Mellanox cards in 10GbE mode. :)

Examples (looking quickly on Ebay):

* https://www.ebay.com/itm/J8434A-HP-PROCURVE-SWITCH-cl-10-GBE-CX4-COPPER-MODULE/264154972330
* https://www.ebay.com/itm/HP-ProCurv...00CL-48G-w-J8443AA-10-GbE-Module/312424720477
* https://www.ebay.com/itm/J4906A-HP-...tch-w-J8434A-10GbE-Copper-Module/232639930324

From memory, the 6400cl is the same generation stuff, but is 10GbE only instead of having a bunch of 1GbE ports.

On the plus side, the power draw for CX4 connector cards is lower than 10GBASE-T.
On the negative side... the power draw for the HP Procurve 3400cl switches is around 70W, measured at the wall with a decent meter, with the 10GbE module installed. Pretty much equivalent to leaving a desktop computer on 24/7.

Unless you already have CX4 hardware, you should desperately try to avoid it. The levels of compatibility are miserable compared to SFP+, and some manufacturers have used the CX4 interface in "alternative" manners such as dedicated stacking ports. I warn home users off of CX4 in the 10G Primer not because it can't be made to work, it absolutely *can*, but you might be short hair, money, time, and end up with an excess of frustration by the time you're done.

A moderately sized 10G SFP+ switch will burn watts. One of our Dell 8132F's is chowing through 120W.

Code:

1     System       OK            10.2        121.4
1     Internal(R)  OK            52.6        52.5      01/23/2019 22:17:45
1     Internal(L)  OK            69.3        68.8      01/21/2019 16:40:02

which is pretty normal, but then again it is a full layer 3 switch and is a bit hungrier than other stuff. I should go revisit the 10G Primer with some power numbers for stuff sometime I suppose.

 

[JustinClift]

Thanks @jgreco, that's really good info.

I should go revisit the 10G Primer with some power numbers for stuff sometime I suppose.

That would be really useful for people! :)

Power draw wise, the Mikrotik switches seem much lower:

* https://mikrotik.com/product/crs305_1g_4s_in : super tiny model, 4x 10GbE ports : 10-18W
* https://mikrotik.com/product/crs309_1g_8s_in : 8x 10GbE ports : 17-23W
* https://mikrotik.com/product/crs328_24p_4s_rm : 24x 1GbE + 4x 10GbE : 44-494W. High max power is due to its having PoE ports.

Some pro's and cons:

* Pro for the 4x 10GbE port model - it's fanless, so should be silent.
* Con - They seem to have had problems with their firmware security lately, so probably really only safe for internal networks. eg not in public hosting infrastructure, just to be safe

The above is only from their published specs. Haven't used their switches in person yet, so no idea of real world power draw when measured from the wall. But probably will in future just to see what's they're like.

Remembered another set of pros/cons for the HP switch (as above):

* Pro - they still get firmware updates - or did when I was still using them a year ago, even though far out of warranty - which was nice. :)
* Con - they're not fanless - so not silent. They were kind of annoying to have running in the same room, when everything else was mostly silent. They're not especially loud, just kept getting on my nerves. ;)

 

[Constantin]

At one of the sites, I use the Mikrotik-CSS326-24G-2S-RM and the power consumption is allegedly around 10W. It can be powered by POE, has no fans, metal case, rack mountable, etc. Two SFP+ ports, 24 Gigabit RJ45. It's a great managed L2 switch.

 

[jgreco]

At one of the sites, I use the Mikrotik-CSS326-24G-2S-RM and the power consumption is allegedly around 10W. It can be powered by POE, has no fans, metal case, rack mountable, etc. Two SFP+ ports, 24 Gigabit RJ45. It's a great managed L2 switch.

That seems unlikely. Your typical small 8 port 1G ethernet switch might take around 5W just to power the chipset, and once you start bringing up links, that's about an additional 0.5W per link (somewhat distance dependent). An SFP+ can take up to 1.5W all on its own, so assuming you had the chipset, two SFP+'s, and 24 ports in use, I'd find it hard to believe it eats much less than 20W. I'll believe 10W if you plug it in with nothing else plugged into it, but I can cut that to 0W with no loss of functionality by unplugging its power.

This is an area where I'd love to be wrong. Switchgear wattage overhead sucks. The 48 1G port, quad SFP+ PoE aggr switches here eat around 80W when lightly loaded and the 10/40 switches eat around 120W each. Granted they're all layer 3 gear with all the knobs but they're still space heaters. It seems like no one's really putting much effort into low end switches with 10G, something I'd like to see a lot more of.

 

[Constantin]

I stand corrected. The maximum power per the OEM is 19W, presumably if all ports are active. It's 24W for the CRS326, which enables routerOS L3 mode. I'm pretty sure I got the 10W figure from my use for the thing, which is 2 ports SFP+ and 8 gigabit ports. But let me double-check that number.

Either way, it can't be much as the unit lacks active cooling, uses a metal case, and feels like 2-3*C over ambient. The power efficiency of this switch was one of the key features for me, as it allowed the unit to operate fanless.

 

[jgreco]

The maximum power per the OEM is 19W,

NAILED IT! *does a silly little dance* ;-)

 

[JustinClift]

Reading the specs for that model on the Mikrotik website, it does look like a potentially good option for desktop/office/home-lab use. eg where silent and low power draw are things to consider. :)

 

[Constantin]

NAILED IT! *does a silly little dance* ;-)

Glad I could add some movement / mirth to your day.:)

For those interested in this type switch, the plug power draw of my unit (which is the slightly older CRS226-24G-2S+RM) was 9W at idle ramping up to 13W under my light loads. I presume the CRS326 series would be similar.

The CRS226 series is interesting in that they had a compact model which easily fit inside my cabinet. Mikrotik never released SwOS for the 226, however. Depending on your opinion of SwOS (reported unresolved issues re packet reordering from SFP+ to gigabit) that may be a good thing. But configuring RouterOS for a newbie is even more intimidating than FreeNAS (likely eliciting spontaneous laughter from network admins here who could do it from the CLI with their eyes closed).

 

[melloa]

* https://mikrotik.com/product/crs305_1g_4s_in : super tiny model, 4x 10GbE ports : 10-18W
* https://mikrotik.com/product/crs309_1g_8s_in : 8x 10GbE ports : 17-23W
* https://mikrotik.com/product/crs328_24p_4s_rm : 24x 1GbE + 4x 10GbE : 44-494W. High max power is due to its having PoE ports.

Just got the little brother to test, uplinked to my S2500. The Aruba PSUs are going back left and right here, so might be replacing as they go bad.

 

[Snow]

They also have a 16 Port as well I love it. It can run a Mesh AP network from the Switch as well. I was going to go the Cisco route was at like $1100-$1400. With this set up the MikroTik and 3 Aps costed me less then $650 and 10GigTek sells the SFP+ new for $22. if you go to fs.com you can get them for $16. My used gen Cisco Sfp+_SR_LC where $22 Per.

I herd that the Firmware problems have been Fixed, They have a update on there Forums. The nice thing about their switches is there OS they use is pretty much the same across all the devices. So if you buy AP's from them you get them with RouterOS. Push's 32 Watts with 8 Sfp+ 1Ge
One last thing the Learning curve is just like FN , You will save a good amount of money but you will spend some time learning there stuff!
MikroTIK 16SFP+ 1Ge

 

[melloa]

I'm glad SFP+ is getting to a cost that home users can afford. That switch you've linked would probably be the one I'd go with, if starting my setup today.

 

[jgreco]

If you know what Burroughs B1799, IBM 4341, z-80 and Motorola 68000 are, you can guess my age.

Remember the fun old days of "networking" with 110 baud acoustic couplers? How far we've come.