Faster than 10 gigabit options, switching choices?

mcline

Cadet
Joined
Aug 16, 2019
Messages
2
I've finally hit the point where the 10 GbE LAN is a real-world bottleneck. I have considered a 2x link aggregation, but that feels like a short-term stopgap when what we need is more headroom. Basically, we want to get back to the design assumption that the LAN is never a bottleneck.

Naturally, the cheap eBay surplus of 25 GbE and 40 GbE cards is tempting, as are FS's cheap transceivers.

However, I'm seeing a real lack of 25 and 40 GbE switching gear at anything but an enterprise price point or with a datacenter-level power draw. There doesn't seem to be much gear appropriate for a workgroup instead of a rack or spine. Currently in startup/development mode, so we're being frugal with cash. A $5000+ switch is out of the question.

Has anyone been successful in migrating to a faster than 10 gigabit network without breaking the bank, either in hardware acquisition or power draw? Don't need a 48 port solution. We would be fine with eight ports running faster than 10.

For cost effectiveness, should we be looking at 25 or 40? Either would meet our needs (40, or 25 with room to aggregate).

SSD arrays could nearly saturate 100, so storage won't be the limiting factor.

Is this realistic right now, or is the switching hardware just not available per the needed price and power specs?
 
Joined
Dec 29, 2014
Messages
1,135
I picked up some Cisco Nexus 3064 switched used on eBay, and they are working fine. They have 4 40Gb ports each, and I paid $300-400 for them. You can get 40Gb direct attach cables fairly inexpensively from fs.com for short range. I don't have any hosts yet with 40Gb interfaces, but it is on my radar. I have done a switch to switch link with 40Gb, and it seemed to work. The Chelsio T580 is a 40Gb capable NIC that should work really well with FreeNAS.
 

mcline

Cadet
Joined
Aug 16, 2019
Messages
2
What's your power consumption look like?

This development lab is powered by PG&E. We're paying nearly $0.40/kWh for electricity at the margin, which is obscene. You can double that when factoring in cooling to remove heat generated by the gear. When it comes to energy costs, we're just stuck. Nothing we can do but mitigate where possible, bite the bullet, and pay the bill.
 
Joined
Dec 29, 2014
Messages
1,135
What's your power consumption look like?
I haven't looked, probably because I don't really want to know. I think the Nexus draws more than my old HP 5820X (really a re-branded H3C), but I like the interface better in the Nexus because it is more familiar. I don't know if the H3C stuff has 40G options, but those were much more expensive. Those used Nexus switches were just too good of a deal for me to pass up. The fans are a little louder and push a lot more air than the H3C one did, but it is tolerable.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
Top