Can i setup LAGG without L3 switch?

[sahans]

hi, I want to know how can I set up link aggregation in FreeNAS without l3 switch

 

[sretalla]

LAGG is layer 2, so you can, but you need a managed switch to enable it properly on the switch end (which will almost always also switch at layer 3).

 

[jgreco]

LAGG is layer 2, so you can, but you need a managed switch to enable it properly on the switch end (which will almost always also switch at layer 3).

Ah, what??

Most switches do NOT switch at layer 3. Layer 3 switching requires substantially more complicated silicon. Some switches such as the Mikrotiks can be loaded with firmware that does provide layer 3 functionality, but if you look at what is going on, it isn't doing level 3 switching, but rather level 3 routing in software. Layer 3 switching is done in silicon.

By comparison, almost any managed switch, including many crippleware "web-managed" ones, can do LACP. The $70 Netgear GS108Tv2 switches I buy as breakout switches where more ports are needed are an excellent example; these can do both VLAN and LACP, so it's fun to create a tagged LAG of two uplinks and get six usable ports, each of which can be on any vlan in your switching environment.

 

[sretalla]

it isn't doing level 3 switching, but rather level 3 routing in software

OK, fair enough, but that's what I meant. But indeed, your answer is better for the OP's question.

 

[jgreco]

OK, fair enough, but that's what I meant. But indeed, your answer is better for the OP's question.

And I probably should have completed my thought there...

Layer 3 switching ("in silicon") happens at about the same rate as layer 2 switching, which is to say, up to almost 1.5 million packets per second for a gigabit ethernet port, on each port. This can be a crapton of million PPS across a switch, but there's certainly a lot of switches out there that can do this. They're generally high-end switches that originally sold for high-four-figure or five-figure sums. Most such devices will only be able to support a few thousand IP routes, as an additional limitation.

By way of comparison, you need a LOT of CPU to do this in software, but you can support "full table" routing if you have sufficient memory. The problem is that the software forwarding rates available aren't that great. If we look at an accelerated 10G router such as the Ubiquiti EdgeRouter Infinity, which is 8x10G ports, it is only rated for 18 million packets per second, and this thing has a 16-core CPU that is MUCH beefier than your typical switch. Even with that, it couldn't switch full line rate on a 24-port 1GbE switch (which requires 35.6Mpps to be able to qualify as "wire speed").

The reason it is important to understand this difference is because if you are doing something like setting up iSCSI or other storage in a routed environment, you will find yourself bottlenecking if you're using software routing.

I find this stuff fascinating. I should probably point out that my office printer stand is an Ascend GRF-400, one of the very first true route-in-silicon routers.

But to circle back around to the OP, yes it should be easy to find an inexpensive layer 2 switch that does LACP.

 

[no_connection]

I got the feeling that back in the day managed switch and L3 switch was pretty much same thing. And as long as you had the destination in routing table it was HW offloaded and only hit CPU when route was unknown. But that changed with gigabit?
Granted I had mostly played with 3com 4400 and the like then. Probably not relevant any more but I do find it interesting.

 

[Yorick]

I got the feeling that back in the day managed switch and L3 switch was pretty much same thing

That depends on the switch and how you define "L3 switch". There are managed L2 switches that cannot route at all. There are also managed L2 switches with "L3 Lite" - they can do static routing but understand no routing protocols at all.

 

[jgreco]

I got the feeling that back in the day managed switch and L3 switch was pretty much same thing. And as long as you had the destination in routing table it was HW offloaded and only hit CPU when route was unknown. But that changed with gigabit?
Granted I had mostly played with 3com 4400 and the like then. Probably not relevant any more but I do find it interesting.

Um, that in no way resembles the history of switching, at least not as I remember it.

There was a long period of time when switching and routing were sharply delineated except perhaps at the high end of the spectrum. That Ascend GRF-400 I mentioned was an $80K device which could sort-of do some switching, but normally no one would waste a $2500 ethernet port on a router to do switching when you could get a switch that was much cheaper per port.

Kalpana arguably created the first ethernet switch in 1989, but as technology rapidly advanced, the introduction of things like 100Mbps ethernet, which was a fail-riffic at interoperability between devices, meant that each end of an ethernet connection needed to be configured in many cases, so a lot of the early switchgear picked up management capabilities to allow you to twiddle the port speed and duplex. This was rapidly followed by companies like Wellfleet and Synoptics who were busy adding value by creating the ability to support multiple partitioned broadcast domains ("vlans") and trunking, which required some way to configure all of that. Most of these did not have web interfaces because the web hadn't been invented quite yet. Some had cheesy telnet UI's (I'm looking at you Bay 350T!) or CLI's.

The challenge for layer 3 switching is the route table. Unlike a MAC table, it involves a complex lookup, which has generally been implemented with TCAM, but this is incredibly expensive for a mere switch. Routers that take a full BGP table need hundreds of thousands of entries. The GRF-400 died and became my printer stand because it only had the ability to support around 170K routes, and it is a *dedicated* router that really did have the hardware capacity to handle that in silicon. So layer 3 switches had basically two different ways to roll. One was to implement a small amount of TCAM and be able to do wire-speed routing for a small number of routes. The other was to advertise "layer 3 routing" as a checkbox item, but to implement it in software and be limited to some tens of thousands of packets per second. As you note, some tried an intermediate path that did some hardware offload with CPU, but that sucked royally for any network where you had traffic exceeding the one or two thousand routes that were cached in the hardware. Performance would be great under light load and then fall off a cliff under the load of many users going to many places on the Internet. That doesn't mean that this stuff didn't work if you threw enough money at it. Lots of companies made lots of gear that combined a bit of both, but it was $$$$$$$ for stuff that worked well.

So in general, no, managed switches and layer 3 switches were not pretty much the same thing. Layer 3 switches were invariably managed switches, but there are a lot of us who liked to build networks cheaply, which usually turned out to be out of managed layer 2 switches, and there were lots of options for that.

 

[no_connection]

Tanks for the history, I guess I only encountered a few number of models and the ppl I knew referred to them as l3 rather than managed. And I was a lot younger as well. I got like 10 3com 4400 from the scrap bin and stacked them and bolted them together to form a rack and played with that. Good times. Some of them actually found use and are still running today.

The only reason I would see for L3 routing on switches is decrease broadcast domain to a single switch and then route between whatever number of subnets needed, any more complicated routing would pass a real router anyway.

 

[Yorick]

The only reason I would see for L3 routing on switches is decrease broadcast domain to a single switch and then route between whatever number of subnets needed,

This is going really VERY far afield, but: Look at modern campus networks like Cisco's SDA. Fully routed IS-IS underlay, with VXLAN and LISP at the edge. This might seem like overkill, but actually, if you can wrap your head around the "network as code" paradigm, greatly simplifies things. One protocol in core (two if you need multicast, that adds PIM-SM), three at the edge. Rather than a good dozen or more in a traditional network. This reduces the chance of running into bugs. Centralized orchestrated config, reduces the chance of human error. Easy segmentation without the need for VLANs, segment with secure group tags instead. Think ACL on steroids.

This idea is still very much in flux, there are features such as SD-WAN transport that are still being improved, and if you step away from the reference architecture ("my underlay must be OSPF!" why tho?), that can be challenging.

As a network engineer, the idea of reducing the "protocol stack" is very, very promising. It does require a whole new way of thinking about deployment and config.

 

[jgreco]

The only reason I would see for L3 routing on switches is decrease broadcast domain to a single switch and then route between whatever number of subnets needed, any more complicated routing would pass a real router anyway.

That's often not what happens in production networks, however. It turns out that for small and medium sized local area networks that it is extremely convenient to run most or all vlans to all switches and then have a virtual network. Done correctly, this allows you to do things like LACP failover when a switch fails, and all sorts of flexibility in segmenting the network intelligently.

 

[jgreco]

This is going really VERY far afield, but: Look at modern campus networks like Cisco's SDA. Fully routed IS-IS underlay, with VXLAN and LISP at the edge. This might seem like overkill, but actually, if you can wrap your head around the "network as code" paradigm, greatly simplifies things. One protocol in core (two if you need multicast, that adds PIM-SM), three at the edge. Rather than a good dozen or more in a traditional network. This reduces the chance of running into bugs. Centralized orchestrated config, reduces the chance of human error. Easy segmentation without the need for VLANs, segment with secure group tags instead. Think ACL on steroids.

This idea is still very much in flux, there are features such as SD-WAN transport that are still being improved, and if you step away from the reference architecture ("my underlay must be OSPF!" why tho?), that can be challenging.

As a network engineer, the idea of reducing the "protocol stack" is very, very promising. It does require a whole new way of thinking about deployment and config.

Well, switch configurations have definitely become insanely complicated, but the "network as code" paradigm isn't particularly relevant in many cases. It's great if you're doing large scale data center design, but it is really akin to Kubernetes-vs-a-bog-standard-server.

Some of us run things like OSPF because there are literally no other options, because it's taken a quarter of a century for OSPF to become standardized enough to be *likely* to work (and even then I have counterexamples). You might like IS-IS but it looks like a grad student's research project if you try to implement it on an existing network. Existing networks do not survive "very much in flux" particularly well. And some of us do clever things with routing protocols. All our major servers run OSPF and inject stubs for what are essentially FreeBSD jails with an alias on lo1. This was a great way to get redundant connections to servers in the pre-VRRP days... and it has a bunch of other benefits too.

I would absolutely adore the ability to get away from some of the complexities of modern switching, but to date it appears like a lot of what the cloud folks are trying to do with their "network as code" crap is really only suitable for certain edge cases. (oh look a bad neteng joke!)

 

[Yorick]

I hear your pain there. If/when I have my first real-world SDA deploy, I'll let you know how it goes :)

 

[no_connection]

With OSPF you can get pretty insane response times. Let it handle core communication routing and run MPLS for traffic. Seen it done between large core routers with long fiber runs in between them for city wide network. Was some time ago. Think it was maybe 1ms pull rate or something crazy like that, so very little packet loss.