Yes I do understand that LACP is not for routing, it's for fault tolerance and (depending on the setup type) load balancing. I also guess that I can control the routes through static routing but it feels like I'd lose the flexibility to change things from my central DHCP server if I'm setting routes in TrueNAS.
Perhaps my understanding of linux networking is wrong (god knows I found out how messed up my understanding of mikrotik architecture was when I posted on the mikrotik forums) but here is my understanding:
- When I connect to a host in the 192.168.20.0/24 subnet, that traffic will default to using the interface in the same subnet, even if it has a lower priority than others.
- When trying to connect to a host in another subnet that the machine is not on (say 192.168.100.0/24) or an internet IP (say 8.8.8.8), this traffic will go over the default route.
- However, when I don't manually specify interface priorities I can't guarantee which interface will be selected as the default route every time.
Why do I want to do this? Well the hosts on the 192.168.20.0/24 subnet will all have 10Gbe connections so I might want to set the MTU on this interface to 9000 as I know all these hosts are connected to the same switch and thus the interface for this subnet traffic can be safely set to use MTU 9000. However hosts on other subnets are on a diff. Switch and that traffic (plus internet traffic) should be limited to MTU 1500.
If I can set an interface priority, I can achieve that Goal and still have most other things handled from my DHCP server. Indeed this is how it's currently setup in my existing Ubuntu install.
This isn't even "Linux" networking. It generally applies to IP stacks with a BSD heritage, which is most modern UNIX systems, OS X, even Windows.
Traffic egress (that is, traffic flowing outward) from a host is normally controlled by the routing table. While there ARE embellishments such as ipfw forward rules or alternate routing tables, let's stick with a typical host design like what you find for TrueNAS.
The routing table on a UNIX system has several elements, including things like the ARP table, which operates at a lower level, and the routing/forwarding engine itself, and then any routing protocols, which operate at a higher level.
The routing engine is responsible for figuring out what to do with a packet. To do this, it looks at the destination IP of the packet, looks it up in the routing table (see "netstat -rn"), and then dispatches the packet appropriately. Modern routing operates primarily on a longest-prefix basis, which means that a route of 10.0.0.0/8 is less specific than 10.123.0.0/16 is less specific than 10.123.45.0/24.
You previously mentioned route metrics. Route metrics are primarily an abstraction from the 1980's that never found wide acceptance at the routing engine level. They can be used as tiebreakers, but for the most part, metrics are handled by upper level routing protocols such as RIP or OSPF, and should not exist at the routing engine level.
The question I usually ask people who are insistent that "metrics are important" is which of these routes should a kernel select:
10.123.0.0/16 metric 10
10.123.45.0/24 metric 20
if handed a packet destined for 10.123.45.67
"well that's confusing!" Metrics may be helpful as a tie-breaker between two identical routes when the kernel is aware of the availability status of the next hop (such as whether or not there is ethernet link or an ARP entry), but in most cases, metrics as part of the basic IP configuration of a system haven't been relevant for years.
So the rest of the time, you should design for longest-prefix, and forget about metrics.
Here's how that works.
When you create an IP interface on your NAS, this implicitly creates a route in your routing table. On FreeBSD, this would look like
Code:
# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.123.40.1 UGS vmx7
10.123.40.0/24 link#1 U vmx7
10.123.40.140 link#1 UHS lo0
10.123.50.0/24 link#2 U vmx8
10.123.50.140 link#2 UHS lo0
127.0.0.1 link#3 UH lo0
Now what this is saying is that the preferred route to hosts on 10.123.40.0 is out via vmx7, and 10.123.50.0 is via vmx8, except that if the host is itself (10.123.40.140 or 10.123.50.140) then use localhost. All other traffic gets punted to 10.123.40.1 via vmx7.
So here's a few points I wanted to make in response to your post:
I can't guarantee which interface will be selected as the default route every time
Of course you can. The thing listed in the "default" line is what's going to be used when there is not a more specific choice for the forwarding engine. I don't know where people get these weird ideas about each interface having its "own" default route.
I'd lose the flexibility to change things from my central DHCP server if I'm setting routes in TrueNAS.
Two points:
1) If you are reliant on this fileserver for any sort of important thing, it should be set to a static IP address. DHCP is unreliable. The DHCP server can be down, its disk can be full, it can run out of leases. I had someone who DESPERATELY wanted to use DHCP in this sort of role, and then one day the hypervisor couldn't renew its lease, and the NAS was offline too, and ... was the DHCP server a VM? I can't remember anymore, but you can get really hosed real quick if you give into the temptation to outclever yourself and make stupid dependencies. Infrastructure should be assigned static IP's. If it's only your Plex library, then, fine, who cares.
2) You can push additional routes via DHCP if you need to ... not that I'm seeing a need here.
will all have 10Gbe connections so I might want to set the MTU on this interface to 9000
May not be a good idea. Jumbo frames are fickle. The theory sounds great, but modern cards and switchgear should be able to do a really good job at 1500, and you avoid a bunch of problems such as
outlined in this article.
Perhaps my understanding of linux networking is wrong
Well, it's not bad to be wrong. I hope you can find some new and interesting bits to digest in this thread. Most of the posters here enjoy helping broaden horizons.
