NIC Priority - TrueNAS-SCALE-22.02.0.1

[bumperdoo]

Hi,

With more than one NIC on a TrueNAS build, how does it decide which NIC to use?

In my setup, I have 2 10GbE NIC connections to a 10GbE switch and one 25GbE NIC to a 25GbE switch.

TIA

 

[sretalla]

Ooooh... you're trying (or even if not, you will have succeeded) to anger the grinch (@jgreco)

He has written several clear articles about why you can't do what you're doing there on TrueNAS CORE... since the network stack is different in Linux (SCALE), maybe there's some redemption possible, but I'm pretty sure the theory is similar.

If you want to use 3 NICs on the same subnet, you need to aggregate them with LAGG/LACP (if your switch can support it).

Although they may not be directly applicable, here is the resource I'm talking about:

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

There have been other more recent discussions in depth about the disasters of multi-subnet "routing" in FreeBSD (CORE), but I guess those may be less applicable.

 

[bumperdoo]

why you can't do what you're doing there on TrueNAS CORE... since the network stack is different in Linux (SCALE)

I'm on TrueNAS Scale, not CORE. Same implication?

 

[jgreco]

since the network stack is different in Linux (SCALE), maybe there's some redemption possible, but I'm pretty sure the theory is similar.

Just because the actual network stack code is different in Linux doesn't mean that there's a different set of RFC's and standards written specially for Linuxites. We've had several people try and fail to demonstrate that Linux is magically different. It hasn't gone well, because this is really about networking fundamentals and how we've moved from very simplistic IP stacks that did indeed talk to a single card to these super-capable and abstract stacks that have layers upon layers of features. I can definitely dig up gear where you can "bind" an IP address to a physical interface, but this is the exception rather than the rule, and generally only happens in systems with extremely limited networking capabilities -- usually ones highly optimized for peak performance by jettisoning all the abstractions.

TrueNAS, both Core and Scale, aren't that. They're basically built on standard-ish Linux and FreeBSD kernels.

With more than one NIC on a TrueNAS build, how does it decide which NIC to use?

As correct as @sretalla is in that this has been answered elsewhere, it may not be in a form that directly addresses this question. It's easy for some of us to see that it's just a variation on the topic.

Ingress traffic (incoming) will be determined by switching topology and MAC address resolution. Your NAS may be able to influence what is advertised to switches out each port for MAC address resolution, but fundamentally it will be "whatever your switches decide to send based on the learned MAC addresses".

Egress traffic is determined by the routing table. I've linked to an article above where someone made the unfortunate error of trying to explain metrics and Linux routing to me and was shown to be very wrong; I am happy to discuss the topic in depth especially if you're interested in learning. It turns out that you really need a proper layer 3 (probably) network design to truly take advantage of multiple network interfaces. You can get part of the way there with random luck or link aggregation. If you were a glutton for punishment, you could get more of the way there with a bunch of manual training of the system as to what is where on the network, but it's really quite difficult and fragile.

 

[bumperdoo]

TrueNAS, both Core and Scale, aren't that.

Is the simplest solution to just use the Intel 25GbE NIC and ignore the 10GbE Broadcoms built into the MOBO?

I love to learn. Networking is def complex.

 

[jgreco]

It depends. Broadcom 10G tends to completely suck under FreeBSD; they're not the best of breed under Linux but they are serviceable there I guess. This is in general, and your specifics may differ.

The real problem is likely to be that the rest of your network is unlikely to be optimal to handle the traffic. Onboard ethernet chipsets tend to deliver 10GBase-T ports, which are suboptimal because they're higher latency and more prone to weird cabling issues. But if you happened to have a workstation near your NAS and you wanted to deliver something like a "private" network, you could certainly try a pointopoint network for that, so they're not entirely useless. Don't dismiss the possibility.

Otherwise, this gets down to your switching environment and what you're trying to accomplish. For example, if you had three VLAN-capable switches, one with 25GbE and two with 10GbE, and you were using the 25GbE as a homelab for some ESXi hypervisor interconnection, one of the 10GbE's as your PC network, and one of the 10GbE's for miscellaneous, you could potentially create three different VLAN's, with 192.168.1.0/24 on one of the 10G's for your existing stuff, then 192.168.2.0/24 for the miscellaneous, and 192.168.3.0/24 on the 25GbE for ESXi. You don't actually need VLAN-capable switches if you just have three non-interconnected switches, but then you may want to interconnect them so that all three vlans can be configured to be accessible from any of the ports. But in the meantime, 25GbE traffic could be isolated to the XL710 interface and your ESXi hypervisors and localized on the "better" switch.

Designing networks is complicated.