TrueNAS core - multiple NICs

[agood]

Well, feels like beating a dead horse but there is no good guide here or anywhere i can find and I have looked for a long time. Windows background here; so while I am familiar with vnet and bridge concepts etc; it is still somehwat confusing and may just come down to wording.

Setup:
TrueNAS core 13
3 NICs: 1 inbuilt, Dual NIC card

The often used phrase that you "cannot use multiple NICs on a subnet" has to be refined as it is misleading and plain not true. You cannot use "Multiple NICs on a single subnet PER HOST"; sure you can use multiple NICs (just one per device or separate network) otherwise you could only use 1 device per subnet which is contrary to the idea of "networking".

So, what I would like and need to achieve:

1 dedicated NIC for host = TRUENAS with static IP
2 NICs in LAGG (or not; does not matter; switch supports it) for everything else = VMs and Jails (Plugins)

What I need to avoid is NAT or VLAN (double NATing does not work for some services and port forwarding and opening ports in a firewall becomes a nightmare); So I would like, like in a virtual switch, attach all devices with a static or DHCP assigned address on the same subnet.

Cheers!

 

[jgreco]

there is no good guide here or anywhere i can find and I have looked for a long time.

Well, this isn't true.

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

The often used phrase that you "cannot use multiple NICs on a subnet" has to be refined as it is misleading and plain not true. You cannot use "Multiple NICs on a single subnet PER HOST"; sure you can use multiple NICs (just one per device or separate network) otherwise you could only use 1 device per subnet which is contrary to the idea of "networking".

Well, you're wrong. You can absolutely have multiple NIC's in a subnet. A NIC is a physical ethernet interface, and abstractions such as LACP or bridges require this in order to function. What isn't allowed is for a host to have multiple separate logical interfaces addressed within a single broadcast domain. So you haven't discovered some "gotcha". As I say in the linked article, it's really a terminology problem. You can try stuff like saying "but you say interfaces and that's clearly a NIC" to muddy the waters, as others have tried at times, but the problem is that we just don't have good unambiguous words for "the term for the logical abstraction of a UNIX network interface that refers to the interface that owns an IP address and not the physical ethernet chipset itself."

This is a tough topic to discuss because people have been taught stupid things and it isn't really that easy to understand what's really going on in the first place. It's more complex in the modern era because the NASware now supports virtual abstractions such as virtual machines, jails with VNET, etc., each of which qualify as legitimately separate hosts with their own IP stacks, despite running on a single physical host.

 

[sretalla]

So I would like, like in a virtual switch, attach all devices with a static or DHCP assigned address on the same subnet.

Look for posts from @Patrick M. Hausen on the topic.

What you will want is a LAGG and a bridge (which is the "switch" you're looking for).

it goes like this:

Physical NIC(s) -> LAGG -> VLAN (if you're doing that) -> Bridge (IP address of the host goes here if you're assigning one).

You then attach your jails/VMs to the bridge rather than to a Physical NIC.

 

[agood]

Look for posts from @Patrick M. Hausen on the topic.

What you will want is a LAGG and a bridge (which is the "switch" you're looking for).

it goes like this:

Physical NIC(s) -> LAGG -> VLAN (if you're doing that) -> Bridge (IP address of the host goes here if you're assigning one).

You then attach your jails/VMs to the bridge rather than to a Physical NIC.

Thanks! Really appreciate everyone's input. It just seems so many bits of information in different areas, trying to collect and make sense of them.
Was easy in the Windows world and despite the notion "What works vs what's right" it worked for many years without too many glitches. In Hyper-V, add HyperV switch (even choose your NIC) - done.

So in order to get my scenario to work, I actually need to tag Switch ports with VLAN IDs?

Sad thing is I need a Unifi controller in order to do this....

 

[sretalla]

So in order to get my scenario to work, I actually need to tag Switch ports with VLAN IDs?

No... as I mention in my post, you only need that step if you're doing VLANs.

 

[agood]

Great, thank you, really appreciate your patience. Just one more thing.....how can I attach multiple bridges to a LAGG? I tried and was only every able to attach one?

 

[sretalla]

how can I attach multiple bridges to a LAGG? I tried and was only every able to attach one?

A LAGG is "one interface" in the sense that it (and all of its child interfaces) is behaving as one "virtual cable".

A Bridge in this case is the equivalent of a Virtual Switch.

In that understanding, how would you normally attach one cable to more than one switch?

Short version: I don't think you can.

But... you can use VLANs to get the "result" you want.

Yes, I know you don't want to use VLANs.

 

[sretalla]

Maybe we're working at this the wrong way around... what are you trying to achieve that needs multiple bridges?

 

[agood]

Thanks. As in my original post:

1 dedicated NIC for host = TRUENAS with static IP
2 NICs (however in LAGG or not; does not matter) for everything else = VMs and Jails (Plugins) a least 1 VM and at least 2 jails

....and they need to be on the same subnet.....

Cheers!

 

[jgreco]

Thanks. As in my original post:

1 dedicated NIC for host = TRUENAS with static IP
2 NICs (however in LAGG or not; does not matter) for everything else = VMs and Jails (Plugins) a least 1 VM and at least 2 jails

....and they need to be on the same subnet.....

Cheers!

So? Why can't you do this?

As I pointed out earlier, this is fine as long as each device has its own IP stack. However, a single host, i.e. a single IP stack, cannot have multiple interfaces on a single subnet. That does NOT work, or at least it does not work the way most people would imagine it to, and then they get angry... usually at me, for having been the one to explain to them how it really does work.

 

[agood]

How?

So? Why can't you do this?

As I pointed out earlier, this is fine as long as each device has its own IP stack. However, a single host, i.e. a single IP stack, cannot have multiple interfaces on a single subnet. That does NOT work, or at least it does not work the way most people would imagine it to, and then they get angry... usually at me, for having been the one to explain to them how it really does work.

 

[jgreco]

How?

By configuring it, of course. You set up one ethernet interface for the TrueNAS host's IP stack, configured to handle the IP address of the NAS. You set up the other ethernet as a member of a bridge, and then you attach your jails (using VNET) or VM's to the bridge. Since both of these virtualization abstractions include their own IP stack, that will work just fine. The bridge is effectively an ethernet switch, and when you connect the ethernet interface that is a member of the bridge to your real upstream ethernet switch, you have in fact connected a real ethernet switch to a virtual ethernet switch. Of course that's fine to do.

The problem is what happens next. People get confused because they fail to be able to visualize what's going on here. You can take a VNET jail or a VM and plug it into the virtual ethernet switch and then those hosts are on the virtual ethernet switch, which is in turn connected via your second ethernet interface to your real ethernet switch. Naturally that works. It doesn't violate the rule about a host having multiple interfaces on a single subnet, because no host HAS multiple interfaces on a single subnet. You have multiple hosts, each with a single interface on the subnet. It's just that some of the hosts are virtual.

 

[sretalla]

Maybe there was some confusion where I specified the bit about "-> Bridge (IP address of the host goes here if you're assigning one)"

Note in particular that my statement about an IP address for the bridge refers to the TrueNAS host (and not the VM/Jail... they get their own IP), which is also optional, since you might have an IP for the TrueNAS host on another NIC/bridge already and don't need another.

 

[agood]

Ahh, ye lil beauty! Last piece of the puzzle!

So for all in the same predicament:

My setup:
1 Truenas server: 2 SSDs on Raid 1 for OS, 3 SSDs for everything else
3 NICs: 1 inbuilt, 1-DualNIC

NICs:

• Inbuilt = system names it (in Networking/Interfaces): em0
• Dual NICs = system names them = igb0 and igb1

Config:

NICs:​

em0 = no DHCP, static IP for the host e.g. 192.168.1.100​

igb0, igb1 = no DHCP; LAGG0 (add new; lacp has to be supported by the switch you are using)​

create a bridge = bridge0 and add lagg0 as its member​

VMs and Jails:

1 VM: Ubuntu: NIC: set adapter: VirtIO (or Intel....) ; set "Nic to attach": bridge0
2 Jails:

Jail 1: vnet_default_interface: NONE; IPv4 Interface: vnet; IPv4 address: e.g. 192.168.1.5; IPv4 router: e.g. 192.168.1.1;​

NETWORK PROPERTIES: interfaces: vnet0:bridge0 (this is the secret!)​

 

[agood]

Jail 2 as above just different IPv4 address same (!) vnet0:bridge0 (vnet0 is the jails own network stack)

Works a treat! Thank you all for your relentless patience and efforts to educate Windows users!

 

[jgreco]

Works a treat! Thank you all for your relentless patience and efforts to educate Windows users!

No worries. If this stuff were easy, everyone would be doing it. In reality, it requires a lot of understanding of how relatively complicated things interact with each other. There are usually people here willing to help push/pull/carry you over the finish line as long as you're putting in some effort too.