Link Aggregation + VLAN Config

CinciTech

Dabbler
Joined
Nov 12, 2020
Messages
11
I am setting up a system which might be overkill (and I'm welcoming that feedback), but I'm also looking to learn how to set this up for a bigger project for a customer. I'm adept with virtualization and networking, but VLANs are new to me.

I have TrueNAS running on an R730XD, and I have (currently) three virtual host machines running ESXi 6.7 (two R410s and one old desktop Frankenstein that's been a very reliable 'server' for over 8 years now). I have a Mikrotik CSS326 smart switch connecting these (all 1G ports), with two cat6 cables run to each machine. I have put a couple of 14TB Exos drives in and I confirmed I can share them with NFS4.1 and map it as a datastore in ESXi (no link aggregation or VLAN).

My first test was to try to ping from ESXi (VMKernel IP: #.#.#.101) to TrueNAS and vice versa, using LACP link aggregation. I created a link aggregation "lagg1" using the two NICs and gave it an IP address (#.#.#.100/24). The ports from my CSS326 connected to the TrueNAS server confirm a trunk is detected and a mac address shows up in the partner column. I was unable to ping from ESXi to TrueNAS but I had no trouble pinging from TrueNAS shell to the ESXi host.

Next I want to create a VLAN to ensure all my VMotion traffic stays separated from normal application traffic. I created a VLAN called "vlan1", and selected lagg1 as the parent interface (because apparently you can't select multiple NICs like you do for the link aggregation). My chosen VLAN ID is 300; I set the VLAN to 300 in the VMotion port group in ESXi, and in my CSS326 for all the ports connect to the NICs between these servers.

Now I am no longer able to ping from the TrueNAS shell to my ESXi host. I'm 99% sure I've done the ESXi part right, and the CSS part is pretty straightforward, so my biggest unknown is TrueNAS. That said, how do I troubleshoot this, and what have I as a TrueNAS newbie likely misconfigured in TrueNAS? Is it necessary to use a link aggregate and VLAN or is one configurable to do both?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Can you draw a diagram of what you intend? I'm unable to picture it myself from your description.
 

CinciTech

Dabbler
Joined
Nov 12, 2020
Messages
11
Sorry for the delayed response, I've been swamped in projects and had to get Visio installed on my desktop.

1612376128748.png


To restate: I want my desktop to access the ESXi & TrueNAS management webUI, but not be able to access the NFS4.1 share. So management is on VLAN 100, and vMotion disk access should go on VLAN300. I'm pretty confident I've got it all working in my managed switch and virtual host. The part I'm fuzzy on is the TrueNAS configuration.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
What I chose to do for FreeNAS on VLAN is to handle the VLAN stuff purely on the switch (Unifi) and router (pfSense on ESXi), and not configure anything VLAN-related on FreeNAS (I couldn't get it to work quickly and figured that "my" approach was easier). So the switch port is tagged and receives a proper (i.e. VLAN) IP address via a DHCP reservation from pfSense. This also implicitly checks that things are working on layer 3, which could be a starting point for you. In general, have you checked the IP addresses and routing? I would assume so, but still ... :smile:
 

CinciTech

Dabbler
Joined
Nov 12, 2020
Messages
11
Thanks for the response. I am using separate IP subnets, 10.1.0.### and 10.3.0.### for the two VLANs. All are using the 255.255.255.0 subnet (I'm not saturated on IPs yet). I am able to ping freely without the VLAN in place; desktop can ping the management IPs & ESXi host can ping a single 10.3.0.### IP of the TrueNAS box.

I'm okay with leaving VLAN for the switches and not configuring it on the TrueNAS, if that's the 'best' way for it to work. After all, desktop computers typically don't need to be configured for the VLAN they're attached to. At the same time, computers typically don't get NIC redundancy/load balancing, so I thought maybe I needed to do more in the TrueNAS config.

I'll give that a shot and confirm if it's working as intended.
 
Joined
Dec 29, 2014
Messages
1,135
I'd like to throw in a few clarifications here. I think there is some terminology mixing that could be confusing, particularly around the word "tagged". I think what @ChrisRJ meant is the switch ports are configured as access ports (meaning only untagged frames are accepted) with those untagged frames ASSIGNED to a particular VLAN. Frames either have an 802.1q header or they don't. Each side of a physical link has its own configuration that determines where UNTAGGED frames go. If this configuration doesn't match, there will be problems on both sides with the untagged VLAN. VLAN's that both sides tag would work correctly.

The other item has to do with link aggregation which is a pet soap box issue for me. Aggregation does NOT mean bonding. Each side uses a hashing algorithm that determines which physical link in a bundle is utilized for a particular conversation. That means any one conversation can get no more than the bandwidth of one physical link. If you want to make sure that the storage network/VLAN has some dedicated bandwidth, it definitely makes sense to have a dedicated NIC/LAGG for that.

I hope that makes sense and doesn't confuse the issue more. As a non-cat person I always say that there is more than one way to skin a cat. The important thing is to have a dead cat at the end. :smile:
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
I'd like to throw in a few clarifications here. I think there is some terminology mixing that could be confusing, particularly around the word "tagged". I think what @ChrisRJ meant is the switch ports are configured as access ports (meaning only untagged frames are accepted) with those untagged frames ASSIGNED to a particular VLAN.
Thanks, that's exactly what I meant.
 

CinciTech

Dabbler
Joined
Nov 12, 2020
Messages
11
Reporting Elliot for not being a cat person. :wink:

Otherwise, this is the part that I'm learning and might be struggling to conceptualize, so I appreciate this. Or I will appreciate it once it fully makes sense to me. I do expect to be tagging all ports when I complete this: the vswitches are completely 100 or 300, (or later, 200 when I split management off of general network use, because if a computer gets hacked no one needs access to management interfaces).

Do I understand your second paragraph correctly that link aggregation is not going to give me 2gbps bandwidth from my ESXi server to TrueNAS, even if my virtual switch (ESXi) has NIC teaming and traffic shaping enabled? I don't actually know if ESXi talking to an NFS datastore is a single conversation or multiple. I presume multiple since there are multiple virtual hard disks on the datastore.
 
Last edited:
Joined
Dec 29, 2014
Messages
1,135
Reporting Elliot for not being a cat person.
Guilty as charged!
Do I understand your second paragraph correctly that link aggregation is not going to give me 2gbps bandwidth from my ESXi server to TrueNAS, even if my virtual switch (ESXi) has NIC teaming and traffic shaping enabled?
Yes, that is exactly what I am saying. If you had two ESXi servers, you could get an aggregate total bandwidth of 2Gbps. ESXi is the same thing. That is why the vSwitch definition looks like this.
1612473220987.png
 
Top