Network - Trunk with multiple vlans floods the network

[shaharsaporta]

Hello,

I'm using latest Truenas 12 on a standard PC, with a dual Intel 1GB NIC, and have issues with using multiple vlans on this machine.
For the discussion - I have designated one port for management, it is configured as an access port on the switch (HP 1920). The 2nd port is configured as a trunk port on the switch, and there are 3 vlans configured on it.
From the Truenas side my configuration is simple - for the physical port I have not configured anything (including IP address) except disabling hardware offload, added the vlans as interfaces (again - no IP address here also) which its parent interface is the physical NIC.
I have 2 jails, one is using vlan2 (for the example) and the other is using vlan3.
The 3rd vlan is for SMB shares and has an IP address, all relevant services have binding to this interface.
I'm assuming this configuration should work (or maybe I'm doing something wrong here).
When I turn on one jail (any of them) everything works fine. When I turn on the 2nd one, the network seem to be flooded, and it feels like the Truenas bridges between vlans in such a way that (for example) a Wifi device gets the wrong IP assignment in terms of wrong subnet on the wrong vlan, and other connected devices can't seem to communicate at all. When I shut the interface in question (the 'trunked' NIC) from the switch side, everything gets back to normal.
I have already tried using all vlans on the same NIC, including management, changed cables and the dual port NIC itself, but the problem seem to be from the Truenas box itself.
It is worth mentioning that other trunked ports on this switch (I have a Wifi AP with 5 vlans, a Hypervisor, and a link to another switch on the network) works fine, so unless it is something specific with this switch and Truenas (or FreeBSD) I can only focus on the Truenas. I have not tried this configuration with earlier versions of Truenas nor Freenas.

Any help appreciated.

Thanks,

Shahar

 

[Morris]

I feel you should take a close look at your switch configuration.

Good luck

 

[shaharsaporta]

I feel you should take a close look at your switch configuration.

Good luck

Thanks, what do you mean by "a close look"? other trunked ports on this switch (and on other switches in my home network) works fine.

 

[jgreco]

Well, I would imagine that @Morris means you should look to validate the configurations. It is certainly possible for a misconfigured TrueNAS host to create a bridging topology, so you should really walk through the entire thing, one step at a time, to find your error.

Don't try to mix tagged and untagged frames. This is a common mistake, and is almost always an error. If you are running multiple networks to a host, make use of VLAN's and run everything as VLAN's. This is less likely to be something that you can get wrong than a mixed environment.

Examine the results of "ifconfig -a" to see if the topology matches what you THINK it is supposed to be. Check the bridge members, etc.

Check the switch configuration to make sure it is coherent and matches up exactly with the TrueNAS side. LACP can be prickly.

Make sure that STP is enabled. This will help find obvious topology issues.

Use "netstat -I em0 1" on em0 (or whatever) and other physical interfaces to look for high traffic rates that could indicate a loop. You should also be able to run this against the vlan virtual devices. Look for high traffic rates. Figure out which direction, and why that traffic is there.

 

[shaharsaporta]

Well, I would imagine that @Morris means you should look to validate the configurations. It is certainly possible for a misconfigured TrueNAS host to create a bridging topology, so you should really walk through the entire thing, one step at a time, to find your error.

Don't try to mix tagged and untagged frames. This is a common mistake, and is almost always an error. If you are running multiple networks to a host, make use of VLAN's and run everything as VLAN's. This is less likely to be something that you can get wrong than a mixed environment.

Examine the results of "ifconfig -a" to see if the topology matches what you THINK it is supposed to be. Check the bridge members, etc.

Check the switch configuration to make sure it is coherent and matches up exactly with the TrueNAS side. LACP can be prickly.

Make sure that STP is enabled. This will help find obvious topology issues.

Use "netstat -I em0 1" on em0 (or whatever) and other physical interfaces to look for high traffic rates that could indicate a loop. You should also be able to run this against the vlan virtual devices. Look for high traffic rates. Figure out which direction, and why that traffic is there.

Thank you, I appreciate the elaboration. You mentioned bridge, can't I just use a physical interface as the parent interface for each vlan? I don't mix tagged and untagged traffic, and don't use LACP here.

 

[Morris]

Thank you, I appreciate the elaboration. You mentioned bridge, can't I just use a physical interface as the parent interface for each vlan? I don't mix tagged and untagged traffic, and don't use LACP here.

Of cause you could put a different VLAN on multiple physicals VLANs. If you don't need the bandwidth, then it's probably better to trunk as you are and find the issue so you don't find your self port starved in the future.

@jgreco nailed most of the common issues. Digging in a little deeper and it will depend on your switch if it matters, per VLAN spanning tree, native VLAN, spanning tree priority, spanning tree startup delay.

I agree with you it sounds like a loop and when matching different implementations of spanning tree together it's easy to wind up with a problem like this. Avoid spanning tree fast, at least till you get rid of the loop.

Good luck,

Morris

 

[jgreco]

Thank you, I appreciate the elaboration. You mentioned bridge, can't I just use a physical interface as the parent interface for each vlan? I don't mix tagged and untagged traffic, and don't use LACP here.

Yes, but you seem to be having some flooding issues, which is symptomatic of a bridging loop. This kind of thing doesn't just randomly happen. If it did, the Internet would just stop working. There's got to be a reason for the traffic, and nothing you've said is raising large obvious red flags to my admittedly caffeine-deprived brain.

 

[jgreco]

Of cause you could put a different VLAN on multiple physicals VLANs. If you don't need the bandwidth, then it's probably better to trunk as you are and find the issue so you don't find your self port starved in the future.

@jgreco nailed most of the common issues. Digging in a little deeper and it will depend on your switch if it matters, per VLAN spanning tree, native VLAN, spanning tree priority, spanning tree startup delay.

I agree with you it sounds like a loop and when matching different implementations of spanning tree together it's easy to wind up with a problem like this. Avoid spanning tree fast, at least till you get rid of the loop.

Good luck,

Morris

Plus all of this, which is actually more well-thought-out than my weedy listing of issues.

 

[dak180]

I have 2 jails, one is using vlan2 (for the example) and the other is using vlan3.

I would add: double check the way jails are configured; iocage has its own ideas about how to do networking and without the right parameters it likely is not doing what you want (or expect). If that is your issue (check via ifconfig -a in the main system) I can point you in the right direction since what to do was not terribly well documented the last I checked.

 

[jgreco]

And if you don't understand what you're looking at, dump the entire output of "ifconfig -a" here between CODE tags. I'm not familiar with the iocage stuff, but it sounds like between dak180, Morris, and I, you stand a better than average chance of getting some good interpretation of the results.