Web UI performance issues related to vlans

[tstivers]

So I've been having horrible web UI delays. It takes 30+ seconds to load the login page. Most of the pages work fine once the login page is loaded. But the shell page takes 30+ seconds to load as well. And SSH exhibits similar symptoms. I get expected performance out of file transfers, however. This happens on both Core and Scale.

I think I've at least found the steps to reproduce the problem. Upon initial install, everything is fine. My network configuration is using a single interface, with two vlans on that interface and the interface itself having an IP address. The web UI is accessed and bound to the interface itself, and not a vlan. File transfers and everything else are handled over vlans. It isn't until I add my two vlans that the web UI performance, SSH, shell page, etc degrades. If I access the web UI through one of the vlans instead of the interface, the problem is not there. If the vlans are not configured on the interface, and it's just the interface itself with an IP, the problem is not there.

I've experienced this problem with a Terramaster F5-221, which I then replaced with a custom built NAS because the NICs in it were crap Realteks. But now I'm, again, experiencing this problem with my custom X10SDV-4C-TLN2F based build, which has dual Intel X552/X557-AT 10GbE NICs. The new system has 32GB ECC ram. The OS is installed to a cheap consumer SSD, but not to a USB thumb drive. I've replaced all involved network cables multiple times. I've tried different network ports. I've tried using a switch port directly instead of the 10GBase-T SFP+ module.

My current (untried as of yet, will report results later) solution is to use the other 10GbE port on my NAS as a dedicated interface for management/web UI, and put my vlans on a trunk only port. I suspect this will resolve my issues. But I also suspect this may be a bug or some other cryptic configuration issue.

 

[winnielinnie]

I wonder if we're noticing a similar and/or inter-related issue?

"Sluggish" GUI during saturated transfers and replications? Anyone else notice?

I've noticed since FreeNAS 11.x and even now with TrueNAS Core 12.0-U5.1, that when there is a lengthy replication in progress (or simply a high amount of data is being copied / written), the web GUI feels less "snappy" than normal. In some instances, the Dashboard might not load for a few...

www.truenas.com

 

[tstivers]

I wonder if we're noticing a similar and/or inter-related issue?

"Sluggish" GUI during saturated transfers and replications? Anyone else notice?

I've noticed since FreeNAS 11.x and even now with TrueNAS Core 12.0-U5.1, that when there is a lengthy replication in progress (or simply a high amount of data is being copied / written), the web GUI feels less "snappy" than normal. In some instances, the Dashboard might not load for a few...

www.truenas.com

In my case, there's no data pool configured. This happens after a fresh install and only configuring the network. My theory on how to solve it didn't work, though. It seems to be a routing issue.

 

[tstivers]

So the problem seems to be a matter of routing.

I have a desktop on 192.168.10.100/24
TrueNAS has the IPs 10.10.10.101/24 and 192.168.10.101/24
The web UI is listening on 10.10.10.101/24 and *not* 192.168.10.101/24.
When I try to access the web ui on 10.10.10.101/24 from 192.168.10.101/24 is when the delay happens.
If I set the web UI to listen on 192.168.10.101/24, I get no delay when accessing it from 192.168.10.100/24.

I'm guessing it's because the client sends the request to the router, expecting to get the response from the router. But, instead, it gets the response directly from the TrueNAS machine, bypassing the router, because they both share a vlan and so TrueNAS knows how to access it directly, even though the IP the request came in on isn't within that vlan. I'm not sure why it does eventually work if this is the case, but it does eventually figure it out after 30 seconds or so.

I'm not sure exactly what's happening. But my guess is it seems to be a matter of routing. I guess this is probably a non-standard configuration. I want the web UI on 10.10.10.101/24 so that it's always at the same IP address regardless of where I'm accessing it from. But I don't want samba being routed. Plus samba through a router is a pain, you lose the NAS in network places and it's a pain to make that work over a router. Thus why the NAS and desktop share a vlan, but the web UI isn't on an IP address in that vlan.

 

[tstivers]

*from 192.168.10.100/24 is when they delay happens

 

[tstivers]

I believe the phrase for what I'm experiencing is asymmetric routing.

 

[jgreco]

It's totally a standard configuration, but your implementation sounds broken.

Something needs to be getting the traffic from the PC to the NAS. Is the router forwarding traffic to 10.10.10.0/24 somewhere? Is the PC?

What you are experiencing is probably just a broken network, not asymmetric routing. Asymmetric routing works fine when done correctly.

 

[tstivers]

It's totally a standard configuration, but your implementation sounds broken.

Something needs to be getting the traffic from the PC to the NAS. Is the router forwarding traffic to 10.10.10.0/24 somewhere? Is the PC?

What you are experiencing is probably just a broken network, not asymmetric routing. Asymmetric routing works fine when done correctly.

So I have a 24 port switch and a router. The router is connected to the switch via fiber.

The router is configured to allow vlan10 (192.168.10.0/24) to access vlan1011 (10.10.10.0/24).
The client is connected to an access port on the switch (vlan10). The client's IP address is 192.168.10.100/24 with a default gateway of 192.168.10.1.
TrueNAS is connected to a trunk port with a default vlan of 1011 on the switch and the IP 10.10.10.101/24 with a default gateway of 10.10.10.1.
TrueNAS has an additional vlan configured, vlan 10, with the IP 192.168.10.101/24. This vlan is intended to be used only for samba and other services.
TrueNAS' web UI is configured to listen on 10.10.10.101/24 only.

The client, being in 192.168.10.0/24, does not know 10.10.10.101/24, so sends packets for that IP to the default gateway, 10.10.10.1. The default gateway knows 10.10.10.101/24 and has a firewall configured to allow 192.168.10.101/24 to access 10.10.10.101/24, and so happily forwards the packet to its destination.

TrueNAS receives the packet on 10.10.10.101/24 from the IP address 192.168.10.100/24, via the router. But the TrueNAS has a vlan configured in the subnet 192.168.10.0/24, and thus knows of 192.168.10.100/24. Therefore, TrueNAS does not need to send the reply through a gateway, but instead sends it directly to 192.168.10.100/24. This causes a delay.

The only thing I can think of is the router could be holding the connection open, but eventually times out after not receiving a reply. And then, somehow, the browser does some magic to pick up the response packets that are being sent directly to the client, and load the page.

I'm fairly certain asymmetric routing is what is happening here, though.

 

[jgreco]

So I have a 24 port switch and a router. The router is connected to the switch via fiber.

The router is configured to allow vlan10 (192.168.10.0/24) to access vlan1011 (10.10.10.0/24).
The client is connected to an access port on the switch (vlan10). The client's IP address is 192.168.10.100/24 with a default gateway of 192.168.10.1.
TrueNAS is connected to a trunk port with a default vlan of 1011 on the switch and the IP 10.10.10.101/24 with a default gateway of 10.10.10.1.
TrueNAS has an additional vlan configured, vlan 10, with the IP 192.168.10.101/24. This vlan is intended to be used only for samba and other services.
TrueNAS' web UI is configured to listen on 10.10.10.101/24 only.

The client, being in 192.168.10.0/24, does not know 10.10.10.101/24, so sends packets for that IP to the default gateway, 10.10.10.1. The default gateway knows 10.10.10.101/24 and has a firewall configured to allow 192.168.10.101/24 to access 10.10.10.101/24, and so happily forwards the packet to its destination.

TrueNAS receives the packet on 10.10.10.101/24 from the IP address 192.168.10.100/24, via the router. But the TrueNAS has a vlan configured in the subnet 192.168.10.0/24, and thus knows of 192.168.10.100/24. Therefore, TrueNAS does not need to send the reply through a gateway, but instead sends it directly to 192.168.10.100/24. This causes a delay.

That absolutely should NOT cause a delay; delivery via a directly connected interface direct into l2 switching is as good as it gets with Ethernet.

The only thing I can think of is the router could be holding the connection open, but eventually times out after not receiving a reply. And then, somehow, the browser does some magic to pick up the response packets that are being sent directly to the client, and load the page.

I'm fairly certain asymmetric routing is what is happening here, though.

So here's a quick test.

On your PC, create a static route for 10.10.10.101/32 pointed at 192.168.10.101.

Report back what happens. Everything should suddenly work fine, I'm guessing.

This then points at the "router", which for most end users, isn't actually a router but some sort of NAT gateway, This then becomes problematic because of poor interactions between NAT and firewall and implementation.

 

[tstivers]

It's definitely an asymmetric routing issue.

My router is an RB4011 and the switch is a CRS326.
RB4011 has a stateful firewall. The router has a rule in the forward chain to drop all invalid packets.

The client sends a syn, via the router.
TrueNAS receives the syn and responds with the ack, sending it directly to the client. The router never sees the ack.
The client sends the syn, ack via the router.
The router's firewall drops the syn, ack, because it's an invalid packet as far as the firewall is concerned, because it hasn't seen the ack.

If I disable the rule that drops all invalid packets in my forward chain, the problem goes away.

That still leaves the question of how the connection eventually manages to get established anyway.

I can think of a few ways to get around this. The first of which is a specific rule that prevents that packet being dropped as invalid. But maybe there's a better approach that I'm not aware of.

 

[jgreco]

Turn off stateful filtering, add a rule that allows the packets, or install a direct static route.

The connection is probably established once the state table drops the entry for the syn packet.

 

[tstivers]

I've come across a few threads now on this issue that didn't get resolved or ended up resolved with the OP not having a clue how they fixed it. Spent a good 3 weeks on it and bought a whole new NAS that I probably didn't need to because I thought the F5-221 couldn't use both NICs at the same time, when it was likely my firewall blocking what it saw as invalid packets the whole time. I'll leave this here, which is what explained the problem to me and helped me to understand what was going on, how to diagnose it, and what the solution was, so that people will hopefully not have to keep making threads about this and will know what the problem is and how to fix it.

Appreciate your help @jgreco

Adventures with Asymmetric Routing and Firewalls

Things that I have written

www.growse.com