Multiple VLANs and Asymmetric Routing (how to avoid this issue)

[planedrop]

I think this would best be explained with a sample scenario to make it make sense.

TrueNAS is on 2 subnets

• LAN = 10.10.10.0/24
• Management = 10.10.11.0/24

SMB shares need to be accessible on LAN, but WebGUI is disabled. However, a single IP on LAN needs to be able to connect to the web GUI for management, firewall rules allow said IP to connect to the management interface IP of the TrueNAS system. But TrueNAS replies to it on the LAN interface from it's LAN IP since it is connected in that subnet as well. This causes the WebGUI to refresh and crash constantly.

Any way to avoid this being an issue in TrueNAS? I've not had this issue with any other WebGUI management system, not ProxMox, not Xen Orchestra, etc.... Seems this is a somewhat common use case that can't be done with TrueNAS.

 

[IOSonic]

Friend, I don't know BSD, but you sound like you know what you're doing already, so perhaps I can nudge you onto the right track.

My guess is that your system is working as intended, and FreeBSD is trying to be helpful; your routing policies are telling your box to send anything destined to the LAN subnet out of the interface directly connected to it, as specified in your route table. This would be a default routing entry. It sounds like you want to override this behavior for a single IP address, on a single port. Doing it for all traffic destined to that IP (instead of just a single port) would introduce other problems, and you'd have another asymmetric routing issue to solve.

The way I would do this on Linux is by creating a new route table, and then add a routing entry that routes all traffic destined to your management box on the LAN through the Management subnet interface; I'd then create a policy to match the traffic that applies. The problem here, is that these tools do not exist on TrueNAS v12 & I have no idea how to do it without them.

If you do, I hope this sparks an idea for resolution, or even better...someone who has better insight into the nature of the problem with a full solution to fix it, will come along.

 

[Elliot Dierksen]

In system -> general, you can specify which IP addresses FreeNAS binds to the web server like below.

I have mine bound to a single IP out of the several that are assigned. If I click on the item, it allows me to choose another one or the 0.0.0.0 one meaning all interfaces.

If think that would mean you would need the default gateway to be on the same IP network as the web gui address.

 

[planedrop]

In system -> general, you can specify which IP addresses FreeNAS binds to the web server like below.
View attachment 48769
I have mine bound to a single IP out of the several that are assigned. If I click on the item, it allows me to choose another one or the 0.0.0.0 one meaning all interfaces.
View attachment 48772
If think that would mean you would need the default gateway to be on the same IP network as the web gui address.

I'll give this another shot, however if I recall correctly this doesn't solve the issue. The problem is that TrueNAS doesn't use "reply-to" functions on their interfaces, so if I connect from a IP that is on another interface IP, it will reply out that interface even if I connected to an entirely different one (cross subnet).

 

[planedrop]

Friend, I don't know BSD, but you sound like you know what you're doing already, so perhaps I can nudge you onto the right track.

My guess is that your system is working as intended, and FreeBSD is trying to be helpful; your routing policies are telling your box to send anything destined to the LAN subnet out of the interface directly connected to it, as specified in your route table. This would be a default routing entry. It sounds like you want to override this behavior for a single IP address, on a single port. Doing it for all traffic destined to that IP (instead of just a single port) would introduce other problems, and you'd have another asymmetric routing issue to solve.

The way I would do this on Linux is by creating a new route table, and then add a routing entry that routes all traffic destined to your management box on the LAN through the Management subnet interface; I'd then create a policy to match the traffic that applies. The problem here, is that these tools do not exist on TrueNAS v12 & I have no idea how to do it without them.

If you do, I hope this sparks an idea for resolution, or even better...someone who has better insight into the nature of the problem with a full solution to fix it, will come along.

I do think this is correct, it's doing what it should be doing, however I wish there was a way around this, other systems use functions to reply out the right interface (by other I mean literally every other webGUI I've ever used).

I'll see if I can manually adjust the routing table to fix this, sounds like one possible option.

The other option might just be for me to setup a NAT config on the firewall to hide the source of my IP from the LAN so that TrueNAS doesn't attempt to reply to it.

I'll see what I can come up with and update this post.

 

[planedrop]

In system -> general, you can specify which IP addresses FreeNAS binds to the web server like below.
View attachment 48769
I have mine bound to a single IP out of the several that are assigned. If I click on the item, it allows me to choose another one or the 0.0.0.0 one meaning all interfaces.
View attachment 48772
If think that would mean you would need the default gateway to be on the same IP network as the web gui address.

Yeah so just gave this another shot, Management net is the default gateway in global config, and disabled all but the Management net WebGUI bind, can use the WebGUI just fine but it still crashes every few seconds and has to be refreshed (only when connecting cross subnet, on the same subnet once I added back the LAN WebGUI bind it works fine for very long periods of time).

Definitely appears to be asymmetric routing, just wish ixsystems would do something about this since everyone else seems to have it figured out. Having your management interface be on the network the system is being accessed is just a bad idea security wise and is something that should always be avoided, and as an admin I shouldn't have to go through a lot of hoops to make it work right.

 

[The violonist]

Hello.

Same problem here... Please let me know if you found a workaround ?

Thanks,

 

[planedrop]

Hello.

Same problem here... Please let me know if you found a workaround ?

Thanks,

So far I have not found a solution to this, seems to just be a limitation within TrueNAS unlike most other platforms, it will always continue to respond out the interface within the subnet that the IP contacted it on. It's unfortunate since there are clear workarounds for this that other applications use, like reply-to functionality that makes sure the same interface (virtual or not) that the request came in is used for the reply.

 

[Elliot Dierksen]

A lot of this is just IP routing. Default gateway is exactly what it says: Where any traffic without a more specific route will be sent. You can add additional static routes that point out different interfaces. It does get more complicated if you only bind the GUI to one interface, but the FreeNAS has interfaces on multiple networks. I have multiple networks in my setup, and I bind the management GUI to the only network from which I manage the device. My internal DNS gives out different IP address from my NAS depending on the source IP of the requestor. That part is used for accessing storage, not management.

 

[planedrop]

A lot of this is just IP routing. Default gateway is exactly what it says: Where any traffic without a more specific route will be sent. You can add additional static routes that point out different interfaces. It does get more complicated if you only bind the GUI to one interface, but the FreeNAS has interfaces on multiple networks. I have multiple networks in my setup, and I bind the management GUI to the only network from which I manage the device. My internal DNS gives out different IP address from my NAS depending on the source IP of the requestor. That part is used for accessing storage, not management.

Right but in my case I'm trying to access the webGUI from a different subnet in which the management GUI is bound to, said subnet also exists on another interface on the NAS, so the NAS is trying to respond back out a interface that is NOT bound to the GUI. This is very atypical behavior in my experience, every other solution I use doesn't behave this way, ProxMox, XCP-ng, PFSense, etc.... They all respond back out the interface the initial request came in from, instead of going out the interface that said subnet is on.

I'm not sure I'm making perfect sense with this, so I'll try to bullet point it:

• PC on 192.168.10.0/24 network, lets say 192.168.10.5
• TrueNAS on 192.168.11.0/24 network AND 192.168.10.0/24
• WebGUI bound to 192.168.11.5, only want WebGUI accessible on this subnet but TrueNAS has a interface on the 10.0 network for SMB access, but PC needs to contact webGUI cross subnet as it's not dual homed
• PC tries to connect to 192.168.11.5 (firewall rules allow this in both directions)
• WebGUI connects initially but then starts attempting to reply to PC on the 192.168.10.0/24 subnet interface instead of the 192.168.11.0/24 which the request was initiated on
• By this, TrueNAS is NOT responding out the interface in which the request was made
• Default gateways and the like don't affect this since there is a route table entry for both networks on the TrueNAS

And while this isn't some huge deal, I just allow the WebGUI on my main LAN, this can be an issue for a lot of enterprise configs and isn't best practice from a netsec standpoint. I manage a pretty decent sized org and have a good number of servers with interfaces both in the Management VLAN and the main LAN, with firewall rules to allow one PC on the main LAN to cross into the Management VLAN, and every single product I use replies out the right interface other than TrueNAS.

 

[Patrick M. Hausen]

Right but in my case I'm trying to access the webGUI from a different subnet in which the management GUI is bound to, said subnet also exists on another interface on the NAS, so the NAS is trying to respond back out a interface that is NOT bound to the GUI.

This is how every Unix IP stack on this planet works. Period. Firewalls and other dedicated networking devices can enforce different policies by something called "policy routing", but for every regular host that is not a network device only the destination will be considered for a routing decision. And if the destination is on a directly connected network the packet will go via that interface.

This horse has been beaten to death multiple times on this forum already. Please believe me that this is how it is supposed to work and is not going to change - ever.

If you absolutely need something different, either attach your management station to the management network, or - whatever it is you use as the router between your networks - activate NAT when you go from a different network to the management interface.

Could a mod please close this thread? It's pointless.

Kind regards,
Patrick

 

[planedrop]

This is how every Unix IP stack on this planet works. Period. Firewalls and other dedicated networking devices can enforce different policies by something called "policy routing", but for every regular host that is not a network device only the destination will be considered for a routing decision. And if the destination is on a directly connected network the packet will go via that interface.

This horse has been beaten to death multiple times on this forum already. Please believe me that this is how it is supposed to work and is not going to change - ever.

If you absolutely need something different, either attach your management station to the management network, or - whatever it is you use as the router between your networks - activate NAT when you go from a different network to the management interface.

Could a mod please close this thread? It's pointless.

Kind regards,
Patrick

Absolutely no reason to be angry about someone posting and asking this kind of stuff, the point here isn't that it's not working as intended, I fully understand that is how most stacks work, the point here is asking if there are any roundabout options to make things work as I'd like.

And as I've mentioned before, other products have solutions in place for this already, they'll allow you to either bind the management GUI to a single interface and will ALWAYS reply management traffic out of that network, such as how Xen Orchestra does it, or they'll have reply-to functions to respond out the interface a packet came in on.

So no, this is not something that is "never going to change" since there already have been solid work arounds for it.

The common answer so many people in open source communities have of "you shouldn't want to do it that way" isn't helpful to the community and doesn't solve problems.

 

[planedrop]

A lot of this is just IP routing. Default gateway is exactly what it says: Where any traffic without a more specific route will be sent. You can add additional static routes that point out different interfaces. It does get more complicated if you only bind the GUI to one interface, but the FreeNAS has interfaces on multiple networks. I have multiple networks in my setup, and I bind the management GUI to the only network from which I manage the device. My internal DNS gives out different IP address from my NAS depending on the source IP of the requestor. That part is used for accessing storage, not management.

And actually to add to my last comment, obviously NAT is a way to solve this problem, but I first was trying to find out if there was another solution that I could use in TrueNAS directly.

Finally, if you feel this thread is pointless then maybe just don't waste your time commenting on it? Instead of coming in trying to get people that have legit questions to be closed out so they can't speak about an issue they are facing.

Just because some people aren't as versed in something as you doesn't make the questions not valid, and if the solution is obvious that can be provided instead of asking for a thread to be closed.

 

[Patrick M. Hausen]

Absolutely no reason to be angry about someone posting and asking this kind of stuff

I'm not angry, though I admit I was rather explicit.

the point here isn't that it's not working as intended

My point is that is working exactly as intended.

I fully understand that is how most stacks work, the point here is asking if there are any roundabout options to make things work as I'd like.

No, there are no workarounds around the fundamental concepts of IP routing.

And as I've mentioned before, other products have solutions in place for this already, they'll allow you to either bind the management GUI to a single interface and will ALWAYS reply management traffic out of that network, such as how Xen Orchestra does it, or they'll have reply-to functions to respond out the interface a packet came in on.

Other products probably have a dedicated management plane and a separate IP stack for that. In that case I agree your desired behavior is to be expected. The management plane does not know about the other interfaces and IP addresses used for services. Nice feature, no doubt. ESXi does that - the only other product I am familiar with.

But this is simply not the case with TrueNAS. TrueNAS being a dedicated storage appliance runs everything on a single IP stack and therefore is behaving exactly as is to be expected.

So no, this is not something that is "never going to change" since there already have been solid work arounds for it.

The "not going to change" statement I made implied as long as TrueNAS runs on a single IP stack for the host there is simply no way to change that with any reasonable amount of work. So my "not going to change" is a rather solid estimate. Whatever other products do is not applicable here, save re-implementing the entire concept of how TrueNAS does networking. You can theoretically fake all of what you want with custom firewall rules, but this is not going to work at scale in any generic way applicable to all users. That's why I am confident that my judgement does have some ground and - without being able to speak for iXsystems - they most probably have bigger fish to fry and more important features on their roadmap that a complete revamp of the entire networking stack.

Accept that it is what it is and use NAT on your switch/router/firewall where applicable.

Kind regards,
Patrick

 

[Elliot Dierksen]

Finally, if you feel this thread is pointless then maybe just don't waste your time commenting on it?

I am not trying to be condescending or mean. I am doing my best to explain what is going on. As someone who has been doing networking a long time, I can tell you this is not a problem that is not unique to FreeNAS. IP routing (mostly regardless of the type of device) works independently on each side of the conversation as each host makes its own decision about the path based on its local routing tables and policies. I implemented the changes in my local DNS server because of the exact problem you are describing with firewalls hating asymmetric routing.

This is a bit a kludge, but you could try adding a static route on your workstation for the management IP of the FreeNAS to the IP of the FreeNAS on your local network. For example let's say the FreeNAS had IP 192.168.11.5 (management with GUI bound only to this interface) and 192.168.10.5 on regular LAN. Your workstation is 192.168.10.99 so you would add a static route (persistent perhaps?) like this in windows: route add 192.168.11.5 mask 255.255.255.255 192.168.10.5
This way you would route around the firewall.

 

[planedrop]

I'm not angry, though I admit I was rather explicit.


My point is that is working exactly as intended.


No, there are no workarounds around the fundamental concepts of IP routing.


Other products probably have a dedicated management plane and a separate IP stack for that. In that case I agree your desired behavior is to be expected. The management plane does not know about the other interfaces and IP addresses used for services. Nice feature, no doubt. ESXi does that - the only other product I am familiar with.

But this is simply not the case with TrueNAS. TrueNAS being a dedicated storage appliance runs everything on a single IP stack and therefore is behaving exactly as is to be expected.


The "not going to change" statement I made implied as long as TrueNAS runs on a single IP stack for the host there is simply no way to change that with any reasonable amount of work. So my "not going to change" is a rather solid estimate. Whatever other products do is not applicable here, save re-implementing the entire concept of how TrueNAS does networking. You can theoretically fake all of what you want with custom firewall rules, but this is not going to work at scale in any generic way applicable to all users. That's why I am confident that my judgement does have some ground and - without being able to speak for iXsystems - they most probably have bigger fish to fry and more important features on their roadmap that a complete revamp of the entire networking stack.

Accept that it is what it is and use NAT on your switch/router/firewall where applicable.

Kind regards,
Patrick

Thank you, this makes a lot of sense now. My primary other experience with this has been with hypervisors like XCP-ng and ESXi, so makes sense as to why that's the "norm" I was thinking. I knew IP stacks base things on destination IP so I was just trying to figure out how some other solutions do it (such as ESXi), this clears that up with me a lot.

I'm totally with you now and I'll just do something via my firewall to get this going the way I want.

 

[planedrop]

I am not trying to be condescending or mean. I am doing my best to explain what is going on. As someone who has been doing networking a long time, I can tell you this is not a problem that is not unique to FreeNAS. IP routing (mostly regardless of the type of device) works independently on each side of the conversation as each host makes its own decision about the path based on its local routing tables and policies. I implemented the changes in my local DNS server because of the exact problem you are describing with firewalls hating asymmetric routing.

This is a bit a kludge, but you could try adding a static route on your workstation for the management IP of the FreeNAS to the IP of the FreeNAS on your local network. For example let's say the FreeNAS had IP 192.168.11.5 (management with GUI bound only to this interface) and 192.168.10.5 on regular LAN. Your workstation is 192.168.10.99 so you would add a static route (persistent perhaps?) like this in windows: route add 192.168.11.5 mask 255.255.255.255 192.168.10.5
This way you would route around the firewall.

yeah this makes total sense to me now, I'll either mess with static routes or adjust NAT on my firewall to do what I want. Appreciate all the input here!!

 

[Patrick M. Hausen]

My primary other experience with this has been with hypervisors like XCP-ng and ESXi

I understand. TrueNAS is still a storage appliance with a little bit of type 2 hypervisor thrown in for good measure. And while the hypervisor does work it still really shines when you use FreeBSD native containers aka "jails". These do have their own separate network stack.

 

[jgreco]

The management plane does not know about the other interfaces and IP addresses used for services. Nice feature, no doubt. ESXi does that - the only other product I am familiar with.

It's rare for me to find fault with things @Patrick M. Hausen posts, but I feel this is ... deceptive to proper understanding.

My primary other experience with this has been with hypervisors like XCP-ng and ESXi,

Both of which also work as we're describing. The thing that's tripping you up is that you're likely confusing what would be the hypervisor management (ESXi, etc) which is similar to the TrueNAS management interface, with the overall networking going on. With ESXi, there is *always* a vSwitch involved, which is a software defined switch. In TrueNAS, this is usually also the case, but it is a "bridge". And, unlike a vSwitch, it is possible for some kinds of things such as jails to share the NAS IP networking stack. This can be confusing to newcomers.

ESXi and XCP *absolutely* route stuff asymmetrically when you have multiple ethernet interfaces attached to the management portal. However, in most cases, adding a second, third, fourth, etc., network to the management plane on these systems is done for purposes of storage, vmotion, etc., and such traffic mostly remains "on" the local net, so the asymmetric routing is not noticeable. But it is certainly still there! Anyone who has ever had to deal with cross-site vmotion on a secondary network will be happy to confirm that... all sorts of hilarity ensues.

However, in *every* case, VM's added to ESXi are forced to have their own IP stacks, including their own choices of gateways, etc. This is also true for FreeNAS/TrueNAS VM's, or FreeNAS/TrueNAS jails with VNET defined.

FreeNAS/TrueNAS as network endpoints end up in the same general situation as the ESXi management portal, then. And that was probably the long way around to making that point.

 

[Patrick M. Hausen]

While the "separate network" is obvious for VMs and/or jails - do you know if other integrated "hyperconverged" products like Proxmox also apply that concept to sharing services? That's the one thing that always puzzles me in this recurring discussion. Are we talking VMs? Then TrueNAS is - although being a type 2 HV - no different from any other product out there. It's just the sharing that is lumped up with the management plane in a single stack.

Kind regards,
Patrick