IMHO Very, Very strange TrueNas VLAN-handling. Not working as expected.

[Louis2]

Hello,

My / the problem is, that I do not manage to get a decent vlan separation and routing inside TrueNas.

Different traffic streams should be separated by VLAN's. For that reason, the NAS is connected with vlan-tagged trunks to my switches.

The NAS has two interfaces:
- the first nic (em0) one vlan being the management vlan for emergency access
- the second nic (mlxen0) has a combination of vlan's related to multiple purposes and also has the management vlan

The management VLAN should be tied to the TrueNas GUI and the other VLANS are related to TrueNas storage, VM's and Jails.

To make that work in the GUI we have:
a) the global network configuration
b) interfaces of type interface, vlan, bridge and lagg
Below my comments on the related settings / screens


Global network configuration
I feel very unconformable there:
- It seems to be the configuration page of the TrueNAS GUI (and partly the network layer). However all kind of strange things IMHO
* It should be related to a particular VLAN (in case you use VLAN's), ..... but it is not
* there are a default gateways, nameservers and domains, things which IMHO should be related to VLAN's and not to something global !!!

To have a global Default gateway is even dramatic, traffic should never be allowed to leave a VLAN !

VLAN set-up
- very wired things as well.
* a vlan is assigned to an interface (thats OK)
* a vlan is tagged or not (that is missing, just as PIVD)
* and VLAN addresses are definitely not related to an interface !!
* you should give vlan's a unique name being vlanxx, where it sounds logical that xx equals the vlanno, however .... problem is you have two links carrying the same vlan

Bridge set-up
- tja what is a bridge here, I think it is best compared with an internal unmanaged switch
- and again IP-address do not belong to an bridge
- and what if you tie two interfaces together using a bridge ...... then the IP-numbers should probably ..... be assigned to the bridge ... since you can not assign then to all related ^VLAN-ports/interfaces^


Trying to access the GUI
As written above I am trying to connect the GUI via the traffic trunk and for emergency purposes via the management interface port/trunk as well. Lots of problems there as well:

Scenario-1
VLAN-1 has IP-addresses defined and VLAN-2 as well. The Default gateway is related to VLAN-2
- that does not work for VLAN1 since the traffic is routed back via VLAN-1

Scenario-2
VLAN-1 is on interface-1 (vlan-1a) and on interface-2 (vlan-1b) as well, so I decide to define a bridge (bridge1)
- I assigned vlan-1a and vlan1b to that bridge
- and assigned an IP-address to the bridge
- after that I can assign the bridge IP to the GUI
However ...... I could not access the GUI via that IP at all !!

As example, at the moment I can only use the traffic-vlan (which have the default route) to reach the GUI. Not what I want and not secure.

To summarize:
- the GUI is IMHO very weird
- a couple of things do not work at all (hopefully I am wrong)


Louis

 

[Elliot Dierksen]

To have a global Default gateway is even dramatic, traffic should never be allowed to leave a VLAN !

I am going to disagree with you there. Side note: my day job is as a network engineer for a Cisco partner. That is how IP works. Look for the longest match to the destination in the IP route table. Generally if you haven't overridden anything, it will pick a connected interface if that matches the destination. Then it will look for the longest match in the IP route table. If it doesn't find anything more specific, it will take the default route if it exists (assuming it is not following the classful rules of IP).

As written above I am trying to connect the GUI via the traffic trunk and for emergency purposes via the management interface port/trunk as well. Lots of problems there as well:

If you restrict the GUI to a single IP address, that is the only IP on which FreeNAS will respond to HTTP/HTTPS requests. If your host has a route to the IP to which the FreeNAS GUI is bound, that should work.

I hope that does clarify things a bit.

 

[jgreco]

I think you've kinda misunderstood what vlans are.

Global network configuration
I feel very unconformable there:
- It seems to be the configuration page of the TrueNAS GUI (and partly the network layer). However all kind of strange things IMHO
* It should be related to a particular VLAN (in case you use VLAN's), ..... but it is not
* there are a default gateways, nameservers and domains, things which IMHO should be related to VLAN's and not to something global !!!

The way I read this, this just totally encapsulates a complete misunderstanding of UNIX networking and VLANs. To the UNIX system, a virtual LAN is really no different than a real ethernet LAN; they act the same as if you had hooked up a bunch of real ethernets.

A UNIX host has a single default gateway because what the hell else would make sense. If you connect em0:10.0.1.2 to the network 10.0.1.0/24 and default route points at 10.0.1.1, and then em1:10.0.2.2 to the network 10.0.2.0/24, how can you have a SECOND default route? What does that even mean? You expect the system to magically understand that "some" portion of your network is reachable via em1 while other parts are reachable via em0? No. That's what static route definitions are for; the system needs to unambiguously know what parts of the network are where.

When a libresolv client on the UNIX host is trying to look up "foo.my.domain", how would your hypothetical "nameservers and domains[...]related to VLAN's" even work? You seem to think that it should be able to have separate per-VLAN nameservers, but that makes no sense; what if "foo.my.domain" doesn't resolve on the VLAN1 nameserver but does on the VLAN2 nameserver? What if it resolves on both, but points to different resource records? This is untenable. Windows tries to do this, and causes various failures resulting in hacky fixes in things like OpenVPN (with its "block-outside-dns") because such a design is inherently broken jank-ass crap.

Global network configuration is global precisely BECAUSE it is GLOBAL; these are things that cannot be set on a per-interface basis because that makes no sense.

To have a global Default gateway is even dramatic, traffic should never be allowed to leave a VLAN !

Traffic should never be allowed to leave a VLAN? What the hell? The Internet is BUILT on virtual LANs, and the modern Internet wouldn't work without them. You can make a POLICY choice that you want your VLANs to be isolated, but that is a networking design choice, identical to connecting up an isolated switch to a dedicated ethernet on your FreeNAS host. That is something that you basically design at the switch and router level of your network.

You may wish to wander on over to this article, which is a different brand of a related misunderstanding that often pops up here:

https://www.truenas.com/community/threads/multiple-network-interfaces-on-a-single-subnet.20204/

I understand that networking is difficult, and you're talking to a seasoned network engineer who's been doing extremely complex networking with FreeBSD for more than a quarter of a century, including running multiple Internet ASN's. I am happy to try to straighten you out, since most of this is really simple at the FreeNAS level, but you're going to have to flush some of your preconceived notions about networking.

To start off with:

TrueNAS does not do packet forwarding ("routing") by default, but FreeBSD can be made to do this. It likely won't do what you expect, because most people think routers are magic, but they aren't.

TrueNAS *will* accept packets from any interface destined to its management IP address. This is a common UNIX behaviour, and is basically impossible to prevent without the addition of ipfw rules. See the linked article for a discussion of the true nature of UNIX's virtualized IP stack.

TrueNAS *will* also respond OUT any directly connected IP network for IP addresses reachable on that network; this means that if you have em0 and em1 as described above, with an external router that connects 10.0.1.0/24 and 10.0.2.0/24, and you send a packet addressed to 10.0.1.2 from client 10.0.2.3 to its default router at 10.0.2.1, which the router then delivers to the FreeNAS host, the FreeNAS host's return traffic will egress from em1:10.0.2.2, because it shows as a directly connected route, which is the most preferable.

This is not an error, a bug, or anything of the sort. It is how UNIX host networking works, and it is baked into every modern UNIX at a really low level.

You will need to come to grips with this kind of thing as you correct your understanding of how VLANs interact with the NAS. If this is unacceptable, then you probably cannot build your filesharing operations directly on FreeNAS. However, you CAN create a separate IP stack for a jail, which is independent of the host FreeNAS system's IP processing, so you should be able to connect a VNET jail to a LAN (virtual or physical) and have it be independent of the host.

 

[Louis2]

Hum,

IMHO VLANs do not have any existence right if traffic can flow towards another VLAN without an explicit defined rule. And IMHO normally the
transition from one VLAN to another VLAN should be the consequence of:
- a firewall/router (in my case pfSense) of
- an ACL in a managed switch (.e.g. a MAC-filter or passing a certain traffic type bypassing a firewall for performance reasons)

But apart from that I stay with my opinion that traffic should never ever change VLAN without an explicit defined rule!

Louis

 

[Louis2]

To put it in another way, TrueNas should IMHO act as "a set op application endpoints (the GUI is one of them)" and not as an IP-exchange point.

Louis

 

[jgreco]

But apart from that I stay with my opinion that traffic should never ever change VLAN without an explicit defined rule!

You're welcome to your opinion. Modern networking does not work that way, though, and it is not going to change to be the way you imagine it, because so much stuff in modern computing and networks rely on the abstracted UNIX networking paradigm that underpins all this. It's why things like SDN's and VPNs can work, etc.

If it helps you at all, consider that most small routers (and I mean real routers, not home NAT gateways, although come to think of it...) are actually UNIX or other RTOS boxes of some sort, running some sort of FreeBSD or Linux derivative, such as pfSense or Vyatta.

There are a few exceptions to all of this, of course, since it's software -- it's POSSIBLE (stupid, probably, but possible) to force certain behaviours through firewalling rules that actually change visibility and nexthop, or through alternate fib support, but these aren't things supported by FreeNAS. Just as with actual routing, you can potentially make FreeNAS do them for you, but it is probably a bad idea to do so.

If you really need separate networking environments, you're best off with vnet jails or virtual machines, both of which make truly separate networks trivial to do (but may introduce some new challenges for fileservice).

To put it in another way, TrueNas should IMHO act as "a set op application endpoints (the GUI is one of them)" and not as an IP-exchange point.

The underlying operating system, FreeBSD, is incapable, by design, of that, as are Linux and Solaris and most other modern operating systems.

My best suggestion is that you recognize your wish/opinion as being at odds with reality, and adjusting your understanding to work with the world in the way it actually exists. I know that seems harsh, but I am trying to give you advice which will actually help you design a functional modern network.

The implications of how modern networks and network stacks work have a significant impact on how things need to be designed, and paddling upstream against the current is at best frustrating, and at worst actively hazardous if you assume behaviours that are incorrect.

Now, to quote from a previous article,

I don't actually care to debate with you why you feel [...blabla...] I'm here to document the way it works on FreeBSD, Linux, Apple, and most other UNIX variants. I didn't make these design decisions but I am spending some time here to help you understand them. So please don't belabor the point.

As far as my involvement goes, you are welcome to ask questions that do not involve debate about how IYHO you think things should work. I am definitely happy to help you adjust your strategies or discuss risks of exposing interfaces/IP's/capabilities, or we can even talk about stuff like building ipfw rulesets that might reduce your exposure in your specific case. That's all topical and useful, and both myself and @Elliot Dierksen who are both experienced networking people have already joined this thread, so you can get some good suggestions if you wish. I am particularly paranoid of the sort that I am picking up resonances from you, but I've actually been solving these things in the real world too.

 

[Louis2]

OK,

Let me make a few points:
- my NetGear and Mikrotik switches do not allow to pass traffic from one VLAN to another one, apart from situations where want the switch to do so
- the same holds for pfSense
- I made a mistake in my first mail suggesting that a default route belongs to a vlan. That is of course not true. Lets say it belongs to a routetable used to route "equal type / same zone (!) traffic" into pipes from the related destinations. That probably more or equals a bridge here. In that particular situation it is intended to distribute traffic accross vlans. It is a completely different goal / situation !!

Traffic towards the GUI, the NAS, the VM, the jails needs to answered via the same vlan as the request was send. If not those responses will never reach the source of "the request". (at least it will lead to asymmetric routing). So routing tables should take care of that. And given the fact that the addresses of the source applications are of course unkown, that can IMHO only be done via a default rule pointing to the GW of the involved router (in my case pfSense).

Given this behavoir, assuming there is no solution, I can not even define two VLAN's, because if there is only one default route, traffic from one of the VLAN's will always return via the other VLAN, which does not work of course.

Of course I did split the network in multiple vlans for security reasons, so if TrueNas does not allow to use VLAN's the way they are intended (to separate traffic), than using TrueNAS for more than one VLAN is completely unacceptable from the security point of view !!

Louis

 

[Louis2]

@jgreco, made a point with

If this is unacceptable, then you probably cannot build your filesharing operations directly on FreeNAS. However, you CAN create a separate IP stack for a jail, which is independent of the host FreeNAS system's IP processing, so you should be able to connect a VNET jail to a LAN (virtual or physical) and have it be independent of the host.

So for a jail somethings seems to be possible.

What about VM's?

And of course I hope (I did not use jails and vm's yet) that it is possible to use /access TrueNas pools from the jails/vm's


Louis
PS if it is possilbe to separate vlan's via jails, it should be possible to run e.g. the GUI in a separate IP-environment as well

 

[Elliot Dierksen]

my NetGear and Mikrotik switches do not allow to pass traffic from one VLAN to another one, apart from situations where want the switch to do so

I think this is the key thing here. It sounds to me like you are referring to a VLAN at layer 2. Aka a virtual abstraction of a physically separate LAN/hub/switch. In the case you described, FreeNAS has layer 3 IP addresses on multiple different VLAN's. When you describe pfSense or whatever else, that is crossing between those at layer 3 (IP). Totally different thing. Your layer 3 routing can be done by a host if you enable it, a firewall, or a router (which could also be a layer 3 switch). Hopefully that makes the difference a little more clear.

 

[jgreco]

Let me make a few points:
- my NetGear and Mikrotik switches do not allow to pass traffic from one VLAN to another one, apart from situations where want the switch to do so

This has nothing to do with anything. Two switches also do not magically pass traffic from one to the other unless you connect them with a cable. Switches are not routers. Except that on many Mikrotik switches, you can actually load their Router OS on, and then you actually *can* pass traffic...

But none of this is similar to the situation with a UNIX host. As @Elliot Dierksen says (and hey thanks for saving me the trouble!), a host is a layer 3 object, and switches are layer 2.

- the same holds for pfSense

It certainly does NOT hold. pfSense will definitely move traffic between networks. That is literally its reason for being. It might not actually do so by default, because it is ALSO a firewall by default, but if you turn off the firewall, it will certainly let traffic route between vlans, and it will even forward packets between networks, because forwarding is enabled by default, where it isn't on FreeNAS. On FreeNAS, you do have the guarantee that the NAS is one of the IP endpoints, but you can still get interactions between networks and IP's as I described upthread.

- I made a mistake in my first mail suggesting that a default route belongs to a vlan. That is of course not true. Lets say it belongs to a routetable used to route "equal type / same zone (!) traffic" into pipes from the related destinations. That probably more or equals a bridge here. In that particular situation it is intended to distribute traffic accross vlans. It is a completely different goal / situation !!

I cannot make sense of this word salad.

Traffic towards the GUI, the NAS, the VM, the jails needs to answered via the same vlan as the request was send.

Only in your imagination. In many networks, traffic to a host ingresses via one interface and egresses via another. Remember, a "VLAN" is just a virtual LAN interface, which is not any different to the IP stack than a physical LAN interface. You don't see this in many simple networks because they only have a single ethernet connection into the network, but it certainly can and does happen.

If not those responses will never reach the source of "the request". (at least it will lead to asymmetric routing).

False, (true), in that order.

So routing tables should take care of that.

So, yes. If you desperately need some not-directly-connected traffic to egress in a certain direction, you MUST provide a way for the host to know that. Since most hosts do not support interior routing protocols like OSPF, that leaves you with static routing as your only option there.

And given the fact that the addresses of the source applications are of course unkown, that can IMHO only be done via a default rule pointing to the GW of the involved router (in my case pfSense).

Given this behavoir, assuming there is no solution, I can not even define two VLAN's, because if there is only one default route, traffic from one of the VLAN's will always return via the other VLAN, which does not work of course.

That doesn't follow. As @Elliot Dierksen mentioned above, directly connected interfaces are normally the most-preferred route to an IP network, and I explained that too:

TrueNAS *will* also respond OUT any directly connected IP network for IP addresses reachable on that network; this means that if you have em0 and em1 as described above, with an external router that connects 10.0.1.0/24 and 10.0.2.0/24, and you send a packet addressed to 10.0.1.2 from client 10.0.2.3 to its default router at 10.0.2.1, which the router then delivers to the FreeNAS host, the FreeNAS host's return traffic will egress from em1:10.0.2.2, because it shows as a directly connected route, which is the most preferable.

However, while this generally does what most people would expect, it does have subtle implications regarding traffic from different networks.

Of course I did split the network in multiple vlans for security reasons, so if TrueNas does not allow to use VLAN's the way they are intended (to separate traffic), than using TrueNAS for more than one VLAN is completely unacceptable from the security point of view !!

Well, your opinion of "the way they are intended" is different from both the real world implementation and how they are deployed in production throughout the world. As previously stated, I am not really that interested in debating IYHO how you think things should work when your opinion bears little relationship to an actual reality that neither you nor I can change. Some of the environments I run have hundreds of vlans on them, with FreeNAS being attached to several or even many of them. So I'm happy to help drag you to water, but you've gotta choose to drink on your own. I'm not invested enough here to force you to drink, sorry.

 

[Heracles]

- the same holds for pfSense

Well, my pfSense do it all day long. Actually, do you have an SG-1100 ? Know that this one is doing nothing BUT vlans and its only way to have trafic flowing is by routing from one vlan to another.

VLAN is a concept that exists at layer 2. IP and routing both happen at layer 3. As such, there is no link or relation between VLANs, IP and routing.

A VLAN is only a way to take a single Ethernet port group and to transform it in multiple groups. When you are down to local host like TrueNAS, you are down to taking a group of 1 port to make it looks like many groups of 1 ports.

What you are looking for is dual stacking. TrueNAS can do it with its jails : a jail will have a completely independent TCP / IP network stack. Bind it to a different VLAN and there, your TrueNAS will have its own IP stack (IP Address, Local network, routing table, default gw) while your jail will have a completely different and independent stack (another IP Address, another local network, another routing table, another default gw).

A local VM in TrueNAS would also have an independent IP stack.

You should read on ISO 7 layers network model to understand better about all of this...

 

[Elliot Dierksen]

You should read on ISO 7 layers network model to understand better about all of this...

Well I hate to speak for others, but but I would bet my retirement that @jgreco has read the "OSI" reference model more than once. I know I have...

 

[Heracles]

I was replying to @Louis2.... Never mentioned @jgreco and never thought that he would need to read that reference because he clearly already knows it very well...

 

[Elliot Dierksen]

Never mentioned @jgreco and never thought that he would need to read that reference

Sorry, my mistake. I got carried away there.

 

[Louis2]

VLAN is a concept that exists at layer 2. IP and routing both happen at layer 3. As such, there is no link or relation between VLANs, IP and routing.

Exactly that is why the GUI is in my opinion so confusing and not ok !

To answer your SG-1100 question, my pfSense system, is just like my TrueNas system based on a releatively small PC.


Related to the functionality, I would like to have:
- TrueNAS should behave as a "set of endpoints" and NOT as "an IP-transit point"
- and it should be guaranteed that traffic towards "the intended endpoint" can not reach "another endpoint"
- and that an endpoint can not inject traffic into a another than the intended level-2 VLAN, which would violate the security
(due to routing or intentionally by a hacker)

Those requirement apply IMHO not only to Jails and VM's but also to TrueNas application itself !
- You probably want to keep TrueNas management (the GUI) separated from the storage function (bluezone e.g. v.s. greenzone) and
- if you use the TrueNas system as a store for multiple applications (perhaps related to multiple security zones / level2 vlans), you probably want to have those storage parts strictly separated from each other.

I am not sure that TrueNas makes that possible. At least I still do not understand how to do that :(


For info, at this moment I am building a server intended to replace multiple physical PC's. One of the applications to run on the system will be TrueNas another a web-server etc. To archive this I could install some VM-system, but I was / am strongly considering to implement this using TrueNas as being both the central storage system and the host for other applications running as VM and/or within a Jail.


Louis

 

[Elliot Dierksen]

TrueNAS should behave as a "set of endpoints" and NOT as "an IP-transit point"
- and it should be guaranteed that traffic towards "the intended endpoint" can not reach "another endpoint"
- and that an endpoint can not inject traffic into a another than the intended level-2 VLAN, which would violate the security
(due to routing or intentionally by a hacker)

It doesn't unless you turned on routing in FreeNAS. I am going to pick some IP's for an example of you might infer this. FreeNAS has IP 192.168.1.15 and 192.168.2.15 on different interfaces. A client on the first network (192.168.1.30) makes a connection to 192.168.2.15. Some layer 3 router routes the packets from 192.168.1.30 to 192.168.2.15. FreeNAS will reply from that IP but to reply will come out the interface that has 192.168.1.15 because it has a directly connect interface. That is how routing is supposed to work, but it will break if the client on 192.168.1.30 sent the initial packets to 192.168.1.1 which is a firewall. The firewall would only see one side of the connection, so it would send a reset. Is that what you are talking about?

 

[Louis2]

It doesn't unless you turned on routing in FreeNAS. I am going to pick some IP's for an example of you might infer this. FreeNAS has IP 192.168.1.15 and 192.168.2.15 on different interfaces. A client on the first network (192.168.1.30) makes a connection to 192.168.2.15. Some layer 3 router routes the packets from 192.168.1.30 to 192.168.2.15. FreeNAS will reply from that IP but to reply will come out the interface that has 192.168.1.15 because it has a directly connect interface. That is how routing is supposed to work, but it will break if the client on 192.168.1.30 sent the initial packets to 192.168.1.1 which is a firewall. The firewall would only see one side of the connection, so it would send a reset. Is that what you are talking about?

Elliot, I do not know. I will set-up a second truenas system for more risky testing.

One of the things I would like to test is what will happen if I remove the default route entry in the "main-screen". Might it be that entrys in that screen
are "turning on the routing" !!??

Louis

 

[jgreco]

It doesn't unless you turned on routing in FreeNAS.

Well, that's demonstrably not true in some cases. The remainder of your post discusses one hypothetical case where it IS true, but is not a thorough examination.

Continuing with the example you've given, and which I suspect is what the poster is concerned about:

Let's say a bad guy is trying to break into the management interface of the NAS, which is on the firewalled net 192.168.2.0/24, and the bad guy has obtained purchase on 192.168.1.30 at root level. You might be tempted to think that the firewall at 192.168.1.1 would prevent connections to 192.168.2.15, which is true for the "normal" network configuration, but I can trivially blow by that on the client by adding "route add 192.168.2.15/32 192.168.1.15" which tells the client that there is a way to reach that server via one of the server's other interfaces. Suddenly 192.168.2.15 is NOT firewalled from 192.168.1.30, because the firewall has been taken out of the loop, and I can easily exploit whatever the latest CVE exploit against the management interface is.

Let me be very clear: Routing in FreeNAS, which doesn't exist in the GUI as a knob (and would have to be turned on within FreeBSD as a sysctl), can be OFF, and the attack I've suggested still works. The forwarding sysctl enables packet FORWARDING (ingress one interface, egress another) but the little "hack" we are discussing here is merely traffic addressed to one of the host's interfaces, and a naive sysadmin might not understand all the stuff I've been talking about in this thread and in the "multiple interfaces" post referenced above. This is very much a real threat.

In practice, the only realistic advice, and the thing I've been telling FreeNAS users since day 1, is to not attach untrusted clients on a directly reachable network connected to the NAS. This, and a few more obscure network hacks, are the reason why.

There are some remediations. I'll see if I can circle around to this as I talk with the OP, but I need everybody to be seeing what (I suspect) the OP is concerned about here. It's certainly valid, and I don't want to lose sight of that as I throw nails in the coffin of this guy's "but I think it should work this way" wishlist.

 

[Elliot Dierksen]

Well, that's demonstrably not true in some cases. The remainder of your post discusses one hypothetical case where it IS true, but is not a thorough examination.

I think you illustrated the point better than I did. It will respond for packets to/from its own IP addresses across interfaces, but it wouldn't route the packets to and from IP addresses which are not its own. That is the point the I was trying to make.

 

[jgreco]

Elliot, I do not know. I will set-up a second truenas system for more risky testing.

Please read my response to @Elliot Dierksen 's post. If you are going to set up a second system, that's actually great, because it will let me walk you through some of this and observe things firsthand.

One of the things I would like to test is what will happen if I remove the default route entry in the "main-screen". Might it be that entrys in that screen
are "turning on the routing" !!??

"routing" may be poorly defined in this context.

Packet forwarding between networks is something that routers do. FreeNAS won't do this unless you manually twiddle it on.

A UNIX host also has "routing" that selects a proper outbound interface for traffic based on destination address. This is always on.

This discussion seems to be more about interactions with the host networking stack that you didn't expect or anticipate, which you seem to be calling "routing", but really has more to do with the willingness of a UNIX host to ingress traffic on any interface and deal with it accordingly, while you would prefer a less generous approach. This is also always on.

So under that view of the word, no, there are no entries anywhere that "turn on the routing".