I cannot enter TrueNas after about ^15 min^ when using IPFW for unkown reason

[Louis2]

Hello,

I am trying to add security to TrueNAS by adding IPFW rules.

Doing that, leads to a situation where, after some period (lets say 15 min), I can not access TrueNAS via the network any more. I still can access TrueNAS via other TrueNAS ip's or via the console.

There seems to be some security function active, which blocks the login under certain conditions after some time. That is probably a good thing. HJowever not for known source addresses. The protection function seems to work in conjunction with the FW. I know that there are thinks like "bruteforce" tables and pfctl, however I can not find any thing about such functionality in relation with TrueNAS.

Does any one know which functionality is responsible for this protection mechanism inside TrueNAS?
And how it can be configured?

Sincerely,

Louis

 

[Patrick M. Hausen]

I am trying to add security to TrueNAS by adding IPFW rules.

As you found out you did not add security but instead broke your system. Anything that is not present in the UI is unsupported and will likely break your system. Don't.

 

[jgreco]

The way you add security to TrueNAS is by putting a firewall in front of it. TrueNAS is not designed to do this, and because the firewall fails open, if you THINK you are running a firewall but it has failed to load, your pants are down around your ankles.

 

[Louis2]

jgreco,

I really do not understand your attitude. In 1970 computers where used as internet nodes. Nowadays computers are used as application platforms (servers / computers). As such internet node functionality does not belong on a server. That functionality is / should be provided by switches and routers.

The fact that truenas behaves as an internet node and not as an ip endpoint, for one or multiple vlans, is IMHO not at all ok. Not from the security standpoint and not from the routing standpoint.

That I have to try to work around that more than severe enough!

Louis

 

[Patrick M. Hausen]

TrueNAS can connect to multiple VLANs to serve applications on multiple VLANs. It is neither a router nor a firewall and should not be treated as such. First and foremost don't connect it to the public Internet, ever.

You can publish individual VMs and jails to the public Internet, because they provide sufficient isolation to get the risk down to a level commonly seen as acceptable. If it is acceptable for you, depends on your policy and yours alone.

But you don't have to work around anything. Simply don't connect your TrueNAS to untrusted networks.

 

[Louis2]

Patrick,

TrueNAS is primarily a NAS as secondairy host for other applications. I consider to use the computer for both functions. The routing and security problems I have are related to the NAS functionality.

TrueNAS management and TrueNAS data storage can be reached via multiple vlans, however from both security and routing standpoint that does work as it IMHO shoud (the system behaves like an internet node in state of a set of endpoints.

Traffic coming from the multiple vlans are mixed up on one big internal "ip traffic node" which is insecure
Traffic from the different vlans all return via the default gateway, where the traffic should be returned via the incoming vlan

So despite the beloved TrueNAS functionallity, I can not use TrueNAS for my purpose as long as these two issues are not solved.

Sincerely,

Louis

 

[Patrick M. Hausen]

TrueNAS management and TrueNAS data storage can be reached via multiple vlans

You can explicitly bind the management plane as well as the individual services to specific IP addresses.

Traffic from the different vlans all return via the default gateway, where the traffic should be returned via the incoming vlan

Can you provide a drawing of your network including IP addresses and describe a case of this happening? Because according to all I know about the FreeBSD network stack this is not at all how it works. And I am managing FreeBSD systems since 1994.

 

[Louis2]

Problem is that the internet layers where developed early in history, where the computers where at the same time internet nodes and security was no issue. As I wrote before, nowadays ^internet node functionality^ should be assigned to switches and routers, and computers should behave as endpoints. That is no problem as a computer or server is connected to one logical IP-interface, but it is an issue as the computer / server has multiple logical interfaces.

In my case I am trying to archive the following functionality on one computer
- NAS
- server with Apache / sftp / mail connected to internet
- other things running in vm's etc

TrueNAS has two functions ^management for NAS and VM's" and basic NAS
- the managment functionality should be tied to the management vlan
- the NAS functionality should be tied to the green vlan for secure storage
- the NAS could potentially be tied to the redzone vlan to share certain "data folders"

The "internet server will probably run in a vm connected to the redzone vlan
Probably I will also run a media server connected to the PC vlan

Of course the traffic streams related to all those vlans should stay strictly separated !!

All vlans are implemented using managed swithes and are terminated on my pfSense firewall.

Of course I could more or less archive this functionality by using a separate computer for each function and/or by running multiple TrueNas instances in jails, but that is not what I have in mind (every thing on one physical computer). And even if I did implement one computer/vm per function it would not allow me to separate TrueNas management from "TrueNas Data" and that solution will probably need more data-disks as well.

So the reason that I am testing with IPFW is not because I want a firewall, the reason is that I need one to fix security and routing problems !!


Louis

 

[Patrick M. Hausen]

Bind the management interface to the IP address of the management VLAN
Bind the NAS services to the IP address of the green VLAN
Don't assign any IP address to the red VLAN, use it only as a layer2 link to connect your VMs to - consider using a jail for Apache et. al. instead of a VM --> way less overhead and you can directly mount data folders into the jail without using a network sharing protocol like e.g. NFS

I run that configuration an I know it works as expected. No need to discuss what is or what isn't an "internet node" and what hosts historically did and should not. FreeBSD can act as a router but that is disabled in TrueNAS by default. Unless you enable it the traffic will stay separated.

You can set the default route of the TrueNAS for your management plane so you can install updates conveniently. Sharing services that should only work in one particular VLAN don't need a route. And the public services in VMs and jails are only bridged to the layer 2 VLAN and each have their own network stack, default gateway, etc.

 

[jgreco]

The problem is that when the Internet was invented, security wasn't really a thing.

I count among my good friends one of the original "414's", early computer teens who broke into poorly protected (actually unprotected) systems in the early '80's, inspiration for movies like WarGames and Ferris Bueller. I have more than just a little insight into this.

I've spent many years working the computer security angle, and one of the things is, if you are starting with a platform where security wasn't a fundamental design consideration from the start, you are creating a platform that is hard to defend. Putting some IPFW rules onto a platform that wasn't designed to be secure from the beginning doesn't magically make it secure. It's a false sense of security.

TrueNAS fully exposes all services on every direct-attached network. While this can be manually restricted, such as the Web UI being bound to a single network, the overall design is potentially subvertible if you can reach a directly attached network that the NAS is on. If you really need to run as a single host, it is possible to design a system using either VM's running on FreeNAS, or VM's (including FreeNAS) running on ESXi, that creates a proper networking design. However, this is not a beginner-level project, and I get the strong sense from your discussion so far that you don't have a deep grasp of what VLANs are, or how secure networks would be designed.

 

[Louis2]

Bind the management interface to the IP address of the management VLAN
Bind the NAS services to the IP address of the green VLAN

Patrick, the problem is that you can assign an IP-address to a certain function, but that does not make them vlans.
- they are using the same "bridge / unmanaged switch" and "the same routing table internally" and the same "default gateway". But of course I did use separate IP-ranges for my vlans and assigned IP's belonging to those ranges to the TrueNas gui/function they should access.

haring services that should only work in one particular VLAN don't need a route.

That will not work, since each vlan has its own subnet, however those subnet do transport packages coming from else where. And ... they will be returned via the default gateway / another vlan which is IMHO not at all OK.

When I wrote "VM" read that as "jail" here.

The problem is that when the Internet was invented, security wasn't really a thing.

Exactly that is the problem, as you understand I am trying to work around that. Which is nearly impossible !

Be aware I know a lot about security, but I am not a unix expert, even worse I expect "it" to behave as it should behave from the today point of view, which is ..... not always realistic.

And I know server mother boards do have a managment interface, which to a certain extend, solve the need for a management vlan on gui level. But I am using ordinairy computers not having that functionality

Also note that I did manage to create an IPFW rule set which does provide vlan / subnet separation, however more to test and some issues to solve. Among them this unexpected "lockout" behavior ....


Louis

 

[Patrick M. Hausen]

When I wrote "VM" read that as "jail" here.

Are these VNET jails? If not, that might be the cause of your weird/non-intuitive routing. You almost always want VNET in FreeBSD jails today.

As for the default gateway issue - packets are routed by destination address and by destination only. So if a client in VLAN 1 accesses (successfully) a service in VLAN 2 and if the TrueNAS system has got a connection and an IP address in VLAN 1, then and only then will the TrueNAS of course send the answer packet via VLAN 1 and not via VLAN 2. TrueNAS is behaving like any end node (host) with IP would. Destination address directly connected --> out that interface you go.

You have to rule out such a situation via proper network design. If all traffic is supposed to flow through your pfSense, you can of course always NAT the outbound traffic to all affected VLANs.

 

[Louis2]

No, pfSense is working perfectly. one gateway to the internet and one gateway for each vlan. All computers in the VLAN's are endpoints / there is absolutely no transit via the vlans.

Related to the jails etc. That is something of the (near) future. At this moment in time I have a computer intended to fulfill the described functionality and I have to decide which way to go. In fact I consider two options:
1) use TrueNas as NAS and as host for jails (e.g. apache en media server)
2) to use centos stream (8) as host and run TrueNas in one of the VM's

My original intention was solution 1) ... up to the moment I noticed that TrueNAS did not full fill my security and routing requirements.
So unless I can solve those, I will probably have to choose for solution 2)

Louis

 

[jgreco]

Patrick, the problem is that you can assign an IP-address to a certain function, but that does not make them vlans.

This may represent a misunderstanding of VLANs and the virtual networking stack of modern UNIX hosts on your part.

There has been a bad habit of some in the networking world to ascribe to VLANs some sort of magic qualities for security and separation. VLANs are nothing more than segregated broadcast domains within switchgear, such that the networks are manageable separately. If you have two basic ethernet switches, and hook them up to two separate ethernet ports on your server, you have two real-LAN's. If you number these "1" and "2", and combine them on the same switch, you now have VLAN's. The power of VLAN comes from the ability to then transport these off the switch silicon, still tagged, to other devices, whether switches or servers, bringing multiple networks to a device over a single trunk cable. VLAN's are really nothing more than a virtualized network abstraction.

Exactly that is the problem, as you understand I am trying to work around that. Which is nearly impossible !

No, it *is* impossible, at least to do it correctly, because modern networking designs have made choices that do not allow this to be safely implemented.

Be aware I know a lot about security, but I am not a unix expert,

I hope you can take it if I say I find these statements to be orthogonal.

even worse I expect "it" to behave as it should behave from the today point of view, which is ..... not always realistic.

But at least you're realistic about being unrealistic. Just looking for the upside here.

And I know server mother boards do have a managment interface, which to a certain extend, solve the need for a management vlan on gui level.

Well, sorta. You are basically talking about a secondary host that lives on the server, and which can therefore be independent from the networking of the primary host. This isn't the same thing as a management VLAN.

But I am using ordinairy computers not having that functionality

Also note that I did manage to create an IPFW rule set which does provide vlan / subnet separation,

I can make a pretty well-informed guess that you didn't actually succeed in doing this. Correctly, at least.

It is possible to do this, and I have done it, and in fact designed an automatic firewalling system for ISP systems that did this general thing back in the 1990's.

First off, let's discard the term "VLAN" which is just trashy confusing noise. Let's instead start with two basic networks on two separate switches, attached to em0 and em1 on a FreeBSD host.

Further, let's define network 1 to be 192.168.1.10/24 and network 2 to be 172.16.1.10/24 on the FreeBSD host, with .1 for last octet of the gateway, and we will assume for discussion that these get routed.

The FreeBSD host will have a directly connected route for both 192.168.1.0/24 and 172.16.1.0/24 out the respective em0/em1 interfaces.

The first common misconception is that a host at 192.168.1.100 that sends a packet to 172.16.1.10 will see its traffic transit em1. Indeed, 192.168.1.100, not knowing any better, will send that to the default gateway at 192.168.1.1, which then routes it over to 172.16.1.1, which then transmits it on network 2 to 172.16.1.10. So ingress does indeed occur on em1. However, the FreeBSD host, now trying to send an answer to 192.168.1.100, sees 192.168.1.0/24 as a connected route, and sends the egress traffic out em0. This comes as quite a shock to many neophytes, who naively assume that the IP address for an interface somehow forces all traffic to go in/out via that interface.

You can work around this by manually forcing traffic to the gateway, such as

ipfw add X fwd 192.168.1.1 ip from 192.168.1.10 to any out
ipfw add X+1 fwd 172.16.1.1 ip from 172.16.1.10 to any out

and that will seem to work, but it will have the undesired consequence of also forwarding all OTHER traffic bound for 172.16.1.0/24 to the gateway, forcing the gateway to hairpin. So a slightly more correct construct could be

ipfw add X pass ip from 192.168.1.10 to 192.168.1.0/24 out via em0
ipfw add X+1 fwd 192.168.1.1 ip from 192.168.1.10 to any out
ipfw add X+2 pass ip from 172.16.1.10 to 172.16.1.0/24 out via em1
ipfw add X+3 fwd 172.16.1.1 ip from 172.16.1.10 to any out

which allows the firewall to directly reach connected hosts on em0/em1. I leave as an exercise for the reader the reason for inclusion/exclusion of the via clause.

Ingress traffic is a different problem. In general, we expect routing to handle direction of traffic to a host, but when you have an attacher who has gained purchase on a directly connected network, this is no longer the case. For example, if I am on 192.168.1.100, and I do

# route add 172.16.1.10 192.168.1.10
# ping 172.16.1.10

I am likely to get a response, even if the upstream gateways strictly forbid such traffic. Writing proper ingress rules is complicated. You can try the trite version of

ipfw add X pass ip from 192.168.1.0/24 to 192.168.1.10 in via em0
ipfw add X+1 deny ip from any to 192.168.1.10 in via em0
ipfw add X+2 pass ip from 172.16.1.0/24 to 172.16.1.10 in via em1
ipfw add X+3 deny ip from any to 172.16.1.10 in via em1

but this breaks IP connectivity coming in from off-net. You can try exclusionary rules for directly attached networks,

ipfw add X deny ip from any to 192.168.1.10 in via em1
ipfw add X+1 deny ip from any to 172.16.1.10 in via em0

which is adequate to eliminate the spoofing ping example above, but is insufficient for other types of attacks. It also gets more complicated as the number of interfaces increases.

And of course all of this fails in the face of the more complicated topics of multicast and all that, because lots of traffic to and from the NAS isn't based on the NAS interface IP's. What I've discussed above is very basic and hardly comprehensive.

And then these things can fail again due to changes in the interface definitions. You swap out a network card, forgetting that you've got "em0" littered throughout your firewall, and all of a sudden, defenses vanish. You can work around that by abstracting out interface assignment detection in your firewall script. This is not trivial, I speak from experience.

And then these things can fail AGAIN, because FreeNAS is not designed to have a firewall. If the firewall fails to run for any reason, you end up with your pants around your feet, and you don't even know it. This can be very subtle. Our network monitoring here actually detects some failures of this sort, because we have a remotely detectable "network semaphore" that indicates a firewall is running, but if you haven't built such infrastructure, then I'd say you are in very hazardous waters relying on something to do something it wasn't designed for.

That's why the proper solution for this is to run FreeNAS in the manner in which it was designed, and then make sure you're controlling access upstream with a proper firewall device. This generally simplifies things to a great degree.

however more to test and some issues to solve. Among them this unexpected "lockout" behavior ....

 

[Louis2]

A short replay for the moment:
- I am using trunks containing multiple vlans
- I am using a static route for each vlan
- I defined extra routing tables one per vlan
- I did define the remote pfSense gateway as default gateway on the per vlan routing table
- The first action when a packet arrives is to identify the vlan based on incoming interface and subnet IP's and set the routing table using setfib
- Then I determ the actual processing phase of the arrived package
- To send it to the vlan corresponding input filter and then to the application
- Received back from the application I send the package through the out-put filter and then pass it back to the vlan interface

But it needs more testing and as said I need to solve some strange effects as well


Louis

 

[jgreco]

A short replay for the moment:
- I am using trunks containing multiple vlans
- I am using a static route for each vlan
- I defined extra routing tables one per vlan
- I did define the remote pfSense gateway as default gateway on the per vlan routing table
- The first action when a packet arrives is to identify the vlan based on incoming interface and subnet IP's and set the routing table using setfib
- Then I determ the actual processing phase of the arrived package
- To send it to the vlan corresponding input filter and then to the application
- Received back from the application I send the package through the out-put filter and then pass it back to the vlan interface

Basically, then, you are doing a bunch of pseudogarbage, and it doesn't work.

Let me summarize:

 

[Louis2]

jgrego,

Know that I appreciate your explanation above. That helps to improve my understanding. It will perhaps help me to sort out issues later on. However note that:
- I did opt for a very clean ipfw rule model strictly based on the idea ^cto reate / use vlan's^ and to define vlan related rules
- with as consequence that the per vlan rule sets do not need any rule related to the other vlans or subnets !!!!!
- I know that it does not handle all kind of special traffic especially not ipv6 related, given the complicated addressing modes0
(that multicast etc is not working is probably not an issue at all, its just a pity that I have to emulate vlans, with nasty consequences like this)
- that it seems to work ..... further testing needed especially ipv6 ...

Reading your explanation I especially triggered on two things:
- the gateways are on remote on pfSense. Does that in your opinion imply that every packet coming from an internal source related to that subnet is
automatically forward to the related static route? (gateway ip using the correct directly connected interface)
- the second thing which triggered me is the combination of the following two lines
# ipfw add X pass ip from 192.168.1.10 to 192.168.1.0/24 out via em0
# ipfw add X+1 fwd 192.168.1.1 ip from 192.168.1.10 to any out
They seems to do the same. What is the difference and why both !? And why in that order? And why if routing to the gateway is automatic (first question).


That I did not react on your explanation in the first place, is because my first issue to solve is "the ^15 min block^ to understand what is behind that issue.


Louis