FreeNAS + Fail2ban for ssh block using hosts.allow

[onthax]

I used to use denyhosts but ran into issues with it after an update of freenas in the past.

looked into fail2ban which can do the same thing, hope other people find this useful.

Purpose: Block SSHD attempts after multiple failed attempts as the amount of attacks i get slows my connection down.

Tested and written for FreeNAS 11.1

Create a new jail called fail2ban

create new storage for the jail

Source: /var/log
Destination: /var/newlogmount
Readonly = yes
create directory = yes
mounted = yes

SSH to your base FreeNAS server. Use jls to show the jails:

Code:

root@server:path # jls
   JID  IP Address	  Hostname					  Path
	 5				  fail2ban					  /mnt/storage/jails/fail2ban

jexec the jid.

root@server:path # jexec 5

touch /etc/hosts.evil

root@fail2ban:/ # pkg install security/py-fail2ban

follow the prompts to install

root@fail2ban:/ # pkg install security/py-fail2ban
to update to the latest version (not sure why but the 2nd time i got a newer version)

Code:

cd /usr/local/etc/fail2ban
cp fail2ban.conf fail2ban.local
cp jail.conf jail.local

Modify jail.local

Append any ignore ranges to ignoreip

Code:

ignoreip = 127.0.0.1/8 ::1 192.168.0.0/24

SSHd section under jails, modify it to match

Note:
bantime = -1 means it never gets unblocked, you can set another value but i haven't tested test a great deal as i just manually clear out if i get stuck

Code:

[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
enabled = true
mode   = normal
port	= ssh
logpath = /var/newlogmount/auth.log
bantime = -1
backend = %(sshd_backend)s
action = hostsdeny

modify action.d/hostsdeny.conf

change the following

Code:

[Init]

# Option:  file
# Notes.:  hosts.deny file path.
# Values:  STR  Default:  /etc/hosts.deny
#
file = /etc/hosts.evil

Start the service

service fail2ban onestart

tail -f /etc/hosts.evil

Test from a remote server with failed logins and confirm it gets added and is all working.

Modify /etc/rc.conf. Add the line:

Code:

fail2ban_enable="YES"

Exit your jail
exit

Edit your /etc/hosts.allow file:

above

Code:

ALL : ALL : allow

put in:

Code:

# enable fail2ban
sshd : /mnt/PATHTOJAIL/etc/hosts.evil : deny

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

Confirm you can't access it from the remote host you were testing with before.

Edit /mnt/PATHTOJAIL/fail2ban/etc/hosts.evil and remove all entries.

Move your auth.log (to prevent fail2ban re-adding the entry on reboot)
mv /var/log/auth.log /var/log/auth.log.old

When you are happy with it all, mount the root FS:

mount -uw /

Edit /conf/base/etc/hosts.allow to make changes persistent across reboots.. Be careful what you do with USB mounted. You can brick your system.. This will allow your instruction to check hosts.evil to be re-applied at startup.

Above ALL : ALL : allow put in

Code:

# enable fail2ban
sshd : /mnt/PATHTOJAIL/etc/hosts.evil : deny

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

You should now be good to go.

Feedback appreciated.

 

[yozart]

Hello Ontax,

Thanks for your work and this tutorial, I will perform this on my freenas also.
Maybe a silly question but ... : You installed fail2ban in a jail. Is it protecting the main OS then ?
ssh is connecting to main freenas instance right? Then I was wondering how the jail stops unwanted attempts.

Regards

 

[onthax]

Hi Yozart,

Fail2ban is reading the log file for failed attempts then putting it in a deny file which is read via hosts.allow.

Similar to the way denyhosts worked.

Definitely test it after you have it installed and configured to confirm it's working as expected.

 

[yozart]

Onthax,

thanks for your answer, I tried your tutorial and can tell you it's perfect. I managed to ban my cellphone after a few bad attempts.

thank you very much for your time and your awesome work!

PS: Please note it needed a Jail restart after I modified the /etc/rc.conf

 

[Stranded Camel]

create new storage for the jail

Source: /var/log
Destination: /var/newlogmount
Readonly = yes
create directory = yes
mounted = yes

I want to try this with an iocage jail. I'm a bit confused about the settings above, since from what I understand fail2ban must access and modify some files on its FreeNAS host. So should the FreeNAS host's /var/log be mounted inside the jail to /var/newlogmount?

And in broader terms, is the following what I should be doing?...

Code:

iocage create -n "fail2ban" -r 11.1-RELEASE ip4_addr="vnet0|192.168.0.10/24" defaultrouter="192.168.0.1" vnet="on" allow_raw_sockets="1" boot="on"
iocage fstab -a fail2ban /var/log /var/newmountfile nullfs ro 0 0
iocage exec fail2ban service fail2ban start
iocage exec fail2ban sysrc "fail2ban_enable=YES"

I'm especially unsure about the sysrc "fail2ban_enable=YES" part -- I'm just guessing that the correct option is fail2ban_enable based on other tutorials, but I have no idea how to find out what the correct string is to get fail2ban to autostart when the jail runs.

 

[onthax]

Not sure what an iocage jail is

But what is happening here is the fail2ban application is reading from the host's auth.log and adding failed entries to the hosts.evil file which in then read back by hosts.allow which blocks the connection

This section

Modify /etc/rc.conf. Add the line:

Code:

fail2ban_enable="YES"

is to enable to application fail2ban to start with the jail.
so however you would change this to however you enable autostart on an application in your jail.

 

[yozart]

Onthax, FYI, I didn't had to launch 2 times the install of fail2ban. I got the latest version directly. Now, it doesn't mean it's valid foreveryone.
I am running the latest stable version of Freenas 11.

 

[JohnDigital]

Ive been tinkering with this for a while. I cant get anything to add into the hosts.evil file. My auth.log file is loaded with failed ssh attempts. The command fail2ban-regex /var/host_logs/auth.log /usr/local/etc/fail2ban/filter.d/sshd.conf returns this:

Code:

Running tests
=============

Use   failregex filter file : sshd, basedir: /usr/local/etc/fail2ban
Use		 maxlines : 1
Use	  datepattern : Default Detectors
Use		 log file : /var/host_logs/auth.log
Use		 encoding : UTF-8


Results
=======

Failregex: 959 total
|-  #) [# of hits] regular expression
|   6) [714] ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>(?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*$
|  10) [61] ^refused connect from \S+ \(<HOST>\)
|  15) [133] ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>(?: (?:port \d+|on \S+)){0,2}(?: ssh\d*)?(?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*$
|  18) [43] ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>(?: (?:port \d+|on \S+)){0,2}:\s*11:
|  19) [1] ^<F-NOFAIL>Connection <F-MLFFORGET>closed</F-MLFFORGET></F-NOFAIL> by <HOST>
|  20) [7] ^<F-MLFFORGET><F-NOFAIL>Accepted \w+</F-NOFAIL></F-MLFFORGET> for <F-USER>\S+</F-USER> from <HOST>(?:\s|$)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1949] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
`-

Lines: 1949 lines, 0 ignored, 959 matched, 990 missed
[processed in 0.39 sec]

Missed line(s): too many to print.  Use --print-all-missed to print all 990 lines

Im curious if anyone ever got this going in iocage, im running 11.2RC2?

 

[onthax]

I've been tinkering with this for a while. I can't get anything to add into the hosts.evil file. My auth.log file is loaded with failed ssh attempts. The command fail2ban-regex /var/host_logs/auth.log /usr/local/etc/fail2ban/filter.d/sshd.conf returns this:

Just migrated to 11.2 and got this working

here were the steps i followed, regex worked fine, but i think you are using the ssh filter instead of the hostsdeny, not sure how this works.

Code:

echo '{"pkgs":["security/py-fail2ban"]}' > /tmp/pkg.json
iocage create -n "fail2ban" -p /tmp/pkg.json -r 11.2-RELEASE ip4_addr="vnet0|<IP>/<MASK>" defaultrouter="<GATEWAY>" vnet="on" allow_raw_sockets="1" boot="on"
rm /tmp/pkg.json
iocage fstab -a fail2ban /var/log /mnt/rootlog nullfs rw 0 0
iocage exec fail2ban touch /etc/hosts.evil
iocage exec fail2ban cp /usr/local/etc/fail2ban/fail2ban.conf /usr/local/etc/fail2ban/fail2ban.local
iocage exec fail2ban cp /usr/local/etc/fail2ban/jail.conf /usr/local/etc/fail2ban/jail.local
iocage exec fail2ban sysrc fail2ban_enable=YES
iocage console fail2ban

vi /usr/local/etc/fail2ban/jail.local

Append any ignore ranges to ignoreip that you don't want to get blocked.

Code:

ignoreip = 127.0.0.1/8 ::1 192.168.0.0/24

SSHd section under jails, modify it to match below

Note:
bantime = -1 means it never gets unblocked, you can set another value but i haven't tested test a great deal as i just manually clear out if i get stuck

Code:

[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
enabled = true
mode   = normal
port    = ssh
logpath = /mnt/rootlog/auth.log
bantime = -1
backend = %(sshd_backend)s
action = hostsdeny

modify /usr/local/etc/fail2ban/action.d/hostsdeny.conf

change the following

Code:

[Init]

# Option:  file
# Notes.:  hosts.deny file path.
# Values:  STR  Default:  /etc/hosts.deny
#
file = /etc/hosts.evil

Start the service

service fail2ban onestart

tail -f /etc/hosts.evil


Test from a remote server with failed logins and confirm it gets added and is all working.

Modify /etc/rc.conf. Add the line:

Code:

fail2ban_enable="YES"

Code:

exit

Edit your /etc/hosts.allow file:

above
ALL : ALL : allow

put in:

Code:

# enable fail2ban
sshd : /mnt/PATHTOJAIL/root/etc/hosts.evil : deny

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

Confirm you can't access it from the remote host you were testing with before.

Edit /mnt/PATHTOJAIL/fail2ban/etc/hosts.evil and remove all entries.

When you are happy with it all, mount the root FS:

mount -uw /

Edit /conf/base/etc/hosts.allow to make changes persistent across reboots.. Be careful what you do with USB mounted. You can brick your system.. This will allow your instruction to check hosts.evil to be re-applied at startup.

Above ALL : ALL : allow put in

Code:

# enable fail2ban
sshd : /mnt/PATHTOJAIL/root/etc/hosts.evil : deny

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

let me know how you go.

 

[Volte]

@onthax, the iocage-plugin system makes this pretty repeatable now, so I took a lot of your instructions and distilled it into a plugin!

I got mine working last weekend, it's awesome. Thanks for putting together the guide.

https://github.com/TwilightCoders/iocage-plugin-fail2ban

 

[onthax]

Nice!

Glad it helped

 

[JohnDigital]

I did end up getting it all working. If memory serves I had a regex issue, i edited it manually to fit what the bulk of my errors were. It had something to do with the wording when certificate authentication is enabled, and was not passed. Because of certificate authentication I have set even one wrong login, wrong user, anyone trying to login as root, or attempted password login to ban forever. My hosts.evil file has thousands of blocked addresses in it now. Try once and your dead type of thing. I also set up pfblockerNG at the pfsense level to drop any IP not coming from the U.S. This is all working out very slick. Thank you very much!

 

[andrewjs18]

hitting this error:
Shared object "libdl.so.1" not found, required by "python3.6"

when running this command:
service fail2ban onestart

any thoughts? I'm running freenas 11.1-U7

 

[rvassar]

hitting this error:
Shared object "libdl.so.1" not found, required by "python3.6"

when running this command:
service fail2ban onestart

any thoughts? I'm running freenas 11.1-U7

This seems to be a recurring thread in FreeBSD 11.1 derived systems... Wish they'd find a way to handle it more gracefully.

Is this on the appliance itself or in a jail?

This thread may help:

https://www.ixsystems.com/community...n-jail-with-libdl-so-1-not-found-error.70391/

 

[andrewjs18]

This seems to be a recurring thread in FreeBSD 11.1 derived systems... Wish they'd find a way to handle it more gracefully.

Is this on the appliance itself or in a jail?

This thread may help:

https://www.ixsystems.com/community...n-jail-with-libdl-so-1-not-found-error.70391/

It's from within the jail I set up for fail2ban

 

[rvassar]

It's from within the jail I set up for fail2ban

Take a look at that link. It has a bit of a derived recipe that might help. Same library file, but different application. I'd probably snapshot the jail storage, just to make reverting easy.

 

[TidalWave]

I see that my ipfw list has different entries on an existing jail ruining important services versus the freenas host. I'm on 11.2 so is it still the case that if I want to use fail2ban I would have to create a new jail, or should I install in the jail that I want to protect, or should I install fail2ban on the host freenas?

 

[onthax]

I see that my ipfw list has different entries on an existing jail ruining important services versus the freenas host. I'm on 11.2 so is it still the case that if I want to use fail2ban I would have to create a new jail, or should I install in the jail that I want to protect, or should I install fail2ban on the host freenas?

So thinking about the process
The failed login entry gets written to the host auth.log, the freenas jail reads it in, then adds entries to the hosts.evil file.

You could use it to protect any of the jails, you just need to get each jail access to your hosts.evil file and update your hosts.allow file on each jail to read in the hosts.evil file.

haven't used ipfw with fail2ban, so not sure how that integration works, can you pull in an extra config file?

 

[relli10]

For the benefit of the community, I have just followed onthax's post 9 instructions on 11.3 and it works. However discovered the following problems:

1. Changes in /conf/base/etc/hosts.allow on the FreeNAS host do no survive a reboot.
This is a known bug in FreeNAS raised by Onthax - https://jira.ixsystems.com/browse/NAS-105603

2. Changing bantime=-1 to a value other then -1 in the /usr/local/etc/fail2ban/jail.local config file of the FreeNAS jail, does not work.
Solution is to change the actionunban= line in /usr/local/fail2ban/action.d/hostsdenny.config of the jail, to match the following extract:

Code:

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#

actionunban = IP=$(echo "<ip_value>" | /usr/bin/sed 's/\([][\.]\)/\\\1/g') && /usr/bin/sed -i "" "/^<daemon_list>: $IP$/d" <file>

I'm not very experienced with BSD/*NIX so don't really understand what the changes to the actionban line does, but from my googling it has something to do with the differences between the operation of the sed command in BSD and Linux(GNU). I copied the changes above from this links:
https://unix.stackexchange.com/questions/401905/bsd-sed-vs-gnu-sed-and-i
https://svnweb.freebsd.org/ports/he...n.d_hostsdeny.conf?view=markup&pathrev=475327

 

[Buttshill]

Hi,
I have been following your tutorial and I have one concern.
After /etc/rc.conf, you exit the fail2ban jail then require the /etc/hosts.allow to be edited.

My problem is that my /etc/hosts.allow is completely empty, this implies that something is not quite right.
Any assistance?

Thanks