Alert - Yesterday's SSH Login Failures

[SoonerLater]

My system (12.0-U8.1) is configured to send an email once a day with alerts. One of the included alerts (System \ Alerts) is for "Yesterday's SSH Login Failures." However, the email doesn't include all of the login failures. Where can I see all of the SSH login failures?

TrueNAS @ charlotte.local

New alerts:
* 69 SSH login failures:
Dec 3 18:39:58 charlotte 1 2022-12-03T18:39:58.213547-06:00 charlotte.local sshd 8186 - - Invalid user telecomadmin from 152.89.196.123 port 50994
Dec 3 18:39:58 charlotte 1 2022-12-03T18:39:58.332494-06:00 charlotte.local sshd 8186 - - Failed password for invalid user telecomadmin from 152.89.196.123 port 50994 ssh2
... 65 more ...
Dec 3 22:26:33 charlotte 1 2022-12-03T22:26:33.905856-06:00 charlotte.local sshd 11360 - - Failed password for root from 152.89.196.220 port 31424 ssh2
Dec 3 23:04:44 charlotte 1 2022-12-03T23:04:44.339724-06:00 charlotte.local sshd 11823 - - Failed password for root from 152.89.196.220 port 52502 ssh2


Current alerts:
* 69 SSH login failures:
Dec 3 18:39:58 charlotte 1 2022-12-03T18:39:58.213547-06:00 charlotte.local sshd 8186 - - Invalid user telecomadmin from 152.89.196.123 port 50994
Dec 3 18:39:58 charlotte 1 2022-12-03T18:39:58.332494-06:00 charlotte.local sshd 8186 - - Failed password for invalid user telecomadmin from 152.89.196.123 port 50994 ssh2
... 65 more ...
Dec 3 22:26:33 charlotte 1 2022-12-03T22:26:33.905856-06:00 charlotte.local sshd 11360 - - Failed password for root from 152.89.196.220 port 31424 ssh2
Dec 3 23:04:44 charlotte 1 2022-12-03T23:04:44.339724-06:00 charlotte.local sshd 11823 - - Failed password for root from 152.89.196.220 port 52502 ssh2

Clearly there are rogues trying to gain access to my system, which is why I've enabled 2FA for web admin and SSH access. Nevertheless, I would like to read the alerts in full.

Also... aside from enabling 2FA, is there anything else that I should do to protect my system in general? Is there a "security best practices" article for TrueNAS out there somewhere?

 

[ChrisRJ]

The best practice is to not expose TrueNAS directly to the Internet. You should use a VPN as an additional protection layer.

 

[Davvo]

You need to build a (fire)wall!

152.89.196.220 | Starcrecium Limited | AbuseIPDB

www.abuseipdb.com

Edit: look into /var/log/auth.log

 

[garm]

A really good dry suit will keep you somewhat safe in a septic tank, but why swim in the filth in the first place? just stick to the pool...

 

[SoonerLater]

The best practice is to not expose TrueNAS directly to the Internet. You should use a VPN as an additional protection layer.

Ok... stupid question... how can I access files stored on my TrueNAS server (that sits at home) when I'm not home, if not via SFTP? Sometimes I'm away from home and unexpectedly need a file stored on it. How do I create a VPN to access files on my TrueNAS when I'm away from home? My router doesn't have VPN capability (I think) and neither does TrueNAS (I... think?).

Sorry to be so slow....

 

[danb35]

how can I access files stored on my TrueNAS server (that sits at home) when I'm not home, if not via SFTP?

Nextcloud is a popular option. I've written a script to perform what I think is a pretty comprehensive installation.

For SSH, especially if you're exposing it to the Internet (not recommended as noted above), you should only be using public-key authentication.

How do I create a VPN to access files on my TrueNAS when I'm away from home?

Contrary to your belief, TrueNAS does have the ability to act as an OpenVPN server, but I prefer to run it on the router. Your router may support that directly, or there may be alternative firmware available that can do it. Or you could replace your router with one that does, like OPNsense or pfSense--I'm using OPNsense.

 

[garm]

I’m running a reverse proxy in DMZ that handles traffic to jails. Nothing, not even other clients at home, get to access TrueNAS over network.

I’m looking into replacing my own reverse proxy now with Cloudflare tunnels.

Thread '[Guide] Push the jail to the internet without public IP with Cloudflare tunnel (TrueNAS core / No VM)'
https://www.truenas.com/community/t...-cloudflare-tunnel-truenas-core-no-vm.105566/

Or I might just stick to wireguard and run everything in the clean waters of my own network.

O and all my “file” handling is done with Nextcloud, makes backups way easier.

 

[SoonerLater]

Nextcloud is a popular option. I've written a script to perform what I think is a pretty comprehensive installation.

Please, would you share that?

 

[danb35]

Please, would you share that?

It's in the Resources section, along with its discussion thread, but the script itself is at:

GitHub - danb35/freenas-iocage-nextcloud: Script to create an iocage jail on FreeNAS for the latest Nextcloud 28 release, including Caddy, MariaDB or PostgreSQL, and Let's Encrypt

Script to create an iocage jail on FreeNAS for the latest Nextcloud 28 release, including Caddy, MariaDB or PostgreSQL, and Let's Encrypt - danb35/freenas-iocage-nextcloud

github.com

 

[SoonerLater]

It's in the Resources section, along with its discussion thread, but the script itself is at:

GitHub - danb35/freenas-iocage-nextcloud: Script to create an iocage jail on FreeNAS for the latest Nextcloud 28 release, including Caddy, MariaDB or PostgreSQL, and Let's Encrypt

Script to create an iocage jail on FreeNAS for the latest Nextcloud 28 release, including Caddy, MariaDB or PostgreSQL, and Let's Encrypt - danb35/freenas-iocage-nextcloud

github.com

Thank you!

 

[nathaliealisson]

I’m running a reverse proxy in DMZ that handles traffic to jails. Nothing, not even other clients at home, get to access TrueNAS over network.

I’m looking into replacing my own reverse proxy now with Cloudflare tunnels.

Thread '[Guide] Push the jail to the internet without public IP with Cloudflare tunnel (TrueNAS core / No VM)'
https://www.truenas.com/community/t...-cloudflare-tunnel-truenas-core-no-vm.105566/spider solitaire 2 suit

Or I might just stick to wireguard and run everything in the clean waters of my own network.

O and all my “file” handling is done with Nextcloud, makes backups way easier.

Thanks for your tutorial. Very helpful

 

[James S]

It seems like others I'm having a large number of phishing / brute force (?) attacks. I'm running TrueNas 12.x on a LAN with a single non-standard SSH port open. I use a backup program (Syncovery) which runs SFTP (with password protected SSH keys) over a port-forward through a router for backup.

This arrangement worked great for 7+ years. When running FreeNas occassionally (once a year or less?) I was attacked (but nothing like this volume) and then would change or close the SSH port for a couple of days. All then went quiet. Under a similar setup with TrueNas all is not quiet - I'm getting 200 - 12,000 attempts a day. Naively, perhaps, they do not look sophisticated (just attempts with usernames) but the odd one attempts to use http 1.1. I guess this is not about Truenas but more about a combination of my setup and more active "evil bots"

I don't have a technical background so hoping some smart folks here can help answer a few questions:
(1) Do these attempts consume much resource? From what I've read this is not a big deal?
(2) I guess ideally I should implement PF Sense but I've not had time to research this yet. Should I panic and start on this project yesterday?
(3) I don't really understand how to read the logs properly. Here are some examples:

Jan 11 17:14:51 freenas 1 2023-01-11T17:14:51.904796+08:00 freenas.local sshd 15048 - - Invalid user t from 137.184.126.78 port 42512
Jan 11 17:14:52 freenas 1 2023-01-11T17:14:52.055818+08:00 freenas.local sshd 15048 - - Disconnected from invalid user t 137.184.126.78 port 42512 [preauth]
... 18 more ...
Jan 11 18:47:07 freenas 1 2023-01-11T18:47:07.660192+08:00 freenas.local sshd 17046 - - error: kex_exchange_identification: client sent invalid protocol identifier "GET / HTTP/1.1"

* 732 SSH login failures:
Jan 10 00:01:16 freenas 1 2023-01-10T00:01:16.306796+08:00 freenas.local sshd 70497 - - Invalid user xiaobao from 211.245.31.15 port 42336
Jan 10 00:01:16 freenas 1 2023-01-10T00:01:16.419897+08:00 freenas.local sshd 70497 - - Disconnected from invalid user xiaobao 211.245.31.15 port 42336 [preauth]
... 728 more ...

Why are they reporting high port numbers (e.g., 42512, 42336)? My port foward is way lower XX (but not 22). Is this the evil bot in-bound or TrueNas reply?

Any thoughts will be really appreciated!

 

[ChrisRJ]

Why are they reporting high port numbers (e.g., 42512, 42336)? My port foward is way lower XX (but not 22).

These port number are the originating ports. Your forwarding goes to the target ports on your system.

 

[Whattteva]

GET HTTP seems like an attack attempt targeted for web servers, which obviously your SSH server does not quite understand. The others just seem to be attempts using different usernames.

As someone above me said. The high port numbers are typical from connecting clients. It's not the port that your sshd is bound to.

 

[James S]

Thanks for both the above replies - appreciated. I'm still trying to grasp what is happening here . . .

These port number are the originating ports.

How is the evil bot (EB) even coming in on 5x,xxx?

It's not the port that your sshd is bound to.

So if only one port is open (SSH on XX) why is the machine even talking to EB?
Sorry - these are undoubtedly naive networking questions.

The critical question is are these using NAS resource in a significant way?

 

[Whattteva]

Thanks for both the above replies - appreciated. I'm still trying to grasp what is happening here . . .

How is the evil bot (EB) even coming in on 5x,xxx?

So if only one port is open (SSH on XX) why is the machine even talking to EB?
Sorry - these are undoubtedly naive networking questions.

The critical question is are these using NAS resource in a significant way?

In any TCP/UDP socket-based connection, you have a client and a server. A client may connect to a service on a remote server on port 22, but the client itself has to originate that connection on a port on its local system. This is usually random and a port intentionally from a high number is chosen (hence, the 5xxxx). So in any connection, 2 IP addresses and 2 ports are always involved, the source and the destination.

Think of the connection between your PC's GPU to your monitor. The GPU probably has at least 2 HDMI/DP ports. Similarly, the monitor/TV probably also has 2 HDMI/DP ports. You can use any combination to connect, but there are still ports on both sides. Both the GPU AND the monitor need to have its own ports. So, think of port 22 as the port on the monitor/display and port 5xxx as the port on the GPU side. Makes sense?

 

[elvisimprsntr]

(1) Do these attempts consume much resource? From what I've read this is not a big deal?
(2) I guess ideally I should implement PF Sense but I've not had time to research this yet. Should I panic and start on this project yesterday?
(3) I don't really understand how to read the logs properly. Here are some examples:

1. Repeated brute force attempts will undoubtedly consume some resources, but that should not be your biggest concern.
2. Stop exposing your NAS directly to the internet immediately! Deploy a better firewall that supports VPN connections. Don't convince yourself that you are protected by using a strong password. Passwords and 2FA do not protect against privilege escalation vulnerabilities.
3. The example IP address are likely from a cloud hosting service, which someone is actively using to perform brute force attacks.

137.184.126.78 | DigitalOcean LLC | AbuseIPDB

www.abuseipdb.com

211.245.31.15 | SK Broadband Co Ltd | AbuseIPDB

www.abuseipdb.com

 

[Whattteva]

Also... aside from enabling 2FA, is there anything else that I should do to protect my system in general? Is there a "security best practices" article for TrueNAS out there somewhere?

I don't know if there is a best practice document, but you can certainly find recommendations from various posts around the forums like using VPN.

I personally wouldn't expose anything straight out on the internet (especially the web admin) except maybe SSH with only key-based auth and no password-based auth at all. But VPN really is the best answer.

 

[James S]

In any TCP/UDP socket-based connection, you have a client and a server. A client may connect to a service on a remote server on port 22, but the client itself has to originate that connection on a port on its local system. This is usually random and a port intentionally from a high number is chosen (hence, the 5xxxx). So in any connection, 2 IP addresses and 2 ports are always involved, the source and the destination.

Think of the connection between your PC's GPU to your monitor. The GPU probably has at least 2 HDMI/DP ports. Similarly, the monitor/TV probably also has 2 HDMI/DP ports. You can use any combination to connect, but there are still ports on both sides. Both the GPU AND the monitor need to have its own ports. So, think of port 22 as the port on the monitor/display and port 5xxx as the port on the GPU side. Makes sense?

Thanks! That really helps. Seems like I better pick up a book on the ABC of networking

1. Repeated brute force attempts will undoubtedly consume some resources, but that should not be your biggest concern.
2. Stop exposing your NAS directly to the internet immediately! Deploy a better firewall that supports VPN connections. Don't convince yourself that you are protected by using a strong password. Passwords and 2FA do not protect against privilege escalation vulnerabilities.
3. The example IP address are likely from a cloud hosting service, which someone is actively using to perform brute force attacks.

137.184.126.78 | DigitalOcean LLC | AbuseIPDB

www.abuseipdb.com

211.245.31.15 | SK Broadband Co Ltd | AbuseIPDB

www.abuseipdb.com

Sorry - I'm a bit tired of the stereotyped response, "stop exposing your NAS to the internet". These boxes, one way or another have to be on the 'net, right? From above, I'm on a LAN (so not a WAN address), using a non-standard SSH port, running a port forward to that single port, with SSH keys (also with a long password). The only other ports open are 443 (for the GUI - only exposed to my LAN) and several at 1XXXX to expose the NAS (direct connections) to my VMware machine.
So, can you clarify for me, please, does this count as "exposing ... directly"?
The example IP addresses are often DigitalOcean who seem to do little to monitor / control abuse on their servers

I don't know if there is a best practice document, but you can certainly find recommendations from various posts around the forums like using VPN.

I personally wouldn't expose anything straight out on the internet (especially the web admin) except maybe SSH with only key-based auth and no password-based auth at all. But VPN really is the best answer.

Thanks - yes, I'm configured for SSH and no password.
I'll look into VPN.

 

[Whattteva]

Sorry - I'm a bit tired of the stereotyped response, "stop exposing your NAS to the internet". These boxes, one way or another have to be on the 'net, right? From above, I'm on a LAN (so not a WAN address), using a non-standard SSH port, running a port forward to that single port, with SSH keys (also with a long password). The only other ports open are 443 (for the GUI - only exposed to my LAN) and several at 1XXXX to expose the NAS (direct connections) to my VMware machine.
So, can you clarify for me, please, does this count as "exposing ... directly"?
The example IP addresses are often DigitalOcean who seem to do little to monitor / control abuse on their servers

Thanks - yes, I'm configured for SSH and no password.
I'll look into VPN.

I totally get this sentiment and I do think it's overblown sometimes. In my opinion, there's nothing wrong with exposing SSH so long as you've taken the proper precautions (and it sounds like you have). In fact, it is exactly what it was created for (secure remote access). OpenBSD project (the creator) even enables the service by default and they take great pride in their default install as you can see from their slogan:

Only two remote holes in the default install, in a heck of a long time!

My recommendation for VPN is really more from the standpoint of convenience vs security. SSH only gives you access to exactly that one service (SSH). Sure, you could route other traffic through there through clever use of the SOCKS proxy, but it's a total pain in the neck and must be setup for every individual service manually. Compare that to using VPN where it's basically just one-size-fits-all type of thing. Once you set it up, you get full access to all your services instead of just one specific thing.

Btw, exposing to the internet means it's accessible through your public IP. So if it isn't accessible through your public IP, it's not "exposed to the internet".