Plex hacked - How secure are jails at protecting FreeNAS?

[depasseg]

Apparently my plex plugin jail was attacked and destroyed. I'm guessing plex was attacked (vs a service in the jail), but in any case. Assuming someone was able to login to the jail, what access would they have to the rest of the FreeNAS system?

 

[jgreco]

The biggest risk would be if they gained access to a shell and then had access to the rest of your internal network. This would include access to the FreeNAS box equivalent to what an unwelcomed wifi guest on your network might have, assuming you don't have a segmented network.

It may be possible to break out of a chroot environment, and the jail is basically chroot-on-steroids. I'm not sure what the current state of the art on FreeBSD is for protecting against the chroot/chdir attack, although the basic attack was blunted years ago. Still, us old-timers think it very risky to allow root access inside a chroot environment, and some of us are paranoid enough that we don't even have /bin/sh (normal attack vector) in there.

 

[m0nkey_]

Apparently my plex plugin jail was attacked and destroyed. I'm guessing plex was attacked

Just curious how they got in. Were you running the latest version of Plex?

 

[DrKK]

Also, if you don't mind my asking, did you forward the default port? Or did you use a non-standard port? In my view, using a non-standard port thwarts about 99% of the attacks.

 

[depasseg]

Well I think I figured out what happened. I think I enabled remote access (meaning not via the website, but directly to the server), and I had port forwarding enabled (yes default, doh), and the cherry on top....didn't require authorization (I was having issues so I disabled it for troubleshooting and forgot to turn it back on). That's quite the combination there, isn't it.

So I think, the good news is the jails should be able to hold up (although I need to look into that root access and /bin/sh that @jgreco mentioned)

So, I don't think a vulnerability was exploited (I was on 0.9.15 from Jan-ish). I think it was just stupidity. Now it's time to recover.

First I blocked all inbound traffic for that server. Then I reverted snapshots to mid last week in order to reset some of the server changes. Then I found that it was best to "forget" the server and re-login. And since I kept seeing connections to ireland, mexico, australia and dsl modem, I decided to "forget/sign-out", and then upgrade the plugin, then sign in. My netstat -al finally looks better, but still strange. And I just started running sockstat, which is showing a bunch of question marks for user, command, PID and FD. Seems strange, so I installed a new plugin. Same issue. The weird thing is that they don't show up on a different test box.

Code:

root@plexmediaserver_2:/ # sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
plex     python     68991 10 tcp4   127.0.0.1:32436       *:*
plex     Plex Media 66487 57 tcp4   *:32400               *:*
plex     Plex Media 66487 58 tcp4   127.0.0.1:32401       *:*
plex     Plex Media 66487 65 tcp4   172.16.1.195:40105    45.79.211.86:443
plex     Plex Media 66487 66 udp4   *:32414               *:*
plex     Plex Media 66487 67 udp4   *:1901                *:*
plex     Plex Media 66487 68 udp4   *:32410               *:*
plex     Plex Media 66487 69 udp4   *:32412               *:*
plex     Plex Media 66487 70 udp4   *:32413               *:*
plex     Plex Media 66487 71 udp4   127.0.0.1:12899       *:*
plex     Plex Media 66487 72 udp4   172.16.1.195:60410    *:*
plex     Plex Media 66487 73 udp4   127.0.0.1:40782       *:*
plex     Plex Media 66487 74 udp4   172.16.1.195:19554    *:*
plex     Plex Media 66487 76 tcp4   172.16.1.195:32400    172.16.1.133:56965
plex     Plex Media 66487 86 udp4   172.16.1.195:11336    *:*
root     python2.7  49115 3  tcp4   172.16.1.195:12361    *:*
root     cron       42058 4  dgram  -> /var/run/logpriv
root     syslogd    42005 4  dgram  /var/run/log
root     syslogd    42005 5  dgram  /var/run/logpriv
root     syslogd    42005 6  udp6   *:514                 *:*
root     syslogd    42005 7  udp4   *:514                 *:*
?        ?          ?     ?  tcp4   127.0.0.1:32400       127.0.0.1:29531
?        ?          ?     ?  tcp4   172.16.1.195:40122    139.162.7.93:443
?        ?          ?     ?  tcp4   172.16.1.195:40115    82.94.168.13:443
?        ?          ?     ?  tcp4   172.16.1.195:32400    172.16.1.195:45896
?        ?          ?     ?  tcp4   172.16.1.195:40117    45.79.10.123:443
?        ?          ?     ?  tcp4   172.16.1.195:32400    172.16.1.195:50409
?        ?          ?     ?  tcp4   172.16.1.195:40123    45.79.211.86:443
?        ?          ?     ?  tcp4   172.16.1.195:40120    139.162.216.99:443
?        ?          ?     ?  tcp4   172.16.1.195:40121    45.79.86.131:443
?        ?          ?     ?  tcp4   172.16.1.195:40119    104.214.144.122:443
?        ?          ?     ?  tcp4   172.16.1.195:40116    45.79.211.86:443

 

[DrKK]

The question mark stuff is what you get after those sockets DISCONNECT.

They hang around as question marks are the detritus that remains usually after a connection has completed. They will go away after a time.

You have a lot of 443's on there. Meaning those are https connection you have made outbound to some web server. That one you have at the top 45.79.211.86 is almost certainly to the plex.tv server itself, so that one is fine.

But this sockstat list looks OK to me...

 

[DrKK]

Also I don't care what anyone says, you still look like Frans Bauer.

https://www.youtube.com/watch?v=uXgQZcnE0SY

 

[SweetAndLow]

I think I enabled remote access (meaning not via the website, but directly to the server), and I had port forwarding enabled (yes default, doh), and the cherry on top....didn't require authorization (I was having issues so I disabled it for troubleshooting and forgot to turn it back on). That's quite the combination there, isn't it.

Plex has removed the feature that lets you turn on/off authentication. Not sure what version but I have 9.16.6 and that setting is nowhere to be found.

 

[depasseg]

Not sure what version but I have 9.16.6 and that setting is nowhere to be found.

It's in the preferences.xml file, which the Jail config window "helpfully" let me set.

 

[jgreco]

So I think, the good news is the jails should be able to hold up (although I need to look into that root access and /bin/sh that @jgreco mentioned)

You may not be able to do much about that. The intent in FreeNAS is to use jail capability to provide access to a more-standard UNIX environment, and most of the jails seem to be designed that way. Plus, since volunteers make most of the jails, and most of them depend on existing ports/packages to build the jails, there's a huge dependency on full system functionality within the jail.

In the service provider world, though, what we're always worried about is the attack surface of a service. If you have to expose a ${thing} to the Internet, in order to allow people to access it, you become much more vulnerable to the possibility that someone will find your ${thing} and if there's a security vulnerability, will exploit it. These days the bad guys are making databases of ${thing}s including both full devices but also versions of often-vulnerable or high-stakes software like PHP or SSH. So when there's suddenly a new vulnerability affecting a given bit of software, these guys literally just pull scan results out of an existing database and your server is pwned shortly thereafter.

So, if we assume that this is something that WILL happen, the next step is mitigation. UNIX, as with many operating systems, is written in C, because C is a pseudo-high-level language that is also easily translated into assembly language, which makes it extremely fast and efficient. However, C lacks a true string data type and array bounds, and is instead reliant on the expertise of programmers to manage things. This is a fail. Inevitably even an experienced programmer may fail to spot some place where external data can cause a buffer to be written past the end of the space allocated to it; this is called a "buffer overflow".

A classic example of a buffer overflow exploit where an Internet-facing daemon (let's call it BadSQL) had a known failure to check bounds, and an attacker established a connection to the BadSQL server and then sent a very long query to the daemon that caused it to stomp on the stack. At the end of the request is some code that executes "/bin/sh". When that returns, suddenly /bin/sh executes and the attacker is given a shell over what used to be a remote SQL connection. If the SQL daemon is running as an unprivileged user, then the attacker may need to find a userland vulnerability to gain access to root. Anyways, lots about that here: http://www.cis.syr.edu/~wedu/Teaching/IntrCompSec/LectureNotes_New/Buffer_Overflow.pdf

But jails, jails and chroot environments are weird. There's no particular requirement that a jail has to host a copy of /bin/sh. It could host a shell in an alternate location, or, better yet, not at all. For BadSQL, the jail could be chrooted to /badsql and the executable as /badsql/sbin/badsqld (executed inside the jail as /sbin/badsqld) and that could literally be the only executable file within /badsql. Further, it can be running as non-root-user "badsql". This means that even if you're running the swiss-cheese-holes version of badsql from 20 years ago, the worst an attacker can do (aside from trashing your SQL DB) is to cause badsqld to fail and stop running. Since there is no root privilege running inside the chroot environment, it isn't easily possible to execute a chroot/chdir attack to try to break out of the jail, and since there's no /bin/sh and no other executable tools in there anyways to assist in such a breakout attempt, the difficulty of getting a shell is substantially increased. It may still be possible to exploit and break out of such a jail given sufficient resources and some kernel vulnerabilities, but at that point we're talking something much closer to NSA-skill-level hackers than simple script kiddie exploits.

But the task of actually creating a secured jail in this manner is rather more difficult than just running some basic pkg-install commands on a FreeBSD base system in a jail. Instead, you have to start from empty and then fill the jail with the minimal set of things you truly need.

 

[depasseg]

The question mark stuff is what you get after those sockets DISCONNECT.

They hang around as question marks are the detritus that remains usually after a connection has completed. They will go away after a time.

You have a lot of 443's on there. Meaning those are https connection you have made outbound to some web server. That one you have at the top 45.79.211.86 is almost certainly to the plex.tv server itself, so that one is fine.

But this sockstat list looks OK to me...

Thanks for clarifying that. It makes me feel a little better. At least now I understand what's going on in the second plugin I installed.

I've gone through my plexwatch logs and found "local" users from external IP's from almost a month ago. So I'm rolling back the snapshots.

 

[anodos]

Perhaps it's time to look into adding a pfsense appliance with OpenVPN to your home network.

 

[depasseg]

Looks like plex is still trying to reach out to some "residential" (non-plex sites) from port 32400.

Code:

root@plexmediaserver_1:~ # netstat -al
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 172.16.1.33.43933      ec2-52-51-141-223.eu-west-1.compute.amazonaws.com.https  TIME_WAIT
tcp4       0      0 172.16.1.33.30117      ec2-52-49-89-54.eu-west-1.compute.amazonaws.com.https  TIME_WAIT
tcp4       0      0 localhost.58182        localhost.49876        TIME_WAIT
tcp4       0      0 172.16.1.33.32400      c-76-99-7-124.hsd1.pa.comcast.net.59392  FIN_WAIT_2
tcp4       0      0 172.16.1.33.32400      72.132.3.123.sta.dodo.net.au.61829  FIN_WAIT_2
tcp4       0      0 172.16.1.33.32400      172.58.169.208.39616   FIN_WAIT_2
tcp4       0      0 172.16.1.33.24397      li1021-206.members.linode.com.https  ESTABLISHED
tcp4       0      0 localhost.24383        *.*                    LISTEN
tcp4       0      0 localhost.24347        *.*                    LISTEN
tcp4       0      0 localhost.24342        *.*                    LISTEN
tcp4       0      0 localhost.24331        *.*                    LISTEN
tcp4       0      0 localhost.24277        *.*                    LISTEN
tcp4       0      0 localhost.24231        *.*                    LISTEN
tcp4       0      0 localhost.49876        *.*                    LISTEN
tcp4       0      0 localhost.32401        *.*                    LISTEN
tcp4       0      0 *.32400                *.*                    LISTEN
tcp4       0      0 172.16.1.33.32400      c-76-99-7-124.hsd1.pa.comcast.net.59390  FIN_WAIT_2
tcp4       0      0 172.16.1.33.32400      c-76-99-7-124.hsd1.pa.comcast.net.59391  FIN_WAIT_2
tcp4       0      0 172.16.1.33.12353      *.*                    LISTEN
udp4       0      0 172.16.1.33.11279      *.*                   
udp4       0      0 *.1901                 *.*                   
udp4       0      0 172.16.1.33.26254      *.*                   
udp4       0      0 localhost.10794        *.*                   
udp4       0      0 172.16.1.33.43034      *.*                   
udp4       0      0 localhost.47584        *.*                   
udp4       0      0 *.32413                *.*                   
udp4       0      0 *.32412                *.*                   
udp4       0      0 *.32410                *.*                   
udp4       0      0 *.32414                *.*                   
udp4       0      0 *.syslog               *.*                   
udp6       0      0 *.syslog                                      *.*                                          
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffff804f737e4b0 dgram       0      0        0 fffff816139d1870        0        0
fffff816139d1870 dgram       0      0 fffff8102e43c3f0        0 fffff804f737e4b0        0 /var/run/logpriv
fffff813f87af0f0 dgram       0      0 fffff815dc23c5e8        0        0        0 /var/run/log
root@plexmediaserver_1:~ # netstat -al
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0  81272 172.16.1.33.32400      bc4dfb1ae193.cpe.westmancom.com.2183  ESTABLISHED
tcp4       0      0 172.16.1.33.32400      bc4dfb1ae193.cpe.westmancom.com.2182  ESTABLISHED
tcp4       0      0 172.16.1.33.32400      bc4dfb1ae193.cpe.westmancom.com.2181  ESTABLISHED
tcp4       0      0 172.16.1.33.32400      177.234.12.198.58437   ESTABLISHED
tcp4       0  81437 172.16.1.33.32400      bc4dfb1ae193.cpe.westmancom.com.2145  ESTABLISHED
tcp4       0      0 172.16.1.33.32400      adsl-64-237-236-238.prtc.net.18204  TIME_WAIT
tcp4       0      0 172.16.1.33.32400      c-76-99-7-124.hsd1.pa.comcast.net.59392  FIN_WAIT_2
tcp4       0      0 172.16.1.33.32400      72.132.3.123.sta.dodo.net.au.61829  FIN_WAIT_2
tcp4       0      0 172.16.1.33.32400      172.58.169.208.39616   FIN_WAIT_2
tcp4       0      0 172.16.1.33.24397      li1021-206.members.linode.com.https  ESTABLISHED
tcp4       0      0 localhost.24383        *.*                    LISTEN
tcp4       0      0 localhost.24347        *.*                    LISTEN
tcp4       0      0 localhost.24342        *.*                    LISTEN
tcp4       0      0 localhost.24331        *.*                    LISTEN
tcp4       0      0 localhost.24277        *.*                    LISTEN
tcp4       0      0 localhost.24231        *.*                    LISTEN
tcp4       0      0 localhost.49876        *.*                    LISTEN
tcp4       0      0 localhost.32401        *.*                    LISTEN
tcp4       0      0 *.32400                *.*                    LISTEN
tcp4       0      0 172.16.1.33.12353      *.*                    LISTEN
udp4       0      0 172.16.1.33.11279      *.*                   
udp4       0      0 *.1901                 *.*                   
udp4       0      0 172.16.1.33.26254      *.*                   
udp4       0      0 localhost.10794        *.*                   
udp4       0      0 172.16.1.33.43034      *.*                   
udp4       0      0 localhost.47584        *.*                   
udp4       0      0 *.32413                *.*                   
udp4       0      0 *.32412                *.*                   
udp4       0      0 *.32410                *.*                   
udp4       0      0 *.32414                *.*                   
udp4       0      0 *.syslog               *.*                   
udp6       0      0 *.syslog                                      *.*                                          
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffff804f737e4b0 dgram       0      0        0 fffff816139d1870        0        0
fffff816139d1870 dgram       0      0 fffff8102e43c3f0        0 fffff804f737e4b0        0 /var/run/logpriv
fffff813f87af0f0 dgram       0      0 fffff815dc23c5e8        0        0        0 /var/run/log
root@plexmediaserver_1:~ # sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
plex     python     69400 4  tcp4   127.0.0.1:24383       *:*
plex     python     69372 5  tcp4   127.0.0.1:24331       *:*
plex     python     69371 4  tcp4   127.0.0.1:24342       *:*
plex     python     69370 4  tcp4   127.0.0.1:24347       *:*
plex     python     69344 4  tcp4   127.0.0.1:24277       *:*
plex     python     69325 4  tcp4   127.0.0.1:24231       *:*
plex     python     69313 9  tcp4   127.0.0.1:49876       *:*
plex     Plex Media 69288 57 tcp4   *:32400               *:*
plex     Plex Media 69288 58 tcp4   127.0.0.1:32401       *:*
plex     Plex Media 69288 64 udp4   *:32414               *:*
plex     Plex Media 69288 67 udp4   *:1901                *:*
plex     Plex Media 69288 68 udp4   *:32410               *:*
plex     Plex Media 69288 70 udp4   *:32412               *:*
plex     Plex Media 69288 71 udp4   *:32413               *:*
plex     Plex Media 69288 72 udp4   127.0.0.1:47584       *:*
plex     Plex Media 69288 73 udp4   172.16.1.33:43034     *:*
plex     Plex Media 69288 74 udp4   127.0.0.1:10794       *:*
plex     Plex Media 69288 75 udp4   172.16.1.33:26254     *:*
plex     Plex Media 69288 78 tcp4   172.16.1.33:32400     216.36.180.145:2145
plex     Plex Media 69288 79 tcp4   172.16.1.33:32400     172.16.1.133:54024
plex     Plex Media 69288 81 tcp4   172.16.1.33:24397     45.33.75.206:443
plex     Plex Media 69288 87 tcp4   172.16.1.33:32400     216.36.180.145:2183
plex     Plex Media 69288 89 udp4   172.16.1.33:11279     *:*
plex     Plex Media 69288 94 tcp4   172.16.1.33:32400     172.16.1.133:55263
root     python2.7  21855 3  tcp4   172.16.1.33:12353     *:*
root     cron       21550 4  dgram  -> /var/run/logpriv
root     syslogd    21490 4  dgram  /var/run/log
root     syslogd    21490 5  dgram  /var/run/logpriv
root     syslogd    21490 6  udp6   *:514                 *:*
root     syslogd    21490 7  udp4   *:514                 *:*
?        ?          ?     ?  tcp4   172.16.1.33:32400     64.237.236.238:18204
?        ?          ?     ?  tcp4   172.16.1.33:32400     216.36.180.145:2181
?        ?          ?     ?  tcp4   172.16.1.33:32400     216.36.180.145:2182
?        ?          ?     ?  tcp4   172.16.1.33:32400     76.99.7.124:59392
?        ?          ?     ?  tcp4   172.16.1.33:32400     172.58.169.208:39616
?        ?          ?     ?  tcp4   172.16.1.33:32400     123.3.132.72:61829
root@plexmediaserver_1:~ # 

 

[depasseg]

Perhaps it's time to look into adding a pfsense appliance with OpenVPN to your home network.

I'm running a ubiquiti edge router and a sophos NGFW. And yet these connections are still going. Which makes me think there is something in my plex account that causes my server to reach out.
Time for an explicit outbound block rule.

 

[depasseg]

Seems to stop the outbound, but still curious why it's happening.

Code:

root@plexmediaserver_1:/ # netstat -al
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52302  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52300  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52144  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52142  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55410  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55396  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      66-60-1-80.prtc.net.53471  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      66-60-1-80.prtc.net.63869  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52145  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      pool-72-88-156-183.nwrknj.east.verizon.net.52139  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55411  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55409  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55399  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      host86-188-69-127.range86-188.btcentralplus.com.55395  SYN_RCVD
tcp4       0      0 172.16.1.33.32400      66-60-1-80.prtc.net.53550  SYN_RCVD

 

[SweetAndLow]

If you go to your plex activity do you see lots of streams happening?

 

[depasseg]

No, but that's probably because I'm blocking access. I did see a couple conversion jobs which I deleted. I also saw a lot of "web devices" that I didn't recognize, so I deleted. And Cloud sync, which I've never used, so I deleted that as well.

What's strange is that I rolled back to a snapshot from before the jail was exposed, and it's still making outbound connections to various devices. It's as if plex is telling my server to reach out to them.

 

[Jailer]

If I were you I'd head over to plex.tv and change your password on your account ASAP if you haven't already done so. I also understand you're trying to figure this out but if it were mine I'd destroy that jail and start over. You know for sure your jail is compromised, why mess around with it any longer? Nuke it and start over.

 

[depasseg]

Thanks, I've changed the password and forced a re-login of all devices.

And I've started making a new setup. I would love to figure out how to save my viewed history.

 

[Jailer]

Personally out of paranoia I wouldn't save anything from the old jail just to be safe. Maybe some others more knowledgeable could chime in with their advice.

I'd be scouring your jail's attached storage as well for anything that shouldn't be there.