Hit With Ransomware. zfs Rollback Having No Effect

Status
Not open for further replies.
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
Long terrible horror story cut short: A hobby site of mine, based on drupal was exploited by an RFI that deployed a rootkit, when then owned all my boxes connected to that host (thanks to password-less and convenient ssh certs used for rsync jobs to backup my site to my desktop, then to my NAS and (thankfully) CrashPlan). It eventually encrypted everything on all systems that were not actual system files, and left me a ransom note.

td;dr : Ransomware victim has all files in zfs pool encrypted.

When I look at my snapshots, I see several, so I am optimistic. I rollback without error, but the encrypted files are still on the file system. I delete files, rollback, and they don't re-appear.

I wonder what I am missing, or if the entire FreeNAS system is also owned. I must believe it is, and therefore I know I need to burn it to the ground and start over, but I would really like to retrieve some data first....

Here are my available snapshots:

[root@nas] /mnt# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
Titan@auto-20160408.1700-2w 160K - 599K -
Titan@auto-20160412.1700-2w 0 - 599K -
Titan@auto-20160412.1800-2w 0 - 599K -
Titan@auto-20160413.0900-2w 0 - 599K -
Titan@auto-20160413.1000-2w 0 - 599K -
Titan@auto-20160413.1100-2w 0 - 599K -
Titan@auto-20160413.1200-2w 0 - 599K -
Titan@auto-20160413.1300-2w 0 - 599K -
Titan@auto-20160413.1400-2w 0 - 599K -
Titan@auto-20160413.1500-2w 0 - 599K -
Titan@auto-20160413.1600-2w 0 - 599K -
Titan@auto-20160413.1700-2w 0 - 599K -
Titan@auto-20160413.1800-2w 0 - 599K -
Titan@auto-20160414.0900-2w 0 - 599K -
Titan@auto-20160414.1000-2w 0 - 599K -
Titan@auto-20160414.1100-2w 0 - 599K -
Titan@auto-20160414.1200-2w 0 - 599K -
Titan@auto-20160414.1300-2w 0 - 599K -
Titan@auto-20160414.1400-2w 0 - 599K -
Titan@auto-20160414.1500-2w 0 - 599K -
Titan@auto-20160414.1600-2w 0 - 599K -
Titan@auto-20160414.1700-2w 0 - 599K -
Titan@auto-20160414.1800-2w 0 - 599K -
Titan@auto-20160415.0900-2w 0 - 599K -
Titan@auto-20160415.1000-2w 0 - 599K -
Titan@auto-20160415.1100-2w 0 - 599K -
Titan@auto-20160415.1200-2w 0 - 599K -
Titan@auto-20160415.1300-2w 0 - 599K -
Titan@auto-20160415.1400-2w 0 - 599K -
Titan@auto-20160415.1500-2w 0 - 599K -
Titan@auto-20160415.1600-2w 0 - 599K -
Titan@auto-20160415.1700-2w 0 - 599K -
Titan@auto-20160415.1800-2w 0 - 599K -
Titan@auto-20160416.0900-2w 0 - 599K -
Titan@auto-20160416.1000-2w 0 - 599K -
Titan@auto-20160416.1100-2w 0 - 599K -
Titan@auto-20160416.1200-2w 0 - 599K -
Titan@auto-20160416.1300-2w 0 - 599K -
Titan@auto-20160416.1400-2w 0 - 599K -
Titan@auto-20160416.1500-2w 0 - 599K -
Titan@auto-20160416.1600-2w 0 - 599K -
Titan@auto-20160416.1700-2w 0 - 599K -
Titan@auto-20160416.1800-2w 0 - 599K -
Titan@auto-20160417.0900-2w 0 - 599K -
Titan@auto-20160417.1000-2w 0 - 599K -
Titan@auto-20160417.1100-2w 0 - 599K -
Titan@auto-20160417.1200-2w 0 - 599K -
Titan@auto-20160417.1300-2w 0 - 599K -
Titan@auto-20160417.1400-2w 0 - 599K -
Titan@auto-20160417.1500-2w 0 - 599K -
Titan@auto-20160417.1600-2w 0 - 599K -
Titan@auto-20160417.1700-2w 0 - 599K -
Titan@auto-20160417.1800-2w 0 - 599K -
Titan@auto-20160418.0900-2w 0 - 599K -
Titan@auto-20160418.1000-2w 0 - 599K -
Titan@auto-20160418.1100-2w 0 - 599K -
Titan@auto-20160418.1200-2w 0 - 599K -
Titan@auto-20160418.1300-2w 0 - 599K -
Titan@auto-20160418.1400-2w 0 - 599K -
Titan@auto-20160418.1500-2w 0 - 599K -
Titan@auto-20160418.1600-2w 0 - 599K -
Titan@auto-20160418.1700-2w 0 - 599K -
Titan@auto-20160418.1800-2w 0 - 599K -
Titan@auto-20160419.0900-2w 0 - 599K -
Titan@auto-20160419.1000-2w 0 - 599K -
Titan@auto-20160419.1100-2w 0 - 599K -
Titan@auto-20160419.1200-2w 0 - 599K -
Titan@auto-20160419.1300-2w 0 - 599K -
Titan@auto-20160419.1400-2w 0 - 599K -
Titan@auto-20160419.1500-2w 0 - 599K -
Titan@auto-20160419.1600-2w 0 - 599K -
Titan@auto-20160419.1700-2w 0 - 599K -
Titan@auto-20160419.1800-2w 0 - 599K -
Titan/jails/.warden-template-pluginjail@clean 224K - 894M -

So I restore one:

[root@nas] /mnt# zfs rollback -r Titan@auto-20160419.1800-2w
[root@nas]

Now I check the results:

[root@nas] /mnt# ls -l /mnt/Titan/AudioBooks/
-rw-r--r-- 1 root wheel 8535 Apr 21 16:51 AlbumArtSmall.jpg.gpg
-rw-r--r-- 1 root wheel 37340 Apr 21 16:52 Folder.jpg.gpg
-rw-r--r-- 1 root wheel 27827854 Apr 21 16:53 Louie CK Unmasked 2008-07-12 CF64k.mp3.gpg
-rw-r--r-- 1 root wheel 562 Apr 21 16:51 README_FOR_UNLOCK.txt

Exact same as before. Those encrypted files are also owned by root. They weren't before.

Am I out of luck here?

Many thanks!!
 
jgreco

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Perhaps you should just go look at a snapshot rather than trying a potentially destructive operation like a rollback.

Shut down your NAS, unplug the network, then start it back up. Snapshot images should be available under /mnt/Titan/.zfs/snapshot. Do NOT enter that directory from anywhere except the UNIX command line of your NAS while disconnected from the network; if something nasty is going on, you don't want to give the $badguys access.

Look at the snaps in there to identify which ones are clean. You can create a clean backup tarball from within one of them by going into the appropriate snapshot directory (verify with ls), then running

# tar cvf /mnt/Titan/mystuff.tar .
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
Thanks for that jgreco. Turns out, all folders in /mnt/Titan/.zfs/snapshot...each one named after a snaphot (auto-2016...")....they contain folders for each of the volumes in there...but alas each of those are empty of content.

I guess I have my answer.
 
jgreco

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Uh.... hm. Yeah, that's not particularly good-sounding. I'm not clear on what the interaction between the existing snapshots, rolling back, deleting files, and then further tinkering might have been.

In a crisis situation, it is not a wise move to start making changes until you've actually retrieved a copy of your data.

What happens if you go into /mnt/Titan/.zfs/snapshot and then type "du"? (warning: could take awhile)
 
styno

styno

Patron
Joined
Apr 11, 2016
Messages
466
Look for the snapshots in your dataset as well.... /mnt/Titan/AudioBooks/.zfs/snapshot
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
Ugh. Empty folders everywhere. All dates on the folders show the date of the exploit, and are empty. Sonofagun. This was a particularly thorough attack.
 
styno

styno

Patron
Joined
Apr 11, 2016
Messages
466
At the moment I don't really see how that can happen, if you enabled recursive snapshots on the pool there should be per dataset snapshots. In your example you restored 20160419 and the filesystem shows files from 0421. That smells like different datasets => different snapshots...
 
depasseg

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
Do you only have 1 main dataset (Titan)? Or subdatasets? Snapshots need to go with each dataset.

Is it possible to get a full output of "zfs list" and "zfs list -t snapshot"?
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
styno said:
At the moment I don't really see how that can happen, if you enabled recursive snapshots on the pool there should be per dataset snapshots. In your example you restored 20160419 and the filesystem shows files from 0421. That smells like different datasets => different snapshots...
Click to expand...

Thanks Styno. The exploit happened on the 21st. Is it possible that my attempted restore of the 19th actually restored nothing (due to the aforementioned snapshot folders all being empty), and that is why you still see that datestamp, after an attempted rollback to a snapshot from the 19th?
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
depasseg said:
Do you only have 1 main dataset (Titan)? Or subdatasets? Snapshots need to go with each dataset.

Is it possible to get a full output of "zfs list" and "zfs list -t snapshot"?
Click to expand...

Thanks depasseg for trying to help.

Here you go:

[root@nas] ~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
Titan 6.48T 4.17T 599K /mnt/Titan
Titan/.system 23.2M 4.17T 352K /mnt/Titan/.system
Titan/.system/cores 12.8M 4.17T 12.8M /mnt/Titan/.system/cores
Titan/.system/rrd 288K 4.17T 288K /mnt/Titan/.system/rrd
Titan/.system/samba4 5.72M 4.17T 5.72M /mnt/Titan/.system/samba4
Titan/.system/syslog 4.01M 4.17T 4.01M /mnt/Titan/.system/syslog
Titan/Allison 30.7G 4.17T 30.7G /mnt/Titan/Allison
Titan/Archives 38.1G 4.17T 38.1G /mnt/Titan/Archives
Titan/AudioBooks 5.31G 4.17T 5.31G /mnt/Titan/AudioBooks
Titan/Backups 661G 4.17T 661G /mnt/Titan/Backups
Titan/Bin 88.1G 4.17T 88.1G /mnt/Titan/Bin
Titan/Comics 51.3G 4.17T 51.3G /mnt/Titan/Comics
Titan/John 243G 4.17T 243G /mnt/Titan/John
Titan/Kids 796G 4.17T 796G /mnt/Titan/Kids
Titan/MP3 80.1G 4.17T 80.1G /mnt/Titan/MP3
Titan/Movies 1019G 4.17T 1019G /mnt/Titan/Movies
Titan/MusicVideos 48.0G 4.17T 48.0G /mnt/Titan/MusicVideos
Titan/Shona 308G 4.17T 308G /mnt/Titan/Shona
Titan/Stills 403G 4.17T 403G /mnt/Titan/Stills
Titan/TV 2.01T 4.17T 2.01T /mnt/Titan/TV
Titan/Torrents 207G 4.17T 207G /mnt/Titan/Torrents
Titan/Videos 593G 4.17T 593G /mnt/Titan/Videos
Titan/clone-auto-20160408.1700-2w 160K 4.17T 599K /mnt/Titan/clone-auto-20160408.1700-2w
Titan/jails 1.42G 4.17T 535K /mnt/Titan/jails
Titan/jails/.warden-template-pluginjail 894M 4.17T 894M /mnt/Titan/jails/.warden-template-pluginjail
Titan/jails/.warden-template-pluginjail/clone-clean 16.0K 4.17T 894M /mnt/Titan/jails/.warden-template-pluginjail/clone-clean
Titan/jails/crashplan_1 554M 4.17T 1.40G /mnt/Titan/jails/crashplan_1

[root@nas] ~# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
Titan@auto-20160408.1700-2w 160K - 599K -
Titan@auto-20160412.1700-2w 0 - 599K -
Titan@auto-20160412.1800-2w 0 - 599K -
Titan@auto-20160413.0900-2w 0 - 599K -
Titan@auto-20160413.1000-2w 0 - 599K -
Titan@auto-20160413.1100-2w 0 - 599K -
Titan@auto-20160413.1200-2w 0 - 599K -
Titan@auto-20160413.1300-2w 0 - 599K -
Titan@auto-20160413.1400-2w 0 - 599K -
Titan@auto-20160413.1500-2w 0 - 599K -
Titan@auto-20160413.1600-2w 0 - 599K -
Titan@auto-20160413.1700-2w 0 - 599K -
Titan@auto-20160413.1800-2w 0 - 599K -
Titan@auto-20160414.0900-2w 0 - 599K -
Titan@auto-20160414.1000-2w 0 - 599K -
Titan@auto-20160414.1100-2w 0 - 599K -
Titan@auto-20160414.1200-2w 0 - 599K -
Titan@auto-20160414.1300-2w 0 - 599K -
Titan@auto-20160414.1400-2w 0 - 599K -
Titan@auto-20160414.1500-2w 0 - 599K -
Titan@auto-20160414.1600-2w 0 - 599K -
Titan@auto-20160414.1700-2w 0 - 599K -
Titan@auto-20160414.1800-2w 0 - 599K -
Titan@auto-20160415.0900-2w 0 - 599K -
Titan@auto-20160415.1000-2w 0 - 599K -
Titan@auto-20160415.1100-2w 0 - 599K -
Titan@auto-20160415.1200-2w 0 - 599K -
Titan@auto-20160415.1300-2w 0 - 599K -
Titan@auto-20160415.1400-2w 0 - 599K -
Titan@auto-20160415.1500-2w 0 - 599K -
Titan@auto-20160415.1600-2w 0 - 599K -
Titan@auto-20160415.1700-2w 0 - 599K -
Titan@auto-20160415.1800-2w 0 - 599K -
Titan@auto-20160416.0900-2w 0 - 599K -
Titan@auto-20160416.1000-2w 0 - 599K -
Titan@auto-20160416.1100-2w 0 - 599K -
Titan@auto-20160416.1200-2w 0 - 599K -
Titan@auto-20160416.1300-2w 0 - 599K -
Titan@auto-20160416.1400-2w 0 - 599K -
Titan@auto-20160416.1500-2w 0 - 599K -
Titan@auto-20160416.1600-2w 0 - 599K -
Titan@auto-20160416.1700-2w 0 - 599K -
Titan@auto-20160416.1800-2w 0 - 599K -
Titan@auto-20160417.0900-2w 0 - 599K -
Titan@auto-20160417.1000-2w 0 - 599K -
Titan@auto-20160417.1100-2w 0 - 599K -
Titan@auto-20160417.1200-2w 0 - 599K -
Titan@auto-20160417.1300-2w 0 - 599K -
Titan@auto-20160417.1400-2w 0 - 599K -
Titan@auto-20160417.1500-2w 0 - 599K -
Titan@auto-20160417.1600-2w 0 - 599K -
Titan@auto-20160417.1700-2w 0 - 599K -
Titan@auto-20160417.1800-2w 0 - 599K -
Titan@auto-20160418.0900-2w 0 - 599K -
Titan@auto-20160418.1000-2w 0 - 599K -
Titan@auto-20160418.1100-2w 0 - 599K -
Titan@auto-20160418.1200-2w 0 - 599K -
Titan@auto-20160418.1300-2w 0 - 599K -
Titan@auto-20160418.1400-2w 0 - 599K -
Titan@auto-20160418.1500-2w 0 - 599K -
Titan@auto-20160418.1600-2w 0 - 599K -
Titan@auto-20160418.1700-2w 0 - 599K -
Titan@auto-20160418.1800-2w 0 - 599K -
Titan@auto-20160419.0900-2w 0 - 599K -
Titan@auto-20160419.1000-2w 0 - 599K -
Titan@auto-20160419.1100-2w 0 - 599K -
Titan@auto-20160419.1200-2w 0 - 599K -
Titan@auto-20160419.1300-2w 0 - 599K -
Titan@auto-20160419.1400-2w 0 - 599K -
Titan@auto-20160419.1500-2w 0 - 599K -
Titan@auto-20160419.1600-2w 0 - 599K -
Titan@auto-20160419.1700-2w 0 - 599K -
Titan@auto-20160419.1800-2w 0 - 599K -
Titan/jails/.warden-template-pluginjail@clean 224K - 894M -
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
anodos said:
Click 'system' - 'advanced' - 'save debug' and post resulting tarball here.
Click to expand...

Hi anodos,

That file is 378 GB in size, so I can't copy it here. Not sure there is a practical way to share that.....is there something else, or a subset of that which I can provide you?
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
It looks like this might have snapshots, but I suspect the rest have no snapshots:

zfs list -t snapshot -r Titan/jails/crashplan_1

This will show all snapshots on all datasets under Titan:
zfs list -t snapshot -r Titan
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
rs225 said:
How much is the ransom?
Click to expand...

2 Bitcoins, or $1,200 CAD. I can't give in.

1) They are thieves, how can I trust I will get the key?
2) If I pay them $1,200 now, what's stopping them from asking for more?

I have solid backups of my "Crown Jewels" thank to CrashPlan from my Desktop. All else comes with an acceptable loss threshold. Not desirable, but it is what it is.
 
B

bbqnerd

Dabbler
Joined
Apr 26, 2016
Messages
12
rs225 said:
It looks like this might have snapshots, but I suspect the rest have no snapshots:

zfs list -t snapshot -r Titan/jails/crashplan_1

This will show all snapshots on all datasets under Titan:
zfs list -t snapshot -r Titan
Click to expand...

[root@nas] ~# zfs list -t snapshot -r Titan
NAME USED AVAIL REFER MOUNTPOINT
Titan@auto-20160408.1700-2w 160K - 599K -
Titan@auto-20160412.1700-2w 0 - 599K -
Titan@auto-20160412.1800-2w 0 - 599K -
Titan@auto-20160413.0900-2w 0 - 599K -
Titan@auto-20160413.1000-2w 0 - 599K -
Titan@auto-20160413.1100-2w 0 - 599K -
Titan@auto-20160413.1200-2w 0 - 599K -
Titan@auto-20160413.1300-2w 0 - 599K -
Titan@auto-20160413.1400-2w 0 - 599K -
Titan@auto-20160413.1500-2w 0 - 599K -
Titan@auto-20160413.1600-2w 0 - 599K -
Titan@auto-20160413.1700-2w 0 - 599K -
Titan@auto-20160413.1800-2w 0 - 599K -
Titan@auto-20160414.0900-2w 0 - 599K -
Titan@auto-20160414.1000-2w 0 - 599K -
Titan@auto-20160414.1100-2w 0 - 599K -
Titan@auto-20160414.1200-2w 0 - 599K -
Titan@auto-20160414.1300-2w 0 - 599K -
Titan@auto-20160414.1400-2w 0 - 599K -
Titan@auto-20160414.1500-2w 0 - 599K -
Titan@auto-20160414.1600-2w 0 - 599K -
Titan@auto-20160414.1700-2w 0 - 599K -
Titan@auto-20160414.1800-2w 0 - 599K -
Titan@auto-20160415.0900-2w 0 - 599K -
Titan@auto-20160415.1000-2w 0 - 599K -
Titan@auto-20160415.1100-2w 0 - 599K -
Titan@auto-20160415.1200-2w 0 - 599K -
Titan@auto-20160415.1300-2w 0 - 599K -
Titan@auto-20160415.1400-2w 0 - 599K -
Titan@auto-20160415.1500-2w 0 - 599K -
Titan@auto-20160415.1600-2w 0 - 599K -
Titan@auto-20160415.1700-2w 0 - 599K -
Titan@auto-20160415.1800-2w 0 - 599K -
Titan@auto-20160416.0900-2w 0 - 599K -
Titan@auto-20160416.1000-2w 0 - 599K -
Titan@auto-20160416.1100-2w 0 - 599K -
Titan@auto-20160416.1200-2w 0 - 599K -
Titan@auto-20160416.1300-2w 0 - 599K -
Titan@auto-20160416.1400-2w 0 - 599K -
Titan@auto-20160416.1500-2w 0 - 599K -
Titan@auto-20160416.1600-2w 0 - 599K -
Titan@auto-20160416.1700-2w 0 - 599K -
Titan@auto-20160416.1800-2w 0 - 599K -
Titan@auto-20160417.0900-2w 0 - 599K -
Titan@auto-20160417.1000-2w 0 - 599K -
Titan@auto-20160417.1100-2w 0 - 599K -
Titan@auto-20160417.1200-2w 0 - 599K -
Titan@auto-20160417.1300-2w 0 - 599K -
Titan@auto-20160417.1400-2w 0 - 599K -
Titan@auto-20160417.1500-2w 0 - 599K -
Titan@auto-20160417.1600-2w 0 - 599K -
Titan@auto-20160417.1700-2w 0 - 599K -
Titan@auto-20160417.1800-2w 0 - 599K -
Titan@auto-20160418.0900-2w 0 - 599K -
Titan@auto-20160418.1000-2w 0 - 599K -
Titan@auto-20160418.1100-2w 0 - 599K -
Titan@auto-20160418.1200-2w 0 - 599K -
Titan@auto-20160418.1300-2w 0 - 599K -
Titan@auto-20160418.1400-2w 0 - 599K -
Titan@auto-20160418.1500-2w 0 - 599K -
Titan@auto-20160418.1600-2w 0 - 599K -
Titan@auto-20160418.1700-2w 0 - 599K -
Titan@auto-20160418.1800-2w 0 - 599K -
Titan@auto-20160419.0900-2w 0 - 599K -
Titan@auto-20160419.1000-2w 0 - 599K -
Titan@auto-20160419.1100-2w 0 - 599K -
Titan@auto-20160419.1200-2w 0 - 599K -
Titan@auto-20160419.1300-2w 0 - 599K -
Titan@auto-20160419.1400-2w 0 - 599K -
Titan@auto-20160419.1500-2w 0 - 599K -
Titan@auto-20160419.1600-2w 0 - 599K -
Titan@auto-20160419.1700-2w 0 - 599K -
Titan@auto-20160419.1800-2w 0 - 599K -
Titan/jails/.warden-template-pluginjail@clean 224K - 894M -

Thanks.
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
Make sure the CrashPlan has what you expect, from a system that you know isn't compromised.
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Hit With Ransomware. zfs Rollback Having No Effect"

Similar threads

cmh
Replication status shows "Running" but no zfs send processes.
Replies
9
Views
5K
George Kyriazis
George Kyriazis
P
zpool import crashes system
Replies
4
Views
2K
Yorick
Yorick
S
  • Locked
Odd security log entries daily
Replies
11
Views
4K
fracai
fracai
mpdelbuono
  • Locked
SOLVED Right approach for zfs send with snapshot corruption
Replies
6
Views
3K
Stux
Stux
N
  • Locked
SOLVED Help to Recreate a Pool from Backup (Removable Local Backup/Restore)
Replies
3
Views
5K
devnullius
devnullius
Top