Data loss investigation

[supercoolfool]

Hello, thank you for reading my post. I'm not looking for assistance in recovering data, just to determine how I may have lost it. I hope I've added all the necessary info.

I'm new to FreeNAS having usually used hardware raid in a windows server environment. I recently lost pretty much all the data off my FreeNAS box and I was hoping to determine how.

At this point, I'm fairly certain I was hacked into and my files deleted from the nas, I'm just looking for confirmation. One of my Windows PC's got infected with a cryptography virus encrypting me out of all my backups. I believe at the same time whoever broke in deleted my files from the NAS so I would have to pay the ransom to get my encrypted data back.

As for the FreeNAS box, there are no errors in the logs that would indicate any errors with the pool or drives. The pool is healthy, but I'm missing some folders and tons of files from the server. An automatic scrub had been run at midnight the day before as well and all my data seemed fine afterwards.

There's only 2 issues I've ever had with the box and I was wondering if either of these would cause my data loss and not the hack:

1) When I built the box, I used a Gigabyte GA-Z68X-UD3H-B3 motherboard. There is 1 array of 6 drives in a RAID6, 4 drives are connected to the "intel" sata controller and 2 connected to the Marvel on-board sata controller. Not sure if I was taking this hardware agnostic thing a little too far and that caused my data loss. Obviously in a hardware RAID that's not possible, not sure if this is my issue. Also, the box only has 8 GB of RAM for what used to be about 12 TB of data, so I was below the 1GB of RAM per TB rule.

2) I would have these Timeout errors about once a day in the logs:

Code:

192.168.5.128	Nov 20 18:57:37	freenas	user	notice	ahcich5	Timeout on slot 29 port 0
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	ahcich5	is 00000000 cs 20000000 ss 00000000 rs 20000000 tfd c0 serr 00000000 cmd 0004dd17
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	CAM status: Command timeout
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	Retrying command

I would get these about once a day, could be on any of the drives in the array. I replaced all SATA cables with brand new cables with clips, and would still get the errors. Drives are connected directly to the motherboard.
I tested all the drives using smartd quick tests and all drives passed. I know a quick test won't tell you anything, but the drives are less than 3 months old, have had very little use and there are no SMART flags. If necessary, I can do the long tests and post the results here.

Thank you for taking the time to read my post and I appreciate any assistance.

 

[rs225]

Did you have snapshots set? That would have preserved the files.

Nothing else you mentioned looks like a problem that would cause some files to disappear. You should post your controller and disk model, as sometimes those are relevant to disk errors.

 

[Chris Moore]

Code:

192.168.5.128	Nov 20 18:57:37	freenas	user	notice	ahcich5	Timeout on slot 29 port 0
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	ahcich5	is 00000000 cs 20000000 ss 00000000 rs 20000000 tfd c0 serr 00000000 cmd 0004dd17
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	CAM status: Command timeout
192.168.5.128	Nov 20 18:57:37	freenas	user	notice	(ada5:ahcich5:0:0:0)	Retrying command

This looks like a hardware fault to me. I would suspect the controller in the system board. If there were more errors to look at a pattern might be detected. If I were betting, I would say the Marvel controller was not doing the thing it needed to do.
If you had a snapshot task running, it might be able to be rolled back to a time when the data was there?
Going forward, you might want to get a SAS controller to connect the drives to, for reliability sake, and maybe bump the RAM up a little. I would think 16GB would be plenty but you can get by with what you have.
If you want to give a SAS controller a try, this is a good primer on them:
https://forums.freenas.org/index.ph...imer-on-basic-sas-and-sata.26145/#post-165190
Also, you may need to flash a new SAS controller to the IT (initiator-target) version of the firmware:
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

 

[Chris Moore]

There's only 2 issues I've ever had with the box and I was wondering if either of these would cause my data loss and not the hack:

PS. I don't think this alone would cause the data loss. This could be a symptom of a larger problem.

 

[supercoolfool]

Thx for your quick replies! Much appreciated.

I wasn't able to find a reasonably priced 6 port SATA/SAS HBA card and I was eager to try out Freenas, so I jumped the gun a little. When I rebuild, I'll bite the bullet and go for a good controller, but I don't think I'll go back to freenas. I didn't do enough research beforehand and I wasn't aware there's no "real" data recovery for ZFS. I just loved that it was hardware agnostic, you can add drives to a pool (if I'm not mistaken) or increase the size of the drives in a pool. I've recovered plenty of NTFS, and some Ext2 and Ext3 drives and arrays so I'll probably go that route.

I can post more of the CAM status errors if you think it will provide any additional info.

Here's my controller info:

Chipset (Intel Z68 Express Chipset)

1. 2 x SATA 6Gb/s connectors (SATA3_0, SATA3_1) supporting up to 2 SATA 6Gb/s devices
2. 3 x SATA 3Gb/s connectors (SATA2_2~SATA2_4) supporting up to 3 SATA 3Gb/s devices
3. 1 x eSATA 3Gb/s connector on the back panel supporting up to 1 SATA 3Gb/s device
4. Support for SATA RAID 0, RAID 1, RAID 5, and RAID 10
   * When a RAID set is built across the SATA 6Gb/s and SATA 3Gb/s channels, the system performance of the RAID set may vary depending on the devices being connected.

Marvell 88SE9172 chip:

1. 2 x SATA 6Gb/s connectors (GSATA3_5, GSATA3_6) supporting up to 2 SATA 6Gb/s devices
2. Support for SATA RAID 0 and RAID 1

The drives are Seagate 10 TB Ironwolf drives, ST10000VN0004.

4 drives are connected to the Intel chipset controller, 2 on the Marvell. I could have connected all 6 to the intel controller, but it would have required using an esata to sata adapter cable on one of the drives.

 

[Chris Moore]

I wasn't aware there's no "real" data recovery for ZFS.

It is called snapshots. If you had setup the system to make periodic snapshots, you could have easily reverted to a snapshot where your data was there. It is not the fault of FreeNAS that you didn't learn enough about how to use the system.

 

[garm]

There should also be periodic S.M.A.R.T tests running.

Ransomware is basically mitigated in two ways, the user has no direct access to backups and snapshots.

Now you could set up a schema on NTFS where dedicated users have access and are able to run dedicated software for backup. This will throw off most ransomware. You do how ever not get the long term data integrity you get from ZFS.

With ZFS you can of course set up the same schema of dedicated users, but you also get snapshots. They are incredibly lightweight due to ZFS being CoW. It has two advantages in a home invironmemt. After a snapshot is taken any modification to a file is saved separate to the original file. If you have over 50% of your pool occupied then the ransomware won’t be able to encrypt everything. After the corruption has happened you can simply roll back to the last unaffected snapshot.

 

[supercoolfool]

It is called snapshots. If you had setup the system to make periodic snapshots, you could have easily reverted to a snapshot where your data was there. It is not the fault of FreeNAS that you didn't learn enough about how to use the system.

Thx for your reply. Just to clarify, I wasn't throwing any shade at Freenas, this was entirely my own doing for not doing enough research upfront. I didn't setup snapshots as I was still in the testing/playing around phase and I had another copy of my data.

 

[supercoolfool]

There should also be periodic S.M.A.R.T tests running.

Ransomware is basically mitigated in two ways, the user has no direct access to backups and snapshots.

Now you could set up a schema on NTFS where dedicated users have access and are able to run dedicated software for backup. This will throw off most ransomware. You do how ever not get the long term data integrity you get from ZFS.

With ZFS you can of course set up the same schema of dedicated users, but you also get snapshots. They are incredibly lightweight due to ZFS being CoW. It has two advantages in a home invironmemt. After a snapshot is taken any modification to a file is saved separate to the original file. If you have over 50% of your pool occupied then the ransomware won’t be able to encrypt everything. After the corruption has happened you can simply roll back to the last unaffected snapshot.

Thx for your reply as well. Unfortunately, the data on my freenas box was maliciously deleted, not encrypted. It was the backup copy of my data that was on a drobo connected to a windows PC that got encrypted.

The 2 incidents happened less that 48 hours apart, and the data loss on the Freenas box makes no sense to me unless it was done by hand. Some entire folders are gone and some folders just the contents, but everything reports as healthy.

 

[Chris Moore]

Thx for your reply as well. Unfortunately, the data on my freenas box was maliciously deleted, not encrypted. It was the backup copy of my data that was on a drobo connected to a windows PC that got encrypted.

The 2 incidents happened less that 48 hours apart, and the data loss on the Freenas box makes no sense to me unless it was done by hand. Some entire folders are gone and some folders just the contents, but everything reports as healthy.

Did you have any sort of synchronization program running that automatically copied from one place to the other? If so, I can see how a program might have deleted the copy of the file on FreeNAS once it didn't see the file (because of the encryption) as being present in the origin location.

 

[garm]

It doesn’t really matter if it’s encrypted by ransomware or deleted maliciously. The remedy is the same, snapshots.

 

[Chris Moore]

It doesn’t really matter if it’s encrypted by ransomware or deleted maliciously. The remedy is the same, snapshots.

@supercoolfool pleas don't think I am trying to beat you down about this, but snapshots are really super cool...
At work, I had a user that accidentally deleted a folder from the server that had about 100GB of project data in it. They were almost in tears over it, but I was able to mount the snapshot to a folder so they could step back in time and get their files. That is one of the other beautiful things about snapshots. You can mount them read-only and access it like a historical archive.
I have my system at work setup to make a snapshot every 15 minutes and keep four of them, AND make a snapshot every hour and keep 24 of them, AND make a snapshot every day and keep 31 of them, AND make a snapshot every month and keep 12. It gives me really nice ability to go back for a user when they make a catastrophic mistake, or even for something malicious.

 

[supercoolfool]

Thx for all your help gents, much appreciated. If you ever find the author of Hermes 2.1 ransomware, I would appreciate a "heads up"...I'll buy you a coffee/beer...