Folder Data Suddenly GONE... Please Help!

[{c}guy]

Hey gang... I've recently upgraded from Truenas Core to Scale, and everything has been working great, until today. I have several zfs pools on my server running proxmox 7.4x and have scale running in a vm with my controller card passed through. This setup has been working great for years.

Today while watching a TV show stored on my TANK zfs pool, the video stopped playing and froze. Upon closer inspection, all the data in my TV_Shows folder is gone! Nearly 20TB of shows... have all disappeared... the folder name is still there... but all data inside is gone. Looking around further, another folder called Downloads is also missing all of it's data. Again the folder name is there but no data.

I checked zfs pool status and everything shows no problems, and passed a recent scrub without issue.

Unfortunately I didn't have snapshots enabled for this dataset... am I screwed? Is there some other way to recover this data? I checked zpool history and history and no commands to delete files were issued.

I can't see anything in the logs that would indicate a problem.. just Poof! the data is missing..

Any help would be appreciated....

Thanks,

{c}guy.

 

[Whattteva]

Does the web UI also reflect this lost 20 TiB?

 

[samarium]

If you need to try and rollback txgs, then sooner you make the pool readonly the better.

I guessing that these folders are just folders, not datasets, otherwise zpool history, or zpool events -v might help.

 

[{c}guy]

Unfortunately, the web gui also shows the lost 20Tb... I can't find any reason or log showing a 'rm' command, but that seems like what happened.

 

[Whattteva]

Unfortunately, the web gui also shows the lost 20Tb... I can't find any reason or log showing a 'rm' command, but that seems like what happened.

This isn't a good sign. It probably means the files are gone for good. Well, not really for good, you may not want to do anymore writes to the pool if you want to have any hope of recovery at all. You may be able to recover it with a recovery tool that supports ZFS. I'm not aware of any as I've never had the need for it, but others here may.

Are there any other users with write access possibly blowing it all away?
How does the TV gain access to the movies? Does it possibly have write access? It's possible that the TV app went rogue and thrashed all your files.

 

[{c}guy]

This isn't a good sign. It probably means the files are gone for good. Well, not really for good, you may not want to do anymore writes to the pool if you want to have any hope of recovery at all. You may be able to recover it with a recovery tool that supports ZFS. I'm not aware of any as I've never had the need for it, but others here may.

Are there any other users with write access possibly blowing it all away?
How does the TV gain access to the movies? Does it possibly have write access? It's possible that the TV app went rogue and thrashed all your files.

Yeah, this folder was accessed by several other machines.., radarr, sonarr, and such as well as Plex. I suspect that something with one of those went rogue as you said... I'm going to try data recovery to see if I can pull it back. I did stop writes to that pool immediately when I saw what was happening. Hopefully I caught it in time.

I wish I knew which machine or program caused the error. That's the scary part.. whatever did this, seems to have left no trace.

 

[Patrick M. Hausen]

Sorry, I do not intend to come across as the wise guy, but periodic snapshots ... just do it. It's a no-brainer, really, and one of the reasons to pick ZFS over other filesystems. Sorry for your loss of data if that is the outcome.

 

[{c}guy]

Sorry, I do not intend to come across as the wise guy, but periodic snapshots ... just do it. It's a no-brainer, really, and one of the reasons to pick ZFS over other filesystems. Sorry for your loss of data if that is the outcome.

You are obviously correct, I did have them setup for all of my other datasets... somehow this one was missed. What can I say, I'm human... Will always have them enabled from now on.

 

[samarium]

Never had to do a rollback txg type recovery, so can't add much useful. You might get something useful out of zdb's many options and then zpool import with a txg rollback. That is as much as I can recollect from old reading of the documentation, probably not much of a hint though. Good luck finding some specific info, and with rollback.