99% of my home media dataset just vanished. Any way to determine root cause?

[Siress]

TrueNAS-SCALE-23.10-RC.1

I have a "media" dataset that's far too large to keep even a single snapshot - so I only have whatever was saved during my last off-site backup. Well, at least it was. I've been using these files through Jellyfin as recently as Wednesday without issue. This setup has been stable since RC.1 came out. The only files that remain on the dataset are empty folder structures - and only a tiny percentage of the folder structure at that...

Is there any way for me to find out what caused this?

 

[Siress]

I went through the available history within TrueNAS and only found one unusual blip over the past week. It coincided with another unique event on my network. Attached is the correlation of the ZFS ARC size dropping moments after I created an Home Assistant backup before initiating the a HA Restoration - something I had never done before, on a device I've only had a few days. The HA device had privilage to delete the "media" dataset because I was being lazy and reckless. Can anyone confirm this is sufficient information to determine root cause? No other dataset on my TrueNAS sees anywhere near as much use as the "media" dataset, so it's practically the only thing represented in these logs.

With root cause likely closed, can anyone help me 'un-delete' these files without access to a snapshot, or let me know that's entirely impossible? I'm hesitant to spend the next couple of weeks restoring the backup if the data is there with just a few bits toggled to indicate they're ready to be written over... This dataset is not encrypted.

 

[Davvo]

If something deleted those files, and you don't have a snapshot or a checkpoint, I don't think there are any simple ways to recover those files.
A backup restore would be preferable to a ZFS recovery service.

 

[winnielinnie]

I have a "media" dataset that's far too large to keep even a single snapshot

What do you mean by that? Snapshots don't take up any extra space, and simply "hold on" to the space represented by "deleted" records.

You could create a new snapshot every hour, and as long as you never delete files, all your snapshots combined will not add to your used space on the dataset.

 

[Siress]

Then I have fundamentally misunderstood snapshots. Thank you for the explanation.

That being the case, sounds like I'm reverting to the backup for this dataset. Last time I did a full backup took two weeks... guess I'm in for it now.

 

[Joe-freenas]

TrueNAS-SCALE-23.10-RC.1

I have a "media" dataset that's far too large to keep even a single snapshot - so I only have whatever was saved during my last off-site backup. Well, at least it was. I've been using these files through Jellyfin as recently as Wednesday without issue. This setup has been stable since RC.1 came out. The only files that remain on the dataset are empty folder structures - and only a tiny percentage of the folder structure at that...

Is there any way for me to find out what caused this?

This is either a hardware issue or operator error. Given the very vague details given, I believe/bet, the latter.

Mistake #1:
"have a dataset that's far too large to keep even a single snapshot".
Next time simply don't use ZFS and save money in memory.

Mistake #2:
REALLY important data is never in ONE place/backup only!

You can try paying 1-2K for a data recovery company, or, next time, use few hundred dollars to add either 3 bigger drives, or at least 2 more
drives of the same size, in order to have: 1 mirrored pool. 1 replication pool.

 

[Joe-freenas]

Then I have fundamentally misunderstood snapshots. Thank you for the explanation.

That being the case, sounds like I'm reverting to the backup for this dataset. Last time I did a full backup took two weeks... guess I'm in for it now.

2 weeks? Are you moving 1000 TB!?

CORRECTION: Snapshots CAN consume space, IF the data/file was changed!
i.e. you now have TWO versions of the SAME file, the old and the new one!
Depends on the amount of change or new data!

But if you are already filled beyond 70%, you need to either get bigger hard drives, or delete, distribute data to other drives...

 

[Siress]

For whatever reason, the transfer rate from my old Synolgoy is ~50 Mb/s, despite the network having 20-50x greater bandwidth and using rsync with any number of parallel instances. So, 8TB comes out to ~2 weeks.

 

[PhilD13]

The HA was likely the culprit due to human error on what it was allowed to do. I would make sure you have that issue sorted before you attempt to restore. Without snapshots you will need to restore from the Synology even if it's slow. Not sure why it is so slow, but you should be able to either push the files from Synology to TN with Rsync (might go faster?) or setup a Rsync pull in TN Data Protection tab. Yes it will take awhile but it should move faster than ~50MBs but as long as it works you are good.

The Snapshots do take some space but not a lot, if you add and delete files often as there is a differential between the last snap and current data to keep up with. I have a 20TB share that is heavily used and it only takes up about 385GB of space maximum for any one snapshot some very little.
I take a daily snapshot, and expire them every 7 days (Lifetime of) so I always have a fresh set of snapshots. I also do daily backups of shares to my QNAP