What happened to my data?

[Milt]

I have a 40TB TrueNAS-SCALE-22.12.3.2 server with a 20TB smb-share. The share had 8TB-used/20TB. Today I notice there's only 1.1TB in the share. I see in Storage Reports that on Monday 1/29/24 between the hrs of 2:30pm-3pm over 6TB was lost? All datasets and drives appear to be running nominally with zero errors.

Can I recover the data if someone(Wife children..etc) accidentally deleted the data? Unfortunately I did not have any snapshots enabled on this share

Also how can I find out what actually happened between 2:30-3pm. Can I drop to the shell and look at logs to see if the data was "deleted" or... The crazy thing is there are 6 folders left out of 20-30 missing so why wasn't all the data deleted? Strange..Would like to recover and or find out what happened.

Please advise...

 

[Whattteva]

Without snapshots, probably not. Your best bet is a recovery service/software. I would stop any further use of the pool and take it offline. The longer you leave it up, the more data will be thrashed and make recovery even harder.

 

[Milt]

Sorry for delayed reply. I have question, can I drop to a shell and look at logs to tell me what happened? I mean I'd like to know specifically what occurred like a record of a "delete" command was issued.

 

[Whattteva]

Well, you can check their command history .zsh-histfile but that's assuming that they did it through the shell. I'm guessing that's unlikely cause they probably did it through the share protocol. I'm not sure if there is a way to figure out who did what, but this is exactly what permissions are for. To make sure people don't just randomly delete stuff they're not supposed to.

Hopefully someone else knows how to audit that information (if there is any).

 

[FrankWard]

I have a 40TB TrueNAS-SCALE-22.12.3.2 server with a 20TB smb-share. The share had 8TB-used/20TB. Today I notice there's only 1.1TB in the share. I see in Storage Reports that on Monday 1/29/24 between the hrs of 2:30pm-3pm over 6TB was lost? All datasets and drives appear to be running nominally with zero errors.

Can I recover the data if someone(Wife children..etc) accidentally deleted the data? Unfortunately I did not have any snapshots enabled on this share

Also how can I find out what actually happened between 2:30-3pm. Can I drop to the shell and look at logs to see if the data was "deleted" or... The crazy thing is there are 6 folders left out of 20-30 missing so why wasn't all the data deleted? Strange..Would like to recover and or find out what happened.

Please advise...

If you did not have a snapshot of the dataset, then no, you cannot recover the data with TrueNAS. No problem as long as you have a proper backup. You do have a proper backup right? If not, then you've just learned a tough lesson. 6TB is a lot to lose at once from a user mishap through an SMB share. The only read/write users have on my systems is within their own dataset which also has snapshots in case they wipe something out. Everything else is read-only to prevent issues like this, but it also snapshotted and backed up to an additional NAS just in case.

 

[Milt]

Well, you can check their command history .zsh-histfile but that's assuming that they did it through the shell. I'm guessing that's unlikely cause they probably did it through the share protocol. I'm not sure if there is a way to figure out who did what, but this is exactly what permissions are for. To make sure people don't just randomly delete stuff they're not supposed to.

Hopefully someone else knows how to audit that information (if there is any).

Thats a good point. Because it was deleted "remotely" through the share protocol there's nothing to find through the truenas shell. You are correct about permissions which I am implementing.

 

[Milt]

If you did not have a snapshot of the dataset, then no, you cannot recover the data with TrueNAS. No problem as long as you have a proper backup. You do have a proper backup right? If not, then you've just learned a tough lesson. 6TB is a lot to lose at once from a user mishap through an SMB share. The only read/write users have on my systems is within their own dataset which also has snapshots in case they wipe something out. Everything else is read-only to prevent issues like this, but it also snapshotted and backed up to an additional NAS just in case.

I'm still crying in my cornflakes over this. I'm smarter and skilled then this, got lazy. Good thing I have a second older NAS(RAID.built in 2015) but lost a couple yrs of new data. I did turn on snapshots on the share and noticed a "recursive" option? Will a snapshot of the share protect all of the folders/files within it or do I need the recursive option as well?

 

[Apollo]

I'm still crying in my cornflakes over this. I'm smarter and skilled then this, got lazy. Good thing I have a second older NAS(RAID.built in 2015) but lost a couple yrs of new data. I did turn on snapshots on the share and noticed a "recursive" option? Will a snapshot of the share protect all of the folders/files within it or do I need the recursive option as well?

Snapshot would have saved your data if done properly.
At the minimum, you can have a Recursive snapshot (snapshot creation at specific intervals, and adequate retention period) done at the pool level, and then implement more granular snapshot at the dataset level if needed.

 

[Milt]

Snapshot would have saved your data if done properly.
At the minimum, you can have a Recursive snapshot (snapshot creation at specific intervals, and adequate retention period) done at the pool level, and then implement more granular snapshot at the dataset level if needed.

Thank you for your suggestions! I'll have it all setup by tomorrow,

 

[FrankWard]

I'm still crying in my cornflakes over this. I'm smarter and skilled then this, got lazy. Good thing I have a second older NAS(RAID.built in 2015) but lost a couple yrs of new data. I did turn on snapshots on the share and noticed a "recursive" option? Will a snapshot of the share protect all of the folders/files within it or do I need the recursive option as well?

The recursive option only applies to child datasets. If the share itself was a dataset, then a snapshot of that dataset will protect it all. If there are child datasets within the share dataset, you'll need to use the recursive feature or snapshot the child dataset individually. It took me a build or two to figure out optimal dataset configuration for how I like to back things up. I've come to believe that if you ever think you'll need to go back in time for a specific folder, just make it a dataset so you can have the snapshot capability.

 

[Apollo]

Sorry for delayed reply. I have question, can I drop to a shell and look at logs to tell me what happened? I mean I'd like to know specifically what occurred like a record of a "delete" command was issued.

I was looking for an answer to this question, but without snapshots I couldn't find one.
However, if you have one or more snapshots, you can use the "diff" command, either with "zpool diff ..." or "zfs diff snapshot....".
You can read about the "zfs diff ..." in the openZFS documentation:
BTW, this is not going to tell you who or what command caused an operation to be executed, but it will tell you if a file has been added, removed or modified.

https://openzfs.github.io/openzfs-docs/man/master/8/zfs-diff.8.html

 

[Milt]

The recursive option only applies to child datasets. If the share itself was a dataset, then a snapshot of that dataset will protect it all. If there are child datasets within the share dataset, you'll need to use the recursive feature or snapshot the child dataset individually. It took me a build or two to figure out optimal dataset configuration for how I like to back things up. I've come to believe that if you ever think you'll need to go back in time for a specific folder, just make it a dataset so you can have the snapshot capability.

That's correct! I researched about truenas datasets and found the truenas documentation for "Creating Datasets". First para states what you're saying. Yeah the share is a child dataset and I have snapshots enabled on that child. I took the defaults 2wk retention @00:00 daily but that can eat up my volume as I try to rebuild from the old NAS. I have a 40TB pool and created a child dataset of 20T of which I had used 8TB. So probably snapshot every week not sure about retention. I also plan to setup an rsync with the old NAS until it dies. I have a couple large external drives as a backup.

 

[Milt]

I was looking for an answer to this question, but without snapshots I couldn't find one.
However, if you have one or more snapshots, you can use the "diff" command, either with "zpool diff ..." or "zfs diff snapshot....".
You can read about the "zfs diff ..." in the openZFS documentation:
BTW, this is not going to tell you who or what command caused an operation to be executed, but it will tell you if a file has been added, removed or modified.

https://openzfs.github.io/openzfs-docs/man/master/8/zfs-diff.8.html

Good to know! Now that I'm scared straight I plan to deep dive ZFS. Thank you for the link

 

[Apollo]

That's correct! I researched about truenas datasets and found the truenas documentation for "Creating Datasets". First para states what you're saying. Yeah the share is a child dataset and I have snapshots enabled on that child. I took the defaults 2wk retention @00:00 daily but that can eat up my volume as I try to rebuild from the old NAS. I have a 40TB pool and created a child dataset of 20T of which I had used 8TB. So probably snapshot every week not sure about retention. I also plan to setup an rsync with the old NAS until it dies. I have a couple large external drives as a backup.

Snapshot take a real small amount of space as they don't hold any data others than the list of the blocks it is referencing. So something like a few KB at most.
You can easily have in excess of 100'sK snapshots in a pool.
In the worse case, if you have too many snapshots and it is eating away space on your pool where you running out of space, then you can selectively delete the older snapshots, but you need to be more educated about ZFS overall.
In summary, you can have snapshots spanning for years.
And if you old NAS is running ZFS, then you can possibly handle replication to the new TrueNAS server.