SOLVED Revert a snapshot rollback

[will.mhtz]

Just in between doing a lot of file work and a backup (and snapshot), I needed to move files between datasets using new datasets that were just created. Long story short, I rolled back to a snapshot that was created before all these new files were created as well as the datasets. My new issue is that I need to undo that snapshot rollback. The new datasets "exist" within freenas, but all have lost permissions. The primary goal is to get back everything before the snapshot was rolled back. I can fix the empty datasets later. Any help would be appreciated.

Will


FreeNAS-11.0-U2 (e417d8aa5)
RaidZ1

 

[gpsguy]

Did you read the docs regarding "rollback" before you did it? http://doc.freenas.org/11/storage.html#snapshots

From the docs "The most recent snapshot also has a Rollback Snapshot icon. Clicking the icon asks for con1rmation before rolling back to this snapshot state. Confirming by clicking Yes causes any files that have changed since the snapshot was taken to be reverted back to their state at the time of the snapshot."

My new issue is that I need to undo that snapshot rollback

 

[will.mhtz]

Did you read the docs regarding "rollback" before you did it? http://doc.freenas.org/11/storage.html#snapshots

From the docs "The most recent snapshot also has a Rollback Snapshot icon. Clicking the icon asks for con1rmation before rolling back to this snapshot state. Confirming by clicking Yes causes any files that have changed since the snapshot was taken to be reverted back to their state at the time of the snapshot."

I am aware of how the rollback works. The issue is rolling back the rollback. Effectively returning everything to where it was at this morning. For some reason, brain.exe has stopped working and allowed me to do all this work without backing up. Any and all advice is helpful.

Will

 

[Stux]

I am aware of how the rollback works. The issue is rolling back the rollback. Effectively returning everything to where it was at this morning. For some reason, brain.exe has stopped working and allowed me to do all this work without backing up. Any and all advice is helpful.

Will

I'm not sure you can.

 

[will.mhtz]

I'm not sure you can.

Freenas never deletes anything, it just creates new copies. The files theoretically exist, just are not visible. If there was one really significant folder that was not in the the old snapshot (that is now the active running snapshot), but still exists in the hidden land of previous files (forgive me for not knowing what that area is called within ZFS), would that be possible to recover those files?
Thanks

 

[Stux]

I guess you can rollback the whole pool.

 

[will.mhtz]

(that is kind of the issue... the whole pool was rolled back. Rolled back by 2 days, but still.) I guess what I am hearing is that rolling back the rollback is not a viable option, let alone not even possible. I have accepted that. Is there any way to see the files that were active BEFORE I rolled back to the old snapshot.

 

[fracai]

When you rollback or revert to an older snapshot, you also destroy any snapshots that were created after the one you revert to. This means you cannot rollforward. The only files you'll be able to access at this point are those in the dataset and any captured by older snapshots.

In the future, if you think you might need to undo your rollback, you should instead clone the snapshot you want to rollback to. Or just copy files out of the .zfs/snapshots directory.

 

[Stux]

(that is kind of the issue... the whole pool was rolled back. Rolled back by 2 days, but still.) I guess what I am hearing is that rolling back the rollback is not a viable option, let alone not even possible. I have accepted that. Is there any way to see the files that were active BEFORE I rolled back to the old snapshot.

No. You rolled back a dataset to a previous snapshot. I believe that prunes all the future history.

BUT a pool consists of an unbroken chain of checkpoints. And I believe you can rollback to a recent checkpoint, and thus lose ALL transactions that have occurred since that checkpoint... including rolling back the dataset. and any restoring of files since etc.

 

[will.mhtz]

When you rollback or revert to an older snapshot, you also destroy any snapshots that were created after the one you revert to. This means you cannot rollforward. The only files you'll be able to access at this point are those in the dataset and any captured by older snapshots.

This was rolled back to the most recent snapshot. No other snapshots were created that were newer.

No. You rolled back a dataset to a previous snapshot. I believe that prunes all the future history.

Can you explain how the newly made datasets are still in place despite the upper level dataset being rolled back? Also how is it that there is ~500Gb more used space within that upper level dataset that were not there when the snapshot was created 2 days ago? That comes from some of the transfers between datasets that did not exist when the snapshot was created.

 

[fracai]

This was rolled back to the most recent snapshot. No other snapshots were created that were newer.

Right, but you still don't have anything to "undo" to.

Let's say over time you create snapshots A,B,C,D,E. You edit some files and then rollback to C. You now have snapshots A,B,C. You can't rollback, revert, or undo to D or E. In your scenario, you instead rollback to E. You still only have snapshots A,B,C,D,E. There is nothing to "undo" to.

Now, as Stux has suggested, there may be a way to rollback the pool transactions. I think I have heard stories of people doing this and actually recovering data. You have to use the zdb command and here be dragons. If someone can propose the methodology for doing so I suppose it's worth a shot.

Can you explain how the newly made datasets are still in place despite the upper level dataset being rolled back?

Each dataset is self contained. You can rollback a dataset to an older snapshot without affecting any child datasets, which would have their own set of snapshots.

Also how is it that there is ~500Gb more used space within that upper level dataset that were not there when the snapshot was created 2 days ago? That comes from some of the transfers between datasets that did not exist when the snapshot was created.

That suggests maybe you didn't lose data and it's still available somewhere?
Post the output of 'zpool list', 'zfs list -o space', and 'zfs list -t snap'. Use code tags when posting.

 

[will.mhtz]

I will send that out when I get home: currently in my other office. My fingers are crossed the transaction rollback may help. I have found some other documentation elsewhere that may support the idea that it is possible to some extent.
Thank you for your help!

 

[rs225]

Transaction rollback is more myth than fact. By the time you want to do it, it is too late.

Removing snapshot rollback from the GUI might be a good feature request. Instead, it should simulate it with clones.

 

[will.mhtz]

That suggests maybe you didn't lose data and it's still available somewhere?
Post the output of 'zpool list', 'zfs list -o space', and 'zfs list -t snap'. Use code tags when posting.

Code:

NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT												 
TheRaid	   2.72T  2.63T  87.2G		 -	59%	96%  1.00x  ONLINE  /mnt													 
freenas-boot  14.2G  1.46G  12.8G		 -	  -	10%  1.00x  ONLINE  -	  




NAME														   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD			
TheRaid														 143M  1.75T	 2.36G   1.32T			  0	   439G			
TheRaid/.system												 143M  13.3M	  554K	245K			  0	  12.5M			
TheRaid/.system/configs-9d613bc4d69d4caa9ab03b2439285b53		143M   639K	  522K	117K			  0		  0			
TheRaid/.system/cores										   143M   639K	  522K	117K			  0		  0			
TheRaid/.system/rrd-9d613bc4d69d4caa9ab03b2439285b53			143M  9.73M	 1012K   8.74M			  0		  0			
TheRaid/.system/samba4										  143M   778K	  522K	256K			  0		  0			
TheRaid/.system/syslog-9d613bc4d69d4caa9ab03b2439285b53		 143M   799K	  554K	245K			  0		  0			
TheRaid/ArchiveY												143M  1.75G		 0   1.75G			  0		  0			
TheRaid/BackupS												 143M   117K		 0	117K			  0		  0			
TheRaid/MediaE												  143M   117K		 0	117K			  0		  0			
TheRaid/StorageX												143M   436G		 0	436G			  0		  0			
TheRaid/jails												   143M  1.07G		 0	149K			  0	  1.07G			
TheRaid/jails/.warden-template-pluginjail					   143M   596M	 4.29M	592M			  0		  0			
TheRaid/jails/owncloud_1										143M   502M		 0	502M			  0		  0			
freenas-boot												   12.3G  1.46G		 0	 64K			  0	  1.46G			
freenas-boot/.system										   12.3G  8.45M		 0	242K			  0	  8.22M			
freenas-boot/.system/configs-9d613bc4d69d4caa9ab03b2439285b53  12.3G   193K		 0	193K			  0		  0			
freenas-boot/.system/cores									 12.3G   394K		 0	394K			  0		  0			
freenas-boot/.system/rrd-9d613bc4d69d4caa9ab03b2439285b53	  12.3G  6.37M		 0   6.37M			  0		  0			
freenas-boot/.system/samba4									12.3G  98.5K		 0   98.5K			  0		  0			
freenas-boot/.system/syslog-9d613bc4d69d4caa9ab03b2439285b53   12.3G  1.17M		 0   1.17M			  0		  0			
freenas-boot/ROOT											  12.3G  1.44G		 0	 29K			  0	  1.44G			
freenas-boot/ROOT/11.0-U2									  12.3G  1.44G	  736M	737M			  0		  0			
freenas-boot/ROOT/Initial-Install							  12.3G	 1K		 0	  1K			  0		  0			
freenas-boot/ROOT/default									  12.3G   138K		 0	138K			  0		  0			
freenas-boot/grub											  12.3G  6.46M		 0   6.46M			  0		  0  

all of the files were directly underneath the "TheRaid" directory. Some were moved to "TheRaid/StorageX" before things went south. The snapshot currently in place does not have any of the "archivey, backups, mediae, storagex, or jail" sub-directories. The other issue is that the enitre storage of the server is full due to the copy on write of zfs. Any advice?

 

[fracai]

ZFS COW isn't taking up all of your space. If it were, you'd see more data taken up in the USEDSNAP column. Snapshots are only taking up around 3 GB. Most of your space is taken up directly inside TheRaid and in TheRaid/StorageX (1.75 and .4 TB respectively). Do you mean that you copied data from TheRaid to StorageX so it exists in two places? That can be resolved by deleting data from one of the copies and then then destroying the snapshots.

Can you account for all of that ~2.5 TB spread across those two datasets? If not, could there be any data that is in a StorageX folder that is being shadowed by the dataset?

In general, my advice is to either clear out some space or upgrade your drives to larger sizes. As it is, you're over 90%. At a certain point ZFS changes the algorithm it uses to write data and you'll hit a performance wall. I suspect you've either crossed that or are approaching it.

If you really do have duplicate copies of files due to copying them from one dataset to another I suggest you clean that up. Keep in mind, the more actions you take make it less likely that you can rollback pool transactions successfully. Also note that I have no experience with pool transactions so I can't recommend anything there.

 

[rs225]

If the main problem is that the datasets aren't accessible, then perhaps just creating those directories and then zfs unmount TheRaid/StorageX and then zfs mount TheRaid/StorageX will work.

 

[will.mhtz]

If the main problem is that the datasets aren't accessible, then perhaps just creating those directories and then zfs unmount TheRaid/StorageX and then zfs mount TheRaid/StorageX will work.

That is exactly one of the problems causing the overflow of data. Yes, almost everything in the StorageX is a duplicate of whats underneath TheRaid including some of the newly modified files not included in the old snapshot that was rolled over. Some of the important modified data may have been copied to StorageX before the transfer failed.

Can you account for all of that ~2.5 TB spread across those two datasets? If not, could there be any data that is in a StorageX folder that is being shadowed by the dataset?

If by "shadowed" you mean not visable yet still within the hard drives, that would be correct. To my knowlege and as a result of the snapshot rollback, there is data hidden underneath TheRaid that is newer than that within the snapshot. That data could also be hidden underneath StorageX if that part of the transfer went through.

In general, my advice is to either clear out some space or upgrade your drives to larger sizes. As it is, you're over 90%. At a certain point ZFS changes the algorithm it uses to write data and you'll hit a performance wall. I suspect you've either crossed that or are approaching it.

My intention was never to reach that 90% barrier. It was the copy between datasets that duped the data due to ZFS copy on write.

Because of the snapshot rollback and that each of the datasets were underneath the "TheRaid" dataset. Rolling back TheRaid as a whole has caused havoc on the datasets underneath.

Thank you all for your continued help. Hopefully this horror story will be resolved soon and may help some other people if they unfortunately have this issue.

 

[Stux]

Sounds like maybe you don't want/need to revert your snapshot rollback...

Rather what you need to do is resolve your shaddowed folder/dataset issue right?

 

[will.mhtz]

Sounds like maybe you don't want/need to refer your snapshot rollback...

Rather what you need to do is resolve your shaddowed folder/dataset issue right?

Thats what I had thought, after I had already rolled back.

Thank you everyone for your help.

If the main problem is that the datasets aren't accessible, then perhaps just creating those directories and then zfs unmount TheRaid/StorageX and then zfs mount TheRaid/StorageX will work.

RS225, mounting and unmounting the StorageX dataset worked to bring StorageX back to usable. Fortunately, the data that had been backed up was the last thing to FULLY copy from TheRaid to StorageX. Resulting in all but 2 files to be recovered. I can make those photo edits again fortunately. The client in which these files needed to be delivered to will be very happy with the results.

Everything is backed up! Now I can restart the dataset changes and PROPERLY copy all the files between them WITH backups:);)