Huge Issue after ZFS Pool Upgrade

[DavidHasEdge]

OK so I did some things to my freenas box last night that I've been meaning to do. Moved my jails to an ssd, everything is working fine, decide to finally upgrade my media ZFS pool to get it to stop notifying me of the new features available.

Well, after this upgrade my pool, which was well over 9TB used out of 10.4 TB now reports that its only 8.2TB total with 4.6TB used. After digging into it, I've found my TV shows folder is empty. All my movies appear to be there. All of the disks in my RaidZ are there, none failed, its not degraded. They are all 4TB disks

I'm at a loss at what to do.

Of course no backup of this, I don't have enough storage or money to buy storage to backup 5TB of TV shows. If its gone then I'll start rebuilding but with my ISPs new data caps it will take nearly a year.

 

[DavidHasEdge]

I'm on Freenas 9.3 by the way. Upgraded to it a few weeks ago and had not upgraded my pool yet

 

[joeschmuck]

Post "zpool list" and "zpool status".

EDIT: Might as well add "zfs list" too.

 

[DavidHasEdge]

Post "zpool list" and "zpool status".

EDIT: Might as well add "zfs list" too.

The pool that's having the issues is Media

 

[DavidHasEdge]

 

[joeschmuck]

Where did you have your TV shows located?

 

[DavidHasEdge]

/mnt/Media/TV

The TV folder is still there... But no sub folders.

 

[SweetAndLow]

It looks to me that you probably accidentally deleted your tv shows when you where messing with your jails. Doing a zfs upgrade shouldn't delete data. How did you move your jails and what did you do with the mounted storage.

 

[DavidHasEdge]

It looks to me that you probably accidentally deleted your tv shows when you where messing with your jails. Doing a zfs upgrade shouldn't delete data. How did you move your jails and what did you do with the mounted storage.

I did a snapshot to replicate to my ssd... I used the commands posted by Dusan in this thread https://forums.freenas.org/index.php?threads/relocate-jails-to-ssd-helping-hdd-sleep.16955/

I don't know how I would accidently delete them. How does it explain the differences in size of my array? Before the upgrade I had 10.4 TB avail.

 

[cyberjock]

I'll tell you what you probably did.

When you deleted the jail the hard link to the TV shows directory was active, so you deleted all the directories and subdirectories under it. Plenty of people have done that without realizing it.

 

[DavidHasEdge]

And I was actually using over 9tb.. I've only got 8.2tb possible to use now.

 

[SweetAndLow]

And I was actually using over 9tb.. I've only got 8.2tb possible to use now.

You have 8.2TB available space left not total. Add up your used space and your available space to get your total space before parity. With a raidz1 with 4TB disk you should have 10.66TB usable space after parity and remember to try to keep that under 80% full which is 8.5TB

 

[DavidHasEdge]

Ok, I'm going to give zpool import -nfFX a shot. I know that simulates a transaction rollback, to finalize it I would run -fFX correct? Any other tips for this?

 

[cyberjock]

Uhh... that's not gonna do what you think it does. You need to roll back to a specific transaction (assuming you even can). doing that command is gonna make things much worse for you.

 

[DavidHasEdge]

That doesn't commit any changes without the -n being removed though, right?

 

[cyberjock]

n simulates the given outcome, but doesn't actually DO the outcome.

-f forces the pool (you shouldn't have to do this)
X and x are rollback tools that look for potential corruption and such. They will NOT do what oyu think they will do.

You're going to have to debug zfs to find a transaction to rollback to, and roll back to that transaction manually.

Yes, this is somewhat dangerous. It's very likely you'll break things and it's not something I recommend people do (especially people that think that they can figure it out despite never actually doing it).

You're better off writing the deleted stuff off and retrieving it again.

This is NOT something where you run a command or two and the data magically reappears.

This *IS* something that is time constrained. If you wait too long you'll find you can't even rollback far enough to get the data back. I've never been able to successfully roll back more than about 12 hours, so if its been more than 12 hours, you've pretty much lost the data for good.

Like I said.. that command does NOT do what you think it does.

 

[DavidHasEdge]

n simulates the given outcome, but doesn't actually DO the outcome.

-f forces the pool (you shouldn't have to do this)
X and x are rollback tools that look for potential corruption and such. They will NOT do what oyu think they will do.

You're going to have to debug zfs to find a transaction to rollback to, and roll back to that transaction manually.

Yes, this is somewhat dangerous. It's very likely you'll break things and it's not something I recommend people do (especially people that think that they can figure it out despite never actually doing it).

You're better off writing the deleted stuff off and retrieving it again.

This is NOT something where you run a command or two and the data magically reappears.

This *IS* something that is time constrained. If you wait too long you'll find you can't even rollback far enough to get the data back. I've never been able to successfully roll back more than about 12 hours, so if its been more than 12 hours, you've pretty much lost the data for good.

Like I said.. that command does NOT do what you think it does.

At the longest, this may have been 8 hours ago that I did any actually tinkering with Jails. I did delete a couple since permissions were funny after my move, I assume this is where the hard link debacle occurred.

I'm fine with risking the whole thing, like I said a bunch of movies and tv shows are hardly anything that's actually valuable, the whole pool dies then the whole pool dies.

If you have any advice, having gone down this path before, I'd appreciate it. Writing off the loss and just re downloading I would accept if there wasn't still a "chance".

Thanks for your help.

 

[DavidHasEdge]

To get the zfs transactions, I think I need to run "zdb -ul /mnt/Media" perhaps I have that syntax wrong, though?

 

[cyberjock]

Sorry, rolling back transactions is very complicated and not something I am willing to try to discuss. It would take me some hours to explain how to do it. I'm just not willing to spend that kind of investment on this. It's less of a hassle for you to just get the TV shows again. The changes of not only screwing your pool up, but also not getting the TV shows back is quite high.

I just don't recommend or endorse you going this path, but it is your path and your time.

When I did rollbacks for people in the past, I gave a general timeframe of about 10 hours to do the testing and rollbacks. It is much more complicated than you might think, and there's lots of ways to screw yourself for good.

Good luck.

 

[Apollo]

You said you had snapshots. Can't you just clone them and recover from the clone?