Huge Issue after ZFS Pool Upgrade

[DavidHasEdge]

Well my output from zdb -ul isn't anything I can work with

[root@freenas ~]# zdb -ul /mnt/Media
--------------------------------------------
LABEL 0
--------------------------------------------
failed to read label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to read label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to read label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to read label 3

If I could actually find a transaction ID to work with then I'd give it a shot. I'm not really afraid of losing the pool. The thing I was most concerned about was my Plex metadata, i.e. things I've watched, haven't watched, ect. That's safe and will work fine when I repopulate my TV shows.

Out of curiosity, is removing a jail and it deleting the hard link attached to it expected or would that be a bug?

 

[DavidHasEdge]

You said you had snapshots. Can't you just clone them and recover from the clone?

I used a snapshot to move my Jails, different ZFS pool

 

[DavidHasEdge]

However, I see what you might be saying, would a snapshot of my Jails pool also get the hard link that was there for /mnt/Media/TV ?

 

[Apollo]

I was not thinking about that, but it could be worth inverstigating.
I was just wondering if your multimedia dataset had by any chance snaphots prior to moving the jails. Even with the files deleted, if your snapshot is present you can retrieve the files.

 

[DavidHasEdge]

I was not thinking about that, but it could be worth inverstigating.
I was just wondering if your multimedia dataset had by any chance snaphots prior to moving the jails. Even with the files deleted, if your snapshot is present you can retrieve the files.

No, no snapshots on Media. Should have been... but I didn't see where I was doing anything that "touched" this pool. However, you live, you learn.

 

[Apollo]

Even if you don't perform replication for backup purpose, it is a very good practice to have periodic snapshots taken. It would have preserved your files, even deleted they would still be there on the volume.

 

[DavidHasEdge]

Yeah, I will make sure I have snapshots before any future changes. I have one I just made, just in case.

Thanks to everyone who tried to help. It's really not a huge deal, truth be told I needed to purge some TV shows that were never going to be watched again anyhow. I'm more thankful that there's not some issue with my raid like I originally suspected.

 

[pschatz100]

Out of curiosity, is removing a jail and it deleting the hard link attached to it expected or would that be a bug?

If you deleted your jails from the CLI, then the behavior most likely worked as expected. You didn't just delete the link - you also deleted the directories the link pointed to. You could read up on managing links to learn more.

My heart goes out to you. I had exactly the same experience when I moved some jails for the first time. In my case, however, I had a backup and was able to recover the data. Of course, this thread isn't about the wisdom of making backups - is it?

I hope you can get your data back. Since you say it is not critical data, then consider this a lesson learned. I have a tendency to mess things up, so I have learned a few lessons along the way:

1) Always use the Web interface when possible. Unless you know what you're doing, the CLI can cause lots of grief.
2) Do not use scripts or follow instructions posted on the forum unless you take the time to understand what they do.
3) Backup all data that matters. These back ups should also include configuration and meta data from your jails if that information would be difficult or time consuming to recreate (I back up my Plex meta data on a regular basis.)

 

[DavidHasEdge]

If you deleted your jails from the CLI, then the behavior most likely worked as expected. You didn't just delete the link - you also deleted the directories the link pointed to. You could read up on managing links to learn more.

My heart goes out to you. I had exactly the same experience when I moved some jails for the first time. In my case, however, I had a backup and was able to recover the data. Of course, this thread isn't about the wisdom of making backups - is it?

I hope you can get your data back. Since you say it is not critical data, then consider this a lesson learned. I have a tendency to mess things up, so I have learned a few lessons along the way:

1) Always use the Web interface when possible. Unless you know what you're doing, the CLI can cause lots of grief.
2) Do not use scripts or follow instructions posted on the forum unless you take the time to understand what they do.
3) Backup all data that matters. These back ups should also include configuration and meta data from your jails if that information would be difficult or time consuming to recreate (I back up my Plex meta data on a regular basis.)

The only CLI steps I took were to replicate the snapshot of my old jails pool to my new jails pool. I did delete some jails afterwards, but all through the GUI.

 

[DavidHasEdge]

And it definitely is a lessoned learned about how FreeNAS or FreeBSD in general handles hard links. I've been tinkering with *nix distros for nearly 15 years and had some Solaris classes in college. I'm no expert by any means but I'm pretty comfortable with the CLI and given some MAN pages for reference. This was a new one for me, definitely.

 

[ToBeFrank]

Can someone explain what we're talking about with the hard link here? I want to make sure I don't make the same mistake in the future.

 

[Bidule0hm]

In FreeBSD (and other UNIX based OS) you can make symbolic links and hard links on a file.

A symbolic link is a bit like a shorcut icon on Windows for exemple, it's not the file itself but a shortcut to the file, if you delete it the file isn't deleted.

But you've also the hard links, with them it's like if you've copied the file (but it's not the case, you've only one file) so if you delete the link it's like if you delete the file.

You can find more detailed and precise answers if you search a bit, for example here ;)

 

[ToBeFrank]

I know what symbolic and hard links are. I'm not understanding how deleting a jail would delete files and where the hard link comes in to the picture. Who created the hard link, and why is this particular hard link the only reference left to the data? What exactly happened here?

EDIT: I'm not understanding how deleting a jail would delete files *outside of the jail*.

 

[Bidule0hm]

Oh ok, I've not enough experience with the jails to answer that so I'll let a more experienced member do it ;)

 

[jkh]

In FreeBSD (and other UNIX based OS) you can make symbolic links and hard links on a file.
A symbolic link is a bit like a shorcut icon on Windows for exemple, it's not the file itself but a shortcut to the file, if you delete it the file isn't deleted.
But you've also the hard links, with them it's like if you've copied the file (but it's not the case, you've only one file) so if you delete the link it's like if you delete the file.
You can find more detailed and precise answers if you search a bit, for example here ;)

As others have already said, there is some information missing here. A hard link just increments the link count on the target file, so if you have a file with 4 hard links to it, its link count will be 5 (because the original file is also a reference) and you can delete any 4 of those links and the file will still be accessible through the remaining hard link (in point of fact, a hard link is indistinguishable from the "original" - they're all just directory entries referencing a file by its inode #). So, for files to be destroyed, something else must have happened.

 

[Bidule0hm]

Yes, I forgot to specify that, sorry.

 

[DavidHasEdge]

What exactly happened to me was I had jails, for example, Plex. Plex jail had links pointed to my media (TV and Movies) on my ZFS Pool. These links were made the standard way, though the gui, adding storage to the jail. My jails were on a different ZFS pool that I wanted to migrate to an SSD. I migrated the jails but some had issues so I decided to delete the jail and recreate it. Deleting the jail (through the gui) with the storage still attached also deleted all of my TV directory located on my ZFS pool. For some reason, my Movies were not deleted.

 

[ToBeFrank]

The fact that your storage was on a different pool clued me in that something still didn't make sense because you can't hard link across file systems. I took a look at the freenas and pcbsd source. Indeed, there are no hard links involved. It mounts your storage on a mount point. After some looking around in the source, I discovered that a bug was introduced in freenas on 10/16/14 that made it NOT safe to delete a jail that has storage attached. I have filed bug #8445.

 

[DavidHasEdge]

Thanks for your research, hopefully this helps some people out.

 

[fracai]

The source to deletejail.sh only uses 'rm -rf' to delete a jail if it isn't a ZFS dataset. If it is a dataset it uses 'zfs destroy', which wouldn't modify data on another dataset, and 'rmdir' which wouldn't remove a directory that isn't empty.

The previous posts seem to indicate the jails were ZFS, so there's either another bug here or it was operator error that hasn't been uncovered yet.