Jail upgrade trashed media directory- now what?

[csmuda]

After receiving one of the "Welcome to nginx" messages on my 8.3.0 NAS, I decided to upgrade. Rather than going for 8.3.0-p1, I installed 8.3.1b2. This went fine, and following this I upgraded the 8.3.1-b2 jail.

This... did not go fine. Beta sting! My media directory outside the jail (though mounted within the jail as /media) immediately emptied out, with several hundred GB of red suddenly turning to green on my main data mount. Everything inside my media directory is now gone, though items on the root -not- in the media directory remain. As near as I can tell, the jail upgrade decided to create a new, empty /media directory, but did so with the external mount intact, trashing the folder.

OK, back to a snapshot- wait, why are there no snapshots? Oh, god, snapshots are off. There are no snapshots.

So, aside from quickly turning on snapshots, I have not written to the drive. All the data, consisting of audio and video files, is presumably there, mirrored on two drives, but not in the filesystem. What would be my next step? Photorec? Can it be run directly from the FreeNAS distribution? Or are there other things I should be testing/checking first?

My current plan would be to photorec from the affected drive to an external USB, resort everything tagged back to the media directory, and manually sort out the rest. And to make sure snapshots are on make sure snapshots are on make sure snapshots are on. Does this make sense?

Thanks

 

[joeschmuck]

You know, I'd restore the files from your latest backup if you have one, Photorec could take forever, assuming it works.

I'm glad I read your story, I can't believe upgrading the jail would have caused this problem and a trouble ticket needs to be created to document this serious problem, assuming you didn't make some sort of mistake. I will try to recreate this problem in a VM.

 

[ProtoSD]

This was a known bug in a previous version of FreeNAS that was fixed. It looks like it might have found its way back. I seriously doubt photorec will work, ZFS has a different file structure.

 

[joeschmuck]

So I ran the test myself to see if I could verify the problem and here are my results...

I was unable to get the failure unless I did things out of order.

1) Loaded 8.3.0-Release and related Jails Plugin PBI.
2) Added mount point which pointed to an external media folder in my pool and to \media in the jail.
3) I then upgraded FreeNAS to verion 8.3.1-Beta2 using the GUI upgrade path without issue.
4) I then upgraded the Jails Plugin PBI to the 8.3.1-Beta2 version. The media is intact.

I then tried something a bit different, I reverted my VM to where it was as step 2 above and instead of upgrading to FreeNAS 8.3.1-Beta2, I upgraded the Jails Plugin PBI. This is when the media disappeared.

I ran the test twice to validate my work so at this point in time it appears to be operator error with loading the incorrect Jails Plugin for the version of software running.

@csmuda
If you believe I am incorrect, please create a ticket to report this problem.

Here is the problem report I created for this problem based on my test results.
http://support.freenas.org/ticket/2024

 

[ProtoSD]

The original bug appeared in 8.2.0 Beta 2:

http://forums.freenas.org/showthread.php?6223-Plugins-on-8-2-0-BETA-2

 

[csmuda]

So I ran the test myself to see if I could verify the problem and here are my results...

I was unable to get the failure unless I did things out of order.

1) Loaded 8.3.0-Release and related Jails Plugin PBI.
2) Added mount point which pointed to an external media folder in my pool and to \media in the jail.
3) I then upgraded FreeNAS to verion 8.3.1-Beta2 using the GUI upgrade path without issue.
4) I then upgraded the Jails Plugin PBI to the 8.3.1-Beta2 version. The media is intact.

I then tried something a bit different, I reverted my VM to where it was as step 2 above and instead of upgrading to FreeNAS 8.3.1-Beta2, I upgraded the Jails Plugin PBI. This is when the media disappeared.

I ran the test twice to validate my work so at this point in time it appears to be operator error with loading the incorrect Jails Plugin for the version of software running.

@csmuda
If you believe I am incorrect, please create a ticket to report this problem.

Here is the problem report I created for this problem based on my test results.
http://support.freenas.org/ticket/2024

I will verify when I return home, but pretty sure the 8.3.1b2 upgrade completed as planned... Regardless, thanks for the ticket for the reverse case, this is not good behavior for a NAS.

 

[cyberjock]

Not to discredit or otherwise demean anyone, but there are alot of people that have problems with jails resulting in lost data. I'm not sure I've seen even one case where it wasn't operator error. When I first saw this thread my first thought was "operator error.. somehow".

I'm not sure anyone has actually evaluated the consequences of running the wrong jail version, so this will definitely be interesting. It may warrant a warning somewhere of the consequences. Normally I think the problem is bad permissions, deleting the mountpoint from the jail's CLI(thereby deleting the contents of the mountpoint.. oops).

 

[joeschmuck]

Not to discredit or otherwise demean anyone, but there are alot of people that have problems with jails resulting in lost data. I'm not sure I've seen even one case where it wasn't operator error. When I first saw this thread my first thought was "operator error.. somehow".

I'm not sure anyone has actually evaluated the consequences of running the wrong jail version, so this will definitely be interesting. It may warrant a warning somewhere of the consequences. Normally I think the problem is bad permissions, deleting the mountpoint from the jail's CLI(thereby deleting the contents of the mountpoint.. oops).

The ticket I submitted recommends that anytime you upgrade the jail pbi that the mount points all be disabled until the upgrade is complete. I would think that could be easily performed but then again, I'm not doing the programming. If that cannot be done automatically, maybe a check to see if there are any active mount points and a huge warning telling you to delete them or face the possibility of data loss. To me this is a critical problem if there is an opportunity to accidentally cause data loss through an upgrade, yes even knowing the user may have gotten it wrong. Hey, that is where version checking could come in as well.

Myself, I always reload from scratch on a clean USB drive and then load my config file, then automount the pool, install the plugins and lastly setup the mount points. That is probably why I've never seen this problem before.

 

[csmuda]

Not to discredit or otherwise demean anyone, but there are alot of people that have problems with jails resulting in lost data. I'm not sure I've seen even one case where it wasn't operator error. When I first saw this thread my first thought was "operator error.. somehow".

Ultimately, all cases of problems can be explained as operator error in some way-- but robust systems have less opportunity for operator error. However, I'm now sure that whatever error I may have made in this case definitely wasn't upgrading the jail prior to upgrading the software. (It was not enabling snapshots.)

I was concerned that I'd messed up the upgrade and not noticed, and I'd come home to find the system still at 8.3.0 or something, but it is proudly displaying the appropriate version (FreeNAS-8.3.1-BETA2-x64 (r13015)), and I definitely sent the firmware upgrade over and successfully rebooted prior to the jail update. I don't know if I hit a weird edge case or what, but the system upgrade came first. This is concerning.

 

[joeschmuck]

From my point of view as you stated, the software needs to be more robust, however since you stated that you know you upgraded properly, then that makes it even more important for the jails upgrade to be modified to prevent this error from ever happening again.

 

[William Grzybowski]

I would like to point out that upgrade a jail is totally useless for people that have plugins running and should be avoided at all costs because it does not bring any advantages.

 

[cyberjock]

I would like to point out that upgrade a jail is totally useless for people that have plugins running and should be avoided at all costs because it does not bring any advantages.

Then why even release updates? Just release a "this is the jail" file and leave it at that...

 

[William Grzybowski]

Of course there are changes in the jail, I just did not want to get into details, we are not that stupid :D

The meaning of the message is to scare people to upgrade jails because there is no need to, unless something is not working.

 

[cyberjock]

Of course there are changes in the jail, I just did not want to get into details, we are not that stupid :D

The meaning of the message is to scare people to upgrade jails because there is no need to, unless something is not working.

Ah, ok. I gotcha. I didn't think you were stupid, just really wondered what to think when upgrades to the jail were being made. :P

 

[joeschmuck]

William,
Thank you for updating the code to dismount the mount points. I'm not sure why you couldn't replicate the problem but that's a moot point now.

I didn't know that we do not need upgrade the jails plugin when we upgrade FreeNAS from a previous version. I was always under the impression that the jail must be updated unless there was a note in the readme file which stated there was no change between the versions and it was not necessary. In absence of something stating there was no change, I'm certain all users will load the version which is grouped with the FreeNAS version on the server.

I'm looking forward to testing Beta3 but I really didn't have any issues with Beta2, at least what I look for that is.