FreeNAS 11.2 update killed data

[wlagarde]

DATA LOST.

I upgraded from 11.1-U6.3 to 11.2 inside GUI after saving config. Install crashed. Reinstalled via boot CD created with downloaded 11.2 ISO. Restored config. After restore, there are many directories missing from zpool (RAID Z2). I had a backup of the pool on a single large disk and the data is missing there too. zpool was not upgraded. Reverted back to 11.1-U6.3 and data remain missing.

 

[Apollo]

@wlagarde, are you certian you lost your data? I would understand you could have reinstalled over the wrong disk but with redundancy that would not be an issue in your case.
Is it possible what you believe to be dataset were actually regular folders. I have seen this recently. Unless you go and explore your pool with "mc", you can't really be sure, especially if you have snapshots in place.
I would say the same for your backup drive. If you know for sure the data was copied over or better yet, replicated, then I would be looking at the possibility your dataset set as readonly and as such would not be necessarely mounted. You can verify that by listing the snapshots and searching for the one that are associated with your missing datasets.

 

[fracai]

When did you make the backup? Was the backup connected during the upgrade?
The data missing on your storage and backup sounds like it was missing when you made the backup and the upgrade has nothing to do with it.

 

[wlagarde]

No question all data was present before upgrade and definitely didn't' install over it (The boot/install disk is a 64GB intel SSD while the pool and backup disks are WD Red 4TB and 8TB drives). All disks were connected including backup disk (was an Rsysnc copy of the main RAIDZ2 zpool). The datasets are there and have been imported; however, some of the directories and files within the datasets are there while some are missing. The good news is that the missing directory/files were my "Movies" and 'Music" folders. I can restore my music from my iPhone and the re-rip the movies.

 

[wlagarde]

When did you make the backup? Was the backup connected during the upgrade?
The data missing on your storage and backup sounds like it was missing when you made the backup and the upgrade has nothing to do with it.

Backup was made as standing daily RSync task night before upgrade. The data were there on both the main pool and the backup drive (I checked before the upgrade)

 

[wlagarde]

I just tried detaching the pools and reimporting them and then remounting the shares and the missing directories/data are still missing. What is interesting is that all of the folders on the backup drive are missing while on the primary zpool the "Movies" and "Music" folders are missing,

 

[wlagarde]

Just SSH'd in via Cyberduck and explored the mnt points and while the datasets are mounted, the missing folders are not there.

 

[Apollo]

Can you see anything related to you dataset using the following command:
zpool history your_volume_name_here

If the dataset existed and has been destroyed, if should show in the list.

and also:
zfs diff your_volume_name_here@snashot-date-here

If the dataset or folder in the relevant dataset pointed by the snapshot has been modified, added or deleted, it should appear in the list. If you know which dataset and snapshot were assigned before and after the event you should be able to see what has happened.

 

[Apollo]

Another thought, has the allocated space on you pool with the missing dataset/folders changed? Did you recover space or it the available space been recovered by a similar amount or more of yoour "Movies and "Music" content. If it is not the case, then I think the data is still there, just out of reach for reason unknown.

 

[wlagarde]

I

Another thought, has the allocated space on you pool with the missing dataset/folders changed? Did you recover space or it the available space been recovered by a similar amount or more of yoour "Movies and "Music" content. If it is not the case, then I think the data is still there, just out of reach for reason unknown.

Not running snapshots. Allocated space is unchanged. Attached is zpool history.

.

 

[Ericloewe]

Please review this thread: https://forums.freenas.org/index.php?threads/lost-data-in-upgrade-to-11-2.71585/
And this bug report: https://redmine.ixsystems.com/issues/64512

Also, tell us as much as you can about your system. If possible, include a debug tarball. Before you do that, though, please clarify with the output of zfs list: How much space were you using before and how much space are you using after your data disappeared. (If the answer is "the same in both cases" you didn't lose data and have a dataset mounted on top of a directory or vice-versa due to a misconfiguration.)

 

[wlagarde]

I had more than 50% of a 12TB pool utilized or ~5-6TB used (5x4TB WD Red drives in RAIDZ2 configuration) and I am now seeing 1.95TB...

 

[Apollo]

@wlagarde, the part I am puzzled about in you screenshot if the "zfs send -R RAIDZ2/jails@relocate" without the "zfs receive" command being followed.
After which the "jails" dataset is being destroyed recursively which means that all its descendant children dataset are going to be destroyed as well.

Could it be the source of your problem?

 

[wlagarde]

Motherboard: AS Rock E3C224D4I-14S
Processor: Xeon E3-1240v3 3.4GHz
Memory: 32GB ECC (4x8GB)
Boot Volume: Intel X25-E 64GB
RAIDZ2 Volume (WD Red 4TBx5) with 2 datasets: Shared (main pool) and Backup (time machine backups)
WD Red 8TB second volume with 1 dataset: Archive.

 

[Apollo]

I had more than 50% of a 12TB pool utilized or ~5-6TB used (5x4TB WD Red drives in RAIDZ2 configuration) and I am now seeing 1.95TB...
The allocated sie of 1.95TB is only for RAIDZ2/Shared. It may not describe the entire Pool alloaction as each dataset may have been assigned Quotas, or simply a datset where descentent dataset exist, the parent will provide the cumulative allocated space for the descendant only.

If that makes sense to you.

 

[wlagarde]

@wlagarde, the part I am puzzled about in you screenshot if the "zfs send -R RAIDZ2/jails@relocate" without the "zfs receive" command being followed.
After which the "jails" dataset is being destroyed recursively which means that all its descendant children dataset are going to be destroyed as well.

Could it be the source of your problem?

I had my plexmediaserver pointing to the "Movies" folder and the "TV Shows" folders; however, the TV Shows are there while the "Music: and "Movies" folders are missing. Also to clarify I made no changes to my jails.

 

[Ericloewe]

I had more than 50% of a 12TB pool utilized or ~5-6TB used (5x4TB WD Red drives in RAIDZ2 configuration) and I am now seeing 1.95TB...
View attachment 27532

You need to get the output from a proper shell with a scrollback or, at the very least, increase the size of the webshell to the maximum.

 

[wlagarde]

...or, at the very least, increase the size of the webshell to the maximum.

How is that done?

 

[Ericloewe]

The 80 columns by 25 lines drop-down list at the bottom left of the webshell.

 

[wlagarde]

Here it is: