Lost all files in /mnt/tank/Media after upgrade from FreeNAS to TrueNAS

[MicahPotter]

Hi everyone,

Last night I ran a TrueNAS upgrade on my FreeNAS mini using a USB with TrueNAS 12.0 U8 STABLE. After the upgrade, everything in my tank/Media mount is missing. This was many years of family photos and videos. Seriously need to figure out what happened and how to recover the data.

Anyone have any ideas why this might have happened? Or what I can do to recover this directory.

It appears to be just Media that was affected. I still have lose files laying around in /mnt/tank and everything in /mnt/tank/iocage appears to be there also.

Any help would be greatly appreciated.

 

[sretalla]

Maybe we should start with zfs list | grep Media

And zpool status -v

 

[MicahPotter]

Code:

root@freenas[~]# zfs list | grep Media
tank/Media                                              198K  7.50T      198K  /mnt/tank/Media
root@freenas[~]#

Code:

root@freenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      ada2p2    ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:45:25 with 0 errors on Sun Apr 10 00:45:26 2022
checkpoint: created Wed Apr 13 05:05:19 2022, consumes 32.8M
config:

    NAME                                            STATE     READ WRITE CKSUM
    tank                                            ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/886743ee-49bd-11e4-9967-d050992ed0dc  ONLINE       0     0     0
        gptid/893c6890-49bd-11e4-9967-d050992ed0dc  ONLINE       0     0     0
        gptid/8a103587-49bd-11e4-9967-d050992ed0dc  ONLINE       0     0     0
        gptid/8adf315c-49bd-11e4-9967-d050992ed0dc  ONLINE       0     0     0

errors: No known data errors
root@freenas[~]#

 

[Arwen]

Yes, you do seem to have a problem. With only 198K bytes used in "tank/Media", the files either are elsewhere or gone. Perhaps they were in another dataset of similar name, like "tank/media", (lower case). Or "tank/medias"...

Let us assume they are elsewhere. Please give us the full output of zfs list -t all -r tank.

Further. about how much space would these Media files take up?

You don't list the size of the disks, so please do so as well.

 

[MicahPotter]

Code:

root@freenas[~]# zfs list -t all -r tank
NAME                                                                                          USED  AVAIL     REFER  MOUNTPOINT
tank                                                                                              281G   7.50T     9.25G  /mnt/tank
tank/.bhyve_containers                                                               70.4M  7.50T     70.4M  /mnt/tank/.bhyve_containers
tank/.system                                                                                109M   7.50T      209K  legacy
tank/.system/configs-115df71ff6e542c88acafe634faf995a        24.1M  7.50T     24.1M  legacy
tank/.system/configs-adb946163d914f088dc14617dbc0bec3  38.4M  7.50T      38.4M  legacy
tank/.system/cores                                      1                              63K     1024M      163K  legacy
tank/.system/rrd-115df71ff6e542c88acafe634faf995a               26.0M  7.50T      26.0M  legacy
tank/.system/rrd-adb946163d914f088dc14617dbc0bec3         209K    7.50T      209K  legacy
tank/.system/samba4                                                                  3.09M  7.50T      2.55M  legacy
tank/.system/samba4@wbc-1649820596                                   262K      -           2.50M  -
tank/.system/samba4@wbc-1649823463                                   279K      -            2.53M  -
tank/.system/services                                                                  140K  7.50T      140K  legacy
tank/.system/syslog-115df71ff6e542c88acafe634faf995a          5.84M  7.50T     5.84M  legacy
tank/.system/syslog-adb946163d914f088dc14617dbc0bec3   10.2M  7.50T     10.2M  legacy
tank/.system/webui                                                                     198K  7.50T      198K  legacy
tank/Media                                                                                   198K  7.50T      198K  /mnt/tank/Media
tank/Ubuntu-le4l6                                                                         124G  7.62T     81.4K  -
tank/iocage                                                                                   148G  7.50T     8.30M  /mnt/tank/iocage
tank/iocage/download                                                                  289M  7.50T      128K  /mnt/tank/iocage/download
tank/iocage/download/11.3-RELEASE                                         137M  7.50T      137M  /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/download/11.4-RELEASE                                         152M  7.50T      152M  /mnt/tank/iocage/download/11.4-RELEASE
tank/iocage/images                                                                       145G  7.50T     4.84G  /mnt/tank/iocage/images
tank/iocage/images/kube-21f4do                                                  141G  7.63T     4.48G  -
tank/iocage/jails                                                                             839M  7.50T      128K  /mnt/tank/iocage/jails
tank/iocage/jails/haproxy                                                               839M  7.50T      163K  /mnt/tank/iocage/jails/haproxy
tank/iocage/jails/haproxy/root                                                       839M  7.50T     1.42G  /mnt/tank/iocage/jails/haproxy/root
tank/iocage/log                                                                              140K  7.50T      140K  /mnt/tank/iocage/log
tank/iocage/releases                                                                      1.01G  7.50T      128K  /mnt/tank/iocage/releases
tank/iocage/releases/11.3-RELEASE                                             414M  7.50T      128K  /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.3-RELEASE/root                                      414M  7.50T      414M  /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.4-RELEASE                                             618M  7.50T      128K  /mnt/tank/iocage/releases/11.4-RELEASE
tank/iocage/releases/11.4-RELEASE/root                                     618M  7.50T      616M  /mnt/tank/iocage/releases/11.4-RELEASE/root
tank/iocage/releases/11.4-RELEASE/root@haproxy                     1.92M      -      616M  -
tank/iocage/templates                                                                   128K  7.50T      128K  /mnt/tank/iocage/templates
root@freenas[~]#

Code:

root@freenas[~]# zpool list
NAME        SIZE    ALLOC   FREE    CKPOINT   EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool  14.5G  1.41G     13.1G        -               -                    1%       9%     1.00x      ONLINE     -
tank           10.9T   29.3G     10.8T     35.5M         -                     6%       0%    1.00x      ONLINE     /mnt
root@freenas[~]#

Code:

Disks
ada2 E021434540100000003B 14.91 GiB boot-pool
ada0 WD-WMC4N2505247      2.73  TiB tank
ada1 WD-WMC4N2496562      2.73  TiB tank
ada3 WD-WMC4N2583291      2.73  TiB tank
ada4 WD-WMC4N2495917      2.73  TiB tank

 

[MicahPotter]

Yes, you do seem to have a problem. With only 198K bytes used in "tank/Media", the files either are elsewhere or gone. Perhaps they were in another dataset of similar name, like "tank/media", (lower case). Or "tank/medias"...

I'm certain the dataset was tank/Media. I took a config backup before the upgrade and have tried to restore from that hoping it would help... but no luck.

 

[Kris Moore]

You sure you didn't have them in another dataset by mistake this whole time? The "tank/iocage/images" directory is 145G, which seems a bit suspect.

A long shot is also that perhaps the /mnt/tank/Media directory was never mounted and you were writing files to /mnt/tank/Media directory directly (On the root of the zpool). You can try running "umount /mnt/tank/Media" to see if there are files in that sub-dir after unmounting the dataset.

 

[MicahPotter]

You sure you didn't have them in another dataset by mistake this whole time? The "tank/iocage/images" directory is 145G, which seems a bit suspect.

A long shot is also that perhaps the /mnt/tank/Media directory was never mounted and you were writing files to /mnt/tank/Media directory directly (On the root of the zpool). You can try running "umount /mnt/tank/Media" to see if there are files in that sub-dir after unmounting the dataset.

Hi Kris, the images directory has the 128GB ubuntu disk image in it.

I tried unmounting to see if there were files under the empty mounted dataset, but no luck...

Code:

root@freenas[~]# umount /mnt/tank/Media
root@freenas[~]# cd /mnt/tank
root@freenas[/mnt/tank]# ll
total 115
drwxrwx---+ 8 root  wheel  uarch    10       Apr 13 04:29 ./
drwxr-xr-x    3 root  wheel  uarch     128     Apr 13 04:46 ../
drwxrwx---+ 2 root  wheel  uarch     9        Apr 12 22:02 .AppleDB/
drwxrwx---+ 4 root  wheel  uarch     4        Apr  5  2020 .bhyve_containers/
-rwxrwx---+ 1 root  wheel  uarch     14340 Apr 13 05:38 .DS_Store*
drwxrwx---+ 8 root  wheel  uarch    11       Apr  4  2020 .freenas/
-rwxrwx---+ 1 root  wheel  uarch     0        Jun 28  2020 .windows*
drwxrwx---+ 2 root  wheel  uarch    11       Apr 12 20:51 Downloads/
drwxrwx---+ 9 root  wheel  uarch    11       Apr 12 20:30 iocage/
drwxrwx---+ 2 root  wheel  uarch     2        Apr 13 04:29 Media/
root@freenas[/mnt/tank]# ll Media
total 36
drwxrwx---+ 2 root  wheel  uarch  2 Apr 13 04:29 ./
drwxrwx---+ 8 root  wheel  uarch 10 Apr 13 04:29 ../
root@freenas[/mnt/tank]#

Are there any procedures for recovering lost files on a zfs vol?

 

[winnielinnie]

I'm suspecting it's what @Kris Moore is pointing to. There was a confusion between a directory and a dataset.

Try unmounting, like he said, and see if that "reveals" the Media folder with the contents inside. (Inside the folder.)

And why is "iocage" so large? Did you save multimedia directly into a jail without a mountpoint?

 

[sretalla]

Are there any procedures for recovering lost files on a zfs vol?

Purchase Klennet and attach your drives to a Windows machine to run it... not ideal. (also, depending on what you did to lose the files, perhaps only a long shot) At least you can see if you can recover before you have to pay.


Separately to that, I don't think you can un-mount a zfs mount without using zfs unmount

Not sure why the regular umount command thinks it can do that without throwing an error, but there you go, maybe it works.

 

[MicahPotter]

Separately to that, I don't think you can un-mount a zfs mount without using zfs unmount

Thanks, Sretalla, I've tried unmounting with both and Media is still empty.

 

[anodos]

If you know the file type you're looking for, you can do something like find /mnt/tank/ -type f -name '*.jpg' which will print out all files ending in .jpg (case-sensitive) on mounted filesystems under /mnt/tank (in case you were writing to an unexpected place). This may take a while to complete.

You can also find out what dataset is mounted under a particular path by running the command:
df -T /mnt/tank/Media to see whether the dataset is the expected one.

 

[Patrick M. Hausen]

@sretalla You can do e.g. mount -t zfs zroot/some/dataset /mnt just fine in FreeBSD. And umount, if you like.

 

[winnielinnie]

I tried unmounting to see if there were files under the empty mounted dataset, but no luck...

What is the output of: zfs mount

Also, what is the output of zfs list -r -t filesystem -o name,usedsnap tank

 

[MicahPotter]

What is the output of: zfs mount

Also, what is the output of zfs list -r -t filesystem -o name,usedsnap tank

here's the output of zfs mount `after` remounting Media....

Code:

root@freenas[~]# zfs mount
boot-pool/ROOT/default                                                              /
tank                                                                                              /mnt/tank
tank/.bhyve_containers                                                               /mnt/tank/.bhyve_containers
tank/iocage                                                                                  /mnt/tank/iocage
tank/iocage/jails                                                                          /mnt/tank/iocage/jails
tank/iocage/images                                                                    /mnt/tank/iocage/images
tank/iocage/download                                                                /mnt/tank/iocage/download
tank/iocage/log                                                                          /mnt/tank/iocage/log
tank/iocage/jails/haproxy                                                          /mnt/tank/iocage/jails/haproxy
tank/iocage/releases                                                                 /mnt/tank/iocage/releases
tank/iocage/download/11.3-RELEASE                                      /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/templates                                                               /mnt/tank/iocage/templates
tank/iocage/download/11.4-RELEASE                                     /mnt/tank/iocage/download/11.4-RELEASE
tank/iocage/jails/haproxy/root                                                  /mnt/tank/iocage/jails/haproxy/root
tank/iocage/releases/11.3-RELEASE                                       /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.4-RELEASE                                       /mnt/tank/iocage/releases/11.4-RELEASE
tank/iocage/releases/11.3-RELEASE/root                               /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.4-RELEASE/root                               /mnt/tank/iocage/releases/11.4-RELEASE/root
tank/.system                                                                             /var/db/system
tank/.system/cores                                                                   /var/db/system/cores
tank/.system/samba4                                                               /var/db/system/samba4
tank/.system/syslog-115df71ff6e542c88acafe634faf995a       /var/db/system/syslog-115df71ff6e542c88acafe634faf995a
tank/.system/rrd-115df71ff6e542c88acafe634faf995a             /var/db/system/rrd-115df71ff6e542c88acafe634faf995a
tank/.system/configs-115df71ff6e542c88acafe634faf995a      /var/db/system/configs-115df71ff6e542c88acafe634faf995a
tank/.system/webui                                                                    /var/db/system/webui
tank/.system/services                                                                 /var/db/system/services
tank/Media                                                                                  /mnt/tank/Media
root@freenas[~]#

Code:

root@freenas[~]# zfs list -r -t filesystem -o name,usedsnap tank
NAME                                                                                                 USEDSNAP
tank                                                                                                     0B
tank/.bhyve_containers                                                                       0B
tank/.system                                                                                        0B
tank/.system/configs-115df71ff6e542c88acafe634faf995a                0B
tank/.system/configs-adb946163d914f088dc14617dbc0bec3          0B
tank/.system/cores                                                                              0B
tank/.system/rrd-115df71ff6e542c88acafe634faf995a                       0B
tank/.system/rrd-adb946163d914f088dc14617dbc0bec3                 0B
tank/.system/samba4                                                                         552K
tank/.system/services                                                                         0B
tank/.system/syslog-115df71ff6e542c88acafe634faf995a                0B
tank/.system/syslog-adb946163d914f088dc14617dbc0bec3           0B
tank/.system/webui                                                                            0B
tank/Media                                                                                         0B
tank/iocage                                                                                        0B
tank/iocage/download                                                                       0B
tank/iocage/download/11.3-RELEASE                                              0B
tank/iocage/download/11.4-RELEASE                                              0B
tank/iocage/images                                                                            0B
tank/iocage/jails                                                                                  0B
tank/iocage/jails/haproxy                                                                    0B
tank/iocage/jails/haproxy/root                                                            0B
tank/iocage/log                                                                                   0B
tank/iocage/releases                                                                           0B
tank/iocage/releases/11.3-RELEASE                                                  0B
tank/iocage/releases/11.3-RELEASE/root                                          0B
tank/iocage/releases/11.4-RELEASE                                                  0B
tank/iocage/releases/11.4-RELEASE/root                                          1.92M
tank/iocage/templates                                                                         0B
root@freenas[~]#

 

[MicahPotter]

I appreciate your help everyone. This is all pretty frustrating. All my kids pictures from the last 10 years are just gone like that.

No changes were made to the disks, pool, or the datasets. It was just a simple update where I booted from a USB drive to install TrueNAS on the 16GB flash media drive. It came back up and looked fine until I noticed from my phone that the Media directory was empty, then looked on the server to find nothing there.

 

[winnielinnie]

Right off the bat, it looks like the data is not tucked away in a snapshot.

However, it also appears that tank/Media is still mounted.

You need to 100% confirm that tank/Media is not currently mounted before ruling out the original idea, posted by @Kris Moore, in which you might have saved files directly into the folder (not dataset) of /mnt/tank/Media.

EDIT: Nevermind. I missed this part:

here's the output of zfs mount `after` remounting Media....

 

[winnielinnie]

Another thing that can clue us in:
zfs list -t filesystem -o name,usedds tank

EDIT: Outside of this, I'm beyond confused, and actually scared. Simply upgrading from FreeNAS 11.x to TrueNAS 12.x emptied the contents of an entire dataset? I don't even see how that's possible. Did you at any point create a snapshot of Media? How did you populate Media? Via an SMB share?

 

[MicahPotter]

Right off the bat, it looks like the data is not tucked away in a snapshot.

However, it also appears that tank/Media is still mounted.

You need to 100% confirm that tank/Media is not currently mounted before ruling out the original idea, posted by @Kris Moore, in which you might have saved files directly into the folder (not dataset) of /mnt/tank/Media.

Thanks winnielinni,

I've confirmed it's unmounted and rechecked the directory...

Code:

root@freenas[~]# ls -al /mnt/tank/Media
total 36
drwxrwx---+ 2 root  wheel   2 Apr 11 21:25 .
drwxrwx---+ 8 root  wheel  10 Apr 13 04:29 ..
root@freenas[~]# zfs unmount tank/Media
root@freenas[~]# echo "Again to show it's unmounted..."
Again to show it's unmounted...
root@freenas[~]# zfs unmount tank/Media
cannot unmount 'tank/Media': not currently mounted
root@freenas[~]# ls -al /mnt/tank/Media
total 36
drwxrwx---+ 2 root  wheel   2 Apr 13 04:29 .
drwxrwx---+ 8 root  wheel  10 Apr 13 04:29 ..
root@freenas[~]#

 

[MicahPotter]

Another thing that can clue us in:
zfs list -t filesystem -o name,usedds tank

Code:

root@freenas[~]# zfs list -t filesystem -o name,usedds tank
NAME  USEDDS
tank   9.25G
root@freenas[~]#