SOLVED Dataset showing used space, no data is findable, data needs to be recovered

[BeYoung4Ever]

Hi there,
I have been scrounging the forum and have not been able to make any headway with my issue so far.
I have a main dataset called "Cerberus". In that I have the sub-dataset "plex-share" which contains more than it should. It currently has 2.49TB of data. I cannot see or access the data in SMB, Shell, or in SSH (as root or any other user).
The data is showing as "Used" space, I just cant see or access it with any account, including root. (See attached pic).
I need to recover the data if possible as it included family photos and videos.

Hardware and Software Info​

TrueNAS-13.0-RELEASE
Motherboard: Not sure and not easy to get at.
CPU: Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz
RAM: 16GB
Disks: 1x Crucial CT250MX200SSD1 running as Boot Drive; 2x Western Digital RED 4TB 7200RPM in MIRROR config; and 1x Western Digital RED 4TB 7200RPM that is available to be added when needed.
All Drives are using SATA to the motherboard.
Networking is running off Motherboard NIC.

Issue Details​

(*** is the dataset I am struggling with)

Code:

root@Cerberus[~]# zfs list -o space
    NAME                                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHI                                LD
    Cerberus                                                   1.00T  2.51T       56K     96K             0B      2.5                                1T
    Cerberus/.system                                           1.00T  2.81G     1.56G    852M             0B       42                                7M
    Cerberus/.system/configs-0c028a00ad64496ebc42e91622efb170  1.00T  29.8M        0B   29.8M             0B                                         0B
    Cerberus/.system/configs-5b73e8a039ec468ca0e6b8af17a87983  1.00T  36.4M     1.23M   35.1M             0B                                         0B
    Cerberus/.system/cores                                     1024M   256K      168K     88K             0B                                         0B
    Cerberus/.system/rrd-0c028a00ad64496ebc42e91622efb170      1.00T  11.1M        0B   11.1M             0B                                         0B
    Cerberus/.system/rrd-5b73e8a039ec468ca0e6b8af17a87983      1.00T   330M      314M   15.5M             0B                                         0B
    Cerberus/.system/samba4                                    1.00T  5.99M     5.58M    420K             0B                                         0B
    Cerberus/.system/services                                  1.00T   176K       80K     96K             0B                                         0B
    Cerberus/.system/syslog-0c028a00ad64496ebc42e91622efb170   1.00T    88K        0B     88K             0B                                         0B
    Cerberus/.system/syslog-5b73e8a039ec468ca0e6b8af17a87983   1.00T  13.3M     7.61M   5.66M             0B                                         0B
    Cerberus/.system/webui                                     1.00T   160K       64K     96K             0B                                         0B
    Cerberus/Data                                              1.00T    96K        0B     96K             0B                                         0B
    Cerberus/iocage                                            1.00T  12.5G     15.3M   9.06M             0B      12.                                4G
    Cerberus/iocage/download                                   1.00T   774M       64K     96K             0B       77                                4M
    Cerberus/iocage/download/12.1-RELEASE                      1.00T   371M       80K    371M             0B                                         0B
    Cerberus/iocage/download/12.2-RELEASE                      1.00T   402M        0B    402M             0B                                         0B
    Cerberus/iocage/images                                     1.00T   160K       64K     96K             0B                                         0B
    Cerberus/iocage/jails                                      1.00T  8.75G      136K     96K             0B      8.7                                5G
    Cerberus/iocage/jails/PlexMediaServer                      1.00T  8.75G     1.19M    304K             0B      8.7                                5G
    Cerberus/iocage/jails/PlexMediaServer/root                 1.00T  8.75G     4.89G   3.86G             0B                                         0B
    Cerberus/iocage/log                                        1.00T   300K      200K    100K             0B                                         0B
    Cerberus/iocage/releases                                   1.00T  2.94G       64K     96K             0B      2.9                                4G
    Cerberus/iocage/releases/12.1-RELEASE                      1.00T  1.48G       64K     96K             0B      1.4                                8G
    Cerberus/iocage/releases/12.1-RELEASE/root                 1.00T  1.48G     27.2M   1.46G             0B                                         0B
    Cerberus/iocage/releases/12.2-RELEASE                      1.00T  1.46G        0B     96K             0B      1.4                                6G
    Cerberus/iocage/releases/12.2-RELEASE/root                 1.00T  1.46G        8K   1.46G             0B                                         0B
    Cerberus/iocage/templates                                  1.00T   160K       64K     96K             0B                                         0B
    Cerberus/jails                                             1.00T  5.28G       64K    112K             0B      5.2                                8G
    Cerberus/jails/.warden-template-pluginjail-11.0-x64        1.00T   546M     7.08M    539M             0B                                         0B
    Cerberus/jails/.warden-template-standard-11.0-x64          1.00T  1.84G     69.2M   1.77G             0B                                         0B
    Cerberus/jails/plexmediaserver_1                           1.00T  2.90G     77.3M   2.83G             0B                                         0B
*** Cerberus/plex-share                                        1.00T  2.49T     2.49T    144K             0B                                         0B
    boot-pool                                                   199G  9.85G        0B     96K             0B      9.8                                5G
    boot-pool/ROOT                                              199G  9.83G        0B     96K             0B      9.8                                3G
    boot-pool/ROOT/12.0-U2.1                                    199G   324K        0B    324K             0B                                         0B
    boot-pool/ROOT/12.0-U5.1                                    199G   292K        0B    292K             0B                                         0B
    boot-pool/ROOT/12.0-U6                                      199G   312K        0B    312K             0B                                         0B
    boot-pool/ROOT/12.0-U7                                      199G   316K        0B    316K             0B                                         0B
    boot-pool/ROOT/12.0-U8.1                                    199G   316K        0B    316K             0B                                         0B
    boot-pool/ROOT/13.0-RELEASE                                 199G  9.83G     8.33G   1.50G             0B                                         0B
    boot-pool/ROOT/Initial-Install                              199G     8K        0B      8K             0B                                         0B
    boot-pool/ROOT/default                                      199G   440K        0B    440K             0B                                         0B

Code:

root@Cerberus[~]# ls -l /mnt/Cerberus
    total 42
    drwxrwxrwx+ 2 root  wheel   3 May 21  2022 Data
    drwxr-xr-x  9 root  wheel  12 Feb 16  2021 iocage
    drwxr-xr-x  8 root  wheel   8 Feb  2  2021 jails
    drwxrwxrwx+ 2 root  wheel   3 May 21  2022 plex-share

Code:

root@Cerberus[~]# ls -l /mnt/Cerberus/plex-share/
    total 1
    -rwxrwxrwx+ 1 plex  wheel  36 May 21  2022 Test.txt

There should be 2.5TB of data (folders, pictures, videos, excels, word docs, & pdfs) here... but the I am only able to see this one "Test.txt" file that is a few KB that I made about a year ago now.

Code:

root@Cerberus[~]# getfacl /mnt/Cerberus/plex-share/
    # file: /mnt/Cerberus/plex-share/
    # owner: root
    # group: wheel
                owner@:rwxpDdaARWcCos:-------:allow
                owner@:rwxpDdaARWcCos:fdi----:allow
                group@:rwxpDdaARWcCos:-------:allow
                group@:rwxpDdaARWcCos:fdi----:allow
             everyone@:rwxpDdaARWcCos:fd-----:allow
             everyone@:--------------:fd-----:allow

I would greatly appreciate any help with this and if there is anything that will help with troubleshooting, just let me know!

 

[Arwen]

Please supply the output of the following commands in code tags:
zfs list -t all -r Cerberus/plex-share zfs get all Cerberus/plex-share
It appears you have all your data in a snapshot. But, it could be that the dataset is not mounted.

 

[BeYoung4Ever]

Hi Arwen,

I appreciate you reaching out. Here are the results of those 2 commands:

Code:

root@Cerberus[~]# zfs list -t all -r Cerberus/plex-share
NAME                                                     USED  AVAIL     REFER  MOUNTPOINT
Cerberus/plex-share                                     2.49T  1.00T      144K  /mnt/Cerberus/plex-share
Cerberus/plex-share@plex-share manual-2021-03-14_16-40     0B      -     2.26T  -
Cerberus/plex-share@Cerberus manual-2021-03-14_16-40       0B      -     2.26T  -
Cerberus/plex-share@manual-2021-10-02_10-57              237G      -     2.49T  -
Cerberus/plex-share@manual-2023-04-15_22-02               80K      -      144K  -
Cerberus/plex-share@Cerberus-2023-04-16_00-00              0B      -      144K  - 

Code:

root@Cerberus[~]# zfs get all Cerberus/plex-share
NAME                 PROPERTY                 VALUE                     SOURCE
Cerberus/plex-share  type                     filesystem                -
Cerberus/plex-share  creation                 Sat Jun 16 21:40 2018     -
Cerberus/plex-share  used                     2.49T                     -
Cerberus/plex-share  available                1.00T                     -
Cerberus/plex-share  referenced               144K                      -
Cerberus/plex-share  compressratio            1.02x                     -
Cerberus/plex-share  mounted                  yes                       -
Cerberus/plex-share  quota                    none                      local
Cerberus/plex-share  reservation              none                      local
Cerberus/plex-share  recordsize               128K                      default
Cerberus/plex-share  mountpoint               /mnt/Cerberus/plex-share  default
Cerberus/plex-share  sharenfs                 off                       default
Cerberus/plex-share  checksum                 on                        default
Cerberus/plex-share  compression              lz4                       inherited from Cerberus
Cerberus/plex-share  atime                    off                       inherited from Cerberus
Cerberus/plex-share  devices                  on                        default
Cerberus/plex-share  exec                     on                        default
Cerberus/plex-share  setuid                   on                        default
Cerberus/plex-share  readonly                 off                       default
Cerberus/plex-share  jailed                   off                       default
Cerberus/plex-share  snapdir                  hidden                    default
Cerberus/plex-share  aclmode                  restricted                local
Cerberus/plex-share  aclinherit               passthrough               inherited from Cerberus
Cerberus/plex-share  createtxg                66                        -
Cerberus/plex-share  canmount                 on                        default
Cerberus/plex-share  xattr                    on                        default
Cerberus/plex-share  copies                   1                         local
Cerberus/plex-share  version                  5                         -
Cerberus/plex-share  utf8only                 off                       -
Cerberus/plex-share  normalization            none                      -
Cerberus/plex-share  casesensitivity          sensitive                 -
Cerberus/plex-share  vscan                    off                       default
Cerberus/plex-share  nbmand                   off                       default
Cerberus/plex-share  sharesmb                 off                       default
Cerberus/plex-share  refquota                 none                      local
Cerberus/plex-share  refreservation           none                      local
Cerberus/plex-share  guid                     3731335851710786568       -
Cerberus/plex-share  primarycache             all                       default
Cerberus/plex-share  secondarycache           all                       default
Cerberus/plex-share  usedbysnapshots          2.49T                     -
Cerberus/plex-share  usedbydataset            144K                      -
Cerberus/plex-share  usedbychildren           0B                        -
Cerberus/plex-share  usedbyrefreservation     0B                        -
Cerberus/plex-share  logbias                  latency                   default
Cerberus/plex-share  objsetid                 83                        -
Cerberus/plex-share  dedup                    off                       default
Cerberus/plex-share  mlslabel                 none                      default
Cerberus/plex-share  sync                     standard                  default
Cerberus/plex-share  dnodesize                legacy                    default
Cerberus/plex-share  refcompressratio         1.01x                     -
Cerberus/plex-share  written                  0                         -
Cerberus/plex-share  logicalused              2.55T                     -
Cerberus/plex-share  logicalreferenced        67K                       -
Cerberus/plex-share  volmode                  default                   default
Cerberus/plex-share  filesystem_limit         none                      default
Cerberus/plex-share  snapshot_limit           none                      default
Cerberus/plex-share  filesystem_count         none                      default
Cerberus/plex-share  snapshot_count           none                      default
Cerberus/plex-share  snapdev                  hidden                    default
Cerberus/plex-share  acltype                  nfsv4                     default
Cerberus/plex-share  context                  none                      default
Cerberus/plex-share  fscontext                none                      default
Cerberus/plex-share  defcontext               none                      default
Cerberus/plex-share  rootcontext              none                      default
Cerberus/plex-share  relatime                 off                       default
Cerberus/plex-share  redundant_metadata       all                       default
Cerberus/plex-share  overlay                  on                        default
Cerberus/plex-share  encryption               off                       default
Cerberus/plex-share  keylocation              none                      default
Cerberus/plex-share  keyformat                none                      default
Cerberus/plex-share  pbkdf2iters              0                         default
Cerberus/plex-share  special_small_blocks     0                         default
Cerberus/plex-share  org.freenas:description  SMB and Plex DATA         local
Cerberus/plex-share  org.freebsd.ioc:active   yes                       inherited from Cerberus

 

[Arwen]

It appears that your dataset is mounted, (and empty from your prior file listing);
Cerberus/plex-share mounted yes -

The good news is that your snapshot;
NAME USED AVAIL REFER MOUNTPOINT Cerberus/plex-share@manual-2021-10-02_10-57 237G - 2.49T -
contains much, (if not all), of the data.

You can make the snapshot visible using;
zfs set snapdir=visible Cerberus/plex-share

Then examine the data using;
cd /mnt/Cerberus/plex-share/.zfs/snapshot/manual-2021-10-02_10-57 ls -laR

Note that ZFS snapshots are Read / Only. In theory, you can roll back to that snapshot. But, I would make sure you have a backup of your files, which may be now visible in the ZFS snapshot.

 

[BeYoung4Ever]

I am able to see all the data in shell!
Just to make sure I don't accidentally nuke something, how should I make that backup of my files?
Should I roll back to that Snapshot using the TrueNAS interface or is there some shell scripting I should do?

Thank you for your help so far!

 

[BeYoung4Ever]

I should mention that below is the proof that I am able to see the folders in that Snapshot.

Code:

root@Cerberus[....zfs/snapshot/manual-2021-10-02_10-57]# ls
ADMIN                   Movies                  Personal Data           Shows
MediaID.bin             Music                   Pictures and Videos

 

[Arwen]

I don't know how you would do backups, they tend to be person, skill level and site specific.

Rolling back the snapshot has the inherent problem of HOW the files got lost in the first place. If you roll the dataset back to the snapshot, but the problem that caused the files to be deleted is not solved, you could loose your files for good. In theory, you could make another snapshot after rolling back the dataset. But, their is a window of when they are available as R/W.

Anyway, I can't make decisions for you, (my brain is getting tired... been up most of the night). Perhaps someone else can advise you further.

 

[BeYoung4Ever]

Thank you so much for your help!
Id been trying for over 1 year to get that data back!
I am in the process of copying my data to an external drive before rebuilding my system on TrueNAS Scale.

For anyone finding this thread later, I had enough storage on my system to set the Snapshot up as another Dataset.
I then set the permissions to allow my TrueNAS user account to access the data and set up a SMB share with its permissions.
Boom, access to the data again.

 

[Arwen]

Glad you solved the problem.

ZFS snapshots are great for many things, like accidental file deletes and some ransomware file encryption. And I sometimes use snapshots to look at an older version of a file and see what changed.