SOLVED Lost datasets after zfs send/receive

[Xyrgh]

Hi all,

I'm not sure what I've done. I was attempting to duplicate my 'scratch' pool over to my main tank to take my scratch pool offline and reconfigure it.

I performed the following command:

Code:

zfs snapshot -r scratch@migration_base_20190929
zfs send -Rv -i scratch@migration_base scratch@migration_base_20190929 | zfs receive -Fd tank

Some errors flashed up saying unable to unmount some datasets as they were busy, then it failed and I stupidly closed the terminal and tried a reboot. Into the GUI and now most of the datasets under 'tank' are missing. From the size it looks like the data is still there but I can't see the datasets.

Outpool of zpool status:

Code:

pool: tank
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 1 days 08:48:42 with 0 errors on Tue Sep  3 08:48:45 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/1d32e6e5-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/1e6292d1-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/1f9440c4-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/d55538fe-b7b8-11e8-af77-0cc47a816756  ONLINE       0     0     0
            gptid/21e6a774-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/230222c2-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/24290af0-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/25534abf-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/267d9825-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/27aabe93-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/28d867a7-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
            gptid/29f9a575-638a-11e6-9cfc-0cc47a816756  ONLINE       0     0     0
          raidz2-2                                      ONLINE       0     0     0
            gptid/a740a8e7-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0
            gptid/a9bfa172-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0
            gptid/ab07c654-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0
            gptid/ac44c721-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0
            gptid/ad7ea17a-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0
            gptid/aeb90022-f696-11e8-83d1-0cc47a816756  ONLINE       0     0     0

Output of zpool list tank:

Code:

NAME                                                    USED  AVAIL  REFER  MOUNTPOINT
apps                                                   97.8G   125G  29.8G  /mnt/apps
apps/.bhyve_containers                                 70.5M   125G  70.4M  /mnt/apps/.bhyve_containers
apps/Ubuntu-5f6t59p                                    24.2G   143G  5.87G  -
apps/iocage                                            22.9G   125G  4.41M  /mnt/apps/iocage
apps/iocage/download                                    532M   125G    88K  /mnt/apps/iocage/download
apps/iocage/download/11.1-RELEASE                       260M   125G   260M  /mnt/apps/iocage/download/11.1-RELEASE
apps/iocage/download/11.2-RELEASE                       272M   125G   272M  /mnt/apps/iocage/download/11.2-RELEASE
apps/iocage/images                                      144K   125G    88K  /mnt/apps/iocage/images
apps/iocage/jails                                      20.4G   125G    96K  /mnt/apps/iocage/jails
apps/iocage/jails/deluge                               1.39G   125G    96K  /mnt/apps/iocage/jails/deluge
apps/iocage/jails/deluge/root                          1.39G   125G  1.45G  /mnt/apps/iocage/jails/deluge/root
apps/iocage/jails/grafana                               311M   125G    92K  /mnt/apps/iocage/jails/grafana
apps/iocage/jails/grafana/root                          311M   125G  1.19G  /mnt/apps/iocage/jails/grafana/root
apps/iocage/jails/hoard                                64.3M   125G    92K  /mnt/apps/iocage/jails/hoard
apps/iocage/jails/hoard/root                           64.2M   125G  1019M  /mnt/apps/iocage/jails/hoard/root
apps/iocage/jails/jackett                               774M   125G    96K  /mnt/apps/iocage/jails/jackett
apps/iocage/jails/jackett/root                          774M   125G  1.43G  /mnt/apps/iocage/jails/jackett/root
apps/iocage/jails/lidarr                               1.79G   125G    96K  /mnt/apps/iocage/jails/lidarr
apps/iocage/jails/lidarr/root                          1.79G   125G  2.11G  /mnt/apps/iocage/jails/lidarr/root
apps/iocage/jails/nextcloud                            1.33G   125G    96K  /mnt/apps/iocage/jails/nextcloud
apps/iocage/jails/nextcloud/root                       1.33G   125G  1.68G  /mnt/apps/iocage/jails/nextcloud/root
apps/iocage/jails/plex                                 3.53G   125G    96K  /mnt/apps/iocage/jails/plex
apps/iocage/jails/plex/root                            3.53G   125G  3.50G  /mnt/apps/iocage/jails/plex/root
apps/iocage/jails/radarr                               1.94G   125G    96K  /mnt/apps/iocage/jails/radarr
apps/iocage/jails/radarr/root                          1.94G   125G  1.64G  /mnt/apps/iocage/jails/radarr/root
apps/iocage/jails/radarr4k                             1.06G   125G    96K  /mnt/apps/iocage/jails/radarr4k
apps/iocage/jails/radarr4k/root                        1.06G   125G  1.51G  /mnt/apps/iocage/jails/radarr4k/root
apps/iocage/jails/sonarr                               1.53G   125G    96K  /mnt/apps/iocage/jails/sonarr
apps/iocage/jails/sonarr/root                          1.53G   125G  1.52G  /mnt/apps/iocage/jails/sonarr/root
apps/iocage/jails/sonarrv3                              557M   125G    96K  /mnt/apps/iocage/jails/sonarrv3
apps/iocage/jails/sonarrv3/root                         557M   125G  1.48G  /mnt/apps/iocage/jails/sonarrv3/root
apps/iocage/jails/tautulli                             1.00G   125G    96K  /mnt/apps/iocage/jails/tautulli
apps/iocage/jails/tautulli/root                        1.00G   125G  1.32G  /mnt/apps/iocage/jails/tautulli/root
apps/iocage/jails/transmission                          546M   125G    96K  /mnt/apps/iocage/jails/transmission
apps/iocage/jails/transmission-seed                    1.78G   125G    96K  /mnt/apps/iocage/jails/transmission-seed
apps/iocage/jails/transmission-seed/root               1.78G   125G  1.73G  /mnt/apps/iocage/jails/transmission-seed/root
apps/iocage/jails/transmission/root                     545M   125G  1.45G  /mnt/apps/iocage/jails/transmission/root
apps/iocage/jails/tumblr                               1.18G   125G    96K  /mnt/apps/iocage/jails/tumblr
apps/iocage/jails/tumblr/root                          1.18G   125G  2.05G  /mnt/apps/iocage/jails/tumblr/root
apps/iocage/jails/unifi                                1.71G   125G    96K  /mnt/apps/iocage/jails/unifi
apps/iocage/jails/unifi/root                           1.71G   125G  2.64G  /mnt/apps/iocage/jails/unifi/root
apps/iocage/log                                         944K   125G   168K  /mnt/apps/iocage/log
apps/iocage/releases                                   1.92G   125G    88K  /mnt/apps/iocage/releases
apps/iocage/releases/11.1-RELEASE                       984M   125G    88K  /mnt/apps/iocage/releases/11.1-RELEASE
apps/iocage/releases/11.1-RELEASE/root                  984M   125G   973M  /mnt/apps/iocage/releases/11.1-RELEASE/root
apps/iocage/releases/11.2-RELEASE                       981M   125G    88K  /mnt/apps/iocage/releases/11.2-RELEASE
apps/iocage/releases/11.2-RELEASE/root                  981M   125G   968M  /mnt/apps/iocage/releases/11.2-RELEASE/root
apps/iocage/templates                                   144K   125G    88K  /mnt/apps/iocage/templates
apps/nextcloud                                         2.54G   125G    88K  /mnt/apps/nextcloud
apps/nextcloud/db                                       171M   125G  79.1M  /mnt/apps/nextcloud/db
apps/nextcloud/files                                   1.35G   125G  1.34G  /mnt/apps/nextcloud/files
apps/nextcloud/portsnap                                1.03G   125G   924M  /mnt/apps/nextcloud/portsnap
freenas-boot                                           18.8G  88.8G   144K  none
freenas-boot/ROOT                                      18.7G  88.8G   144K  none
freenas-boot/ROOT/11.2-BETA1                            288K  88.8G  1.19G  /
freenas-boot/ROOT/11.2-BETA2                            312K  88.8G  1.18G  /
freenas-boot/ROOT/11.2-BETA3                            260K  88.8G  1.20G  /
freenas-boot/ROOT/11.2-RC1                             1.04G  88.8G  1.10G  /
freenas-boot/ROOT/11.2-RC1-1                            228K  88.8G  1.09G  /
freenas-boot/ROOT/11.2-RC2                              228K  88.8G  1.08G  /
freenas-boot/ROOT/11.2-RELEASE                         44.6M  88.8G  1.08G  /
freenas-boot/ROOT/11.2-RELEASE-U1                       264K  88.8G  1.09G  /
freenas-boot/ROOT/11.2-U2.1                             324K  88.8G  1.07G  /
freenas-boot/ROOT/11.2-U3                              44.6M  88.8G  1.07G  /
freenas-boot/ROOT/11.2-U4.1                            48.9M  88.8G  1.08G  /
freenas-boot/ROOT/11.2-U5                              51.4M  88.8G  1.08G  /
freenas-boot/ROOT/11.2-U6                              17.4G  88.8G  1.08G  /
freenas-boot/ROOT/9.10.2-U3                            17.9M  88.8G   870M  /
freenas-boot/ROOT/9.10.2-U4                            17.0M  88.8G   871M  /
freenas-boot/ROOT/9.10.2-U5                            27.6M  88.8G   882M  /
freenas-boot/ROOT/9.10.2-U6                            26.8M  88.8G   881M  /
freenas-boot/ROOT/Initial-Install                         8K  88.8G   828M  legacy
freenas-boot/ROOT/default                              12.2M  88.8G   841M  legacy
freenas-boot/grub                                      7.31M  88.8G  7.31M  legacy
scratch                                                2.47T  4.55T  2.47T  /mnt/scratch
tank                                                   32.7T  9.42T  27.3G  /mnt/tank
tank/.system                                           1.65M  9.42T   192K  legacy
tank/.system/configs-810048d7feed436fae88f4409435135f   176K  9.42T   176K  legacy
tank/.system/cores                                      176K  9.42T   176K  legacy
tank/.system/rrd-810048d7feed436fae88f4409435135f       176K  9.42T   176K  legacy
tank/.system/samba4                                     440K  9.42T   440K  legacy
tank/.system/syslog-810048d7feed436fae88f4409435135f    352K  9.42T   352K  legacy
tank/.system/webui                                      176K  9.42T   176K  legacy
tank/media                                             31.5T  9.42T  31.5T  /mnt/tank/media
tank/photos                                             118G  9.42T   118G  /mnt/tank/photos
tank/stuff                                             1.06T  9.42T  1.06T  /mnt/tank/stuff

These are the datasets that are supposed to be available, but all are empty except media, photos and stuff.

Code:

-rwxrwx---+  1 root  wheel   0 Sep 29 09:08 .windows
drwxrwx---+  2 root  wheel  11 Aug  1 00:00 configbackup
drwxr-xr-x   2 root  wheel   2 Jun 14  2017 hoard
drwxrwx---+  8 root  wheel   8 Sep 10 09:42 media
drwxr-xr-x   2 root  wheel   2 Jul  4  2017 personal
drwxrwx---+  4 root  wheel   5 Sep 29 16:44 photos
drwxrwx---+  2 root  wheel   2 Sep 29 20:59 scratch-backup
drwxr-xr-x   2 root  wheel   2 Apr 20  2017 shared
drwxrwx---+  7 root  wheel  27 Jun  4 14:33 stuff
drwxrwx---+ 11 root  wheel  12 Jul 20  2018 temp
drwxr-xr-x   2 root  wheel   2 Apr 20  2017 tools

Is this recoverable?

 

[PhiloEpisteme]

zfs receive -Fd tank

I believe this is your problem, specifically the -F flag and that you are targeting tank rather than say tank/scratch. From the documentation

-F Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by zfs send -R -[iI]), destroy snapshots and file systems that do not exist on the sending side.

Is there a specific reason you're using the -Fd flags?

Is this recoverable?

Do you take snapshots of your tank pool as well? If so, the recovery is quite simple.

 

[Xyrgh]

Is there a specific reason you're using the -Fd flags?

It's what's in my notes that I used previously to migrate data, which is obviously wrong :-(

Do you take snapshots of your tank pool as well? If so, the recovery is quite simple.

No I did not, as it's not super important data. I'm ok with losing it, obviously not ideal but lesson learned.

I've marked this as solved as it doesn't appear recoverable and it's my messup. Luckily stupidity saved my stupidity as I hadn't turned off the Plex jail, so a large part of my media survived, in the end I lost around 70GB of data.

I've now rerun the zfs send receive without the -d flag and it's working fine.

 

[PhiloEpisteme]

It's what's in my notes that I used previously to migrate data, which is obviously wrong :-(

Even with those flags, I think it wouldn't have happened if you had created a dataset on the receiving end called scratch, and then used tank/scratch as the destination rather than tank.