missing files in dataset on target after replication to target

[Brad T]

Hi,

I've backed up and restored a dataset using zfs snapshots and a send/receive job between two servers. It seems that the job was interrupted (but I didn't realize this until perhaps too late...) and now zfs says there is data and snapshots are listed, but the file system appears to be empty... I need some help troubleshooting and/or next step suggestions.

environment
Primary: freenas02:/vol01/Backups
Secondary: freenas01:/vol01/Backups
starting OS: TrueNAS 12.0-U1
final OS: TrueNAS 12.0-U2.1

I had previously set up replication jobs to push snapshots from freenas02 to freenas01. This worked well for many years.

My vdevs underpinning the pool vol01 needed reconfiguration on both systems, so I figured I would start with one and then replicate the data back to the other and then repeat again (ie backup from primary to secondary, reconfigure primary, restore from secondary, reconfigure secondary, restore from primary).

1. Confirmed data was present on freenas01, replicated from freenas02.
2. Reconfigured vol01 on freenas02 (primary) to meet revised requirements.
3. In the Replication Tasks section, picked my 'backup vol01/Backup to freenas01' task and clicked the Restore button.
4. Built and ran the restore job, using full filesystem replication. It took hours.
5. (Assumed the replication was successful #idiot )
6. Reconfigured vol01 on freenas01 (secondary) to meet revised requirements.
7. system upgrades
   1. backed up system configuration TrueNAS 12.0-U1 (on both)
   2. removed USB thumb drives with mirrored boot-pool (on both)
   3. installed clean TrueNAS 12.0-U1 on new SSD (on both)
   4. restored backed up configuration (on both)
   5. upgraded to TrueNAS 12.0-U2.1 (on both)
8. Started adhoc replication job on freenas02 to push the vol01/Backups dataset back to freenas01.
9. Job completed very fast... too fast.
10. Sanity check.
    1. Found freenas01:/mnt/vol01/Backups was empty
    2. Found freenas02:/mnt/vol01/Backups was empty
11. Looked backward at email and replication at step #4 to see it had issues.
12. Rebooted just in case there was a read-only mount issue.

Error message from step #4 - retrieved from Replication Tasks job:

Code:

checksum mismatch or incomplete stream. Partially received snapshot is saved. A resuming stream can be generated on the sending system by running: zfs send -t 1-c254f4bb4-d0-789c636064000310a500c4ec50360710e72765a526973030e8419460caa7a515a79680640eb97840e5d990e4932a4b528b41e2360f0b3cb0e82fc94f2fcd4c6160e8f769cd9b50cde0e38024cf0996cf4bcc4d656028cbcf3130d4774a4cce2e2d2876482c2dc9d73532303230343234d3333232b4d4352a6780010098091e0e Command failed with code -1.

Here's what the rest of the system looks like on freenas02 (freenas01 is similar):

• zfs list shows:
  • snapshots are present (non-zero size)
  • vol01/Backups shows 289G used
• symptoms
  • file system is empty
  • snapshots directory is empty

Code:

root@freenas02:/ # zfs list -t all -r vol01/Backups
NAME                                  USED  AVAIL     REFER  MOUNTPOINT
vol01/Backups                         289G  7.48T      153K  /mnt/vol01/Backups
vol01/Backups@auto-20210326.1505-2w  89.5K      -      153K  -
vol01/Backups@auto-20210326.1805-2w  89.5K      -      153K  -
vol01/Backups@auto-20210326.2105-2w  89.5K      -      153K  -
vol01/Backups@auto-20210327.0005-2w  89.5K      -      153K  -
vol01/Backups@auto-20210327.0305-2w  89.5K      -      153K  -
vol01/Backups@auto-20210327.0605-2w  89.5K      -      153K  -
vol01/Backups@auto-20210327.0905-2w  89.5K      -      153K  -
vol01/Backups@auto-20210327.1205-2w     0B      -      153K  -
vol01/Backups@auto-20210327.1505-2w     0B      -      153K  -
vol01/Backups@auto-20210327.1805-2w     0B      -      153K  -

root@freenas02:/ # ls -lah /mnt/vol01/Backups
total 14
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
drwxr-xr-x  13 root  wheel    13B Mar 27 10:29 ..

root@freenas02:/ # zfs set snapdir=visible vol01/Backups

root@freenas02:/ # ls -lah /mnt/vol01/Backups
total 14
drwxr-xr-x   3 root  wheel     2B Mar 22 19:43 .
drwxr-xr-x  13 root  wheel    13B Mar 27 10:29 ..
dr-xr-xr-x+  3 root  wheel     3B Mar 22 19:43 .zfs

root@freenas02:/ # ls -lahR /mnt/vol01/Backups/.zfs
total 1
dr-xr-xr-x+  3 root  wheel     3B Mar 22 19:43 .
drwxr-xr-x   3 root  wheel     2B Mar 22 19:43 ..
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 snapshot

/mnt/vol01/Backups/.zfs/snapshot:
total 5
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 .
dr-xr-xr-x+  3 root  wheel     3B Mar 22 19:43 ..
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210326.1505-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210326.1805-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210326.2105-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.0005-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.0305-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.0605-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.0905-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.1205-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.1505-2w
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 auto-20210327.1805-2w

/mnt/vol01/Backups/.zfs/snapshot/auto-20210326.1505-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210326.1805-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210326.2105-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.0005-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.0305-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.0605-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.0905-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.1205-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.1505-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 ..

/mnt/vol01/Backups/.zfs/snapshot/auto-20210327.1805-2w:
total 1
drwxr-xr-x   2 root  wheel     2B Mar 22 19:43 .
dr-xr-xr-x+ 12 root  wheel    12B Mar 27 19:24 .

root@freenas02:/ # zfs set snapdir=hidden vol01/Backups

I did see on another thread that someone suggested creating a clone of the dataset and the data might show up there (maybe in FreeNAS v 11.3.x?). I tried that, it didn't work.

At this point I'm wondering if the zfs list output showing 289G is just a property of the dataset and has no relation with reality.

Any suggestions for next steps appreciated... Brad