Replication and Used file space difference

[onlineforums]

I am doing a backup of a particular dataset to an external hard drive for off-site backup purposes.

In TrueNAS 13.0 GUI I setup a replication task with the "recursive" checkbox on the particular dataset to backup. For the destination I created a dataset on the external drive pool and selected it as the location. I then ran the replication from the GUI and after many hours it completed successfully.

For visual reasons assume the following:
The dataset I'm backing up is called important and is on the tank pool. The important dataset does have sub-datasets. So tank/important is the source.
External drive pool name is external and the dataset created is called backup. So external/backup is the destination.

After the replication GUI showed SUCCESS of tank/important (with recursive) to external/backup the file space used differs by quite a bit. When I run zfs list -r the externaldrive/backup dataset used column shows 800G whereas the tank/important dataset shows 900G.

Looking at some of the sub datasets they don't match exactly (although some do). I'm really confused on how this could be. Further, when I list the contents of tank/important/.zfs/snapshots it lists many auto@<date> snapshots whereas external/backup/.zfs/snapshots list nothing.

I'm confused. My main goal is to backup the tank/important dataset along with all sub-datasets to an external hard drive, and it seems like most (800G out of 900G) is there but really weird that it doesn't match 900G but maybe that isn't weird?? Any help out there?

 

[onlineforums]

Any feedback on this?

 

[winnielinnie]

Did you configure the replication task to also include the auto snapshots?

 

[onlineforums]

Did you configure the replication task to also include the auto snapshots?

Yes. When I run zfs list -t snapshot | grep external02 it shows plenty of snapshots with "auto@<date>" but it doesn't have all of them. If I run zfs list -t snapshot | grep tank it shows many datasets snapshots that quite simply aren't in the external02.

For clarification I'm doing a local replication via the GUI using Task -> Replication Task. Selected recursive snapshot.
Source: tank/important
Destination: external/backup

The tank/important has tank/important/pictures, tank/important/pictures/family for example and those pictures and pictures/family datasets are created on the external/backup dataset but only the family dataset has snapshots and the pictures dataset does not. On the source (tank) there are snapshots in the pictures dataset.

I can confirm this via that zfs list -t snapshot command but also by ls in the /tank/important/pictures/.zfs/snapshot it lists a ton (all with auto@<date>) and if I ls in the /external/backup/pictures/.zfs/snapshot there are zero snapshots listed. If I do the same thing in the tank/important/pictures/family/.zfs/snapshot and in external/backup/important/pictures/family/.zfs/snapshot then they match identically.

So it seems as though I am in fact missing data. The fact there is over 100 GiB missing using zfs list command again further confirms that somehow the recursive replication done in the GUI did not actually send everything over. The GUI shows the task as completed.

 

[winnielinnie]

Is it possible to show screenshots of this Replication Task? You can blur out or redact any sensitive names.

It will also greatly help to include screenshots of the configured Periodic Snapshot Tasks.

 

[onlineforums]

I think I found out the problem in that using the GUI replication task using the recursive route will only do so many child datasets. I looked and found that even though all of the datasets of the source had a check next to them, the actual source text-box that lists out each of the dataset with a comma between each didn't have all of them. It basically truncated it because I have like 30 child datasets off of the /tank/important dataset. I've manually deleted a main dataset (pictures) since it had data and child data-sets and just did a replication task for recursive of only that pictures dataset which only has about 5 child datasets and it looks as though this is the solution to my particular issue.

It is weird because when I tested it via command line using zfs send | zfs receive the total GiB between source and destination were almost identical but doing it via replication task it was way off by over 100 GiB. I guess the replication task in the GUI doesn't allow a substantial amount of datasets done via clicking around and may have to manually separate via commas in the source text box.