PULL Replication Fails on Creation Time.. bug?

[Richard Durso]

I have TrueNAS-12.0-U2 doing a pull replication from a Linux system with snapshots created by Sanoid. This is working fine. TrueNAS filters snapshots by specified schema names such as autosnap_%Y-%m-%d_%H:%M:%S_monthly.

I'm having a problem with TrueNAS rejecting the snapshot based on creation time even though the snapshot name matches the schema name filter. This is the alert:

Code:

* Replication "tank/docker_projects -> main/docker/docker_backups" failed: zettarepl: zfs send PID is 852589
warning: cannot send 'tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly': not an earlier snapshot from the same fs
Replication cannot continue because existing snapshot
autosnap_2021-03-01_00:00:01_hourly is newer than
'tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly', but has an older date
in the snapshot name. To resolve the error, rename
'tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly' with a date that is older than
autosnap_2021-03-01_00:00:01_hourly or delete snapshot
'tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly' from both the source and destination..

This happens due to a 1 second difference between snapshot name and creation time:

Code:

$ zfs get creation tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly
NAME                                                               PROPERTY  VALUE                  SOURCE
tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly  creation  Mon Mar  1  0:00 2021  

I have to rename all the snapshots that have this problem to be one second sooner...

Code:

sudo zfs rename tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly tank/docker_projects/certbot@autosnap_2021-03-01_00:00:00_monthly

# Verified name matches creation time.
$ zfs get creation tank/docker_projects/certbot@autosnap_2021-03-01_00:00:00_monthly
NAME                                                               PROPERTY  VALUE                  SOURCE
tank/docker_projects/certbot@autosnap_2021-03-01_00:00:00_monthly  creation  Mon Mar  1  0:00 2021  -

As soon as I do that, the replication alert clears:

Code:

The following alert has been cleared:
* Replication "tank/docker_projects -> main/docker/docker_backups" failed: zettarepl: zfs send PID is 852589
warning: cannot send 'tank/docker_projects/certbot@autosnap_2021-03-01_00:00:01_monthly': not an earlier snapshot from the same fs
Replication cannot continue because existing snapshot....

The underlying ZFS send/rec'v wouldn't care about that name difference. TrueNAS / zettarepl seem to be introducing this.

 

[blueduckdock42]

+1 to this. I'm not sure how I could otherwise avoid this error.

 

[lA1tO2oK0sU8nU7h]

Yep, I can't really offer anything in the way of troubleshooting but I have this too. It happened this evening and I was going to open a bug, but JIRA has been loading for 2+ minutes at this point. I will try again next time it happens and see if JIRA is more responsive.

One thing that is notable to me, and points to this as a TrueNAS problem (not related to zfs send/zfs recv) is that when the TrueNAS replication fails, I can go to the source box and run a syncoid push job with no problems on exactly the same root datasets.

 

[BlueSpaceCanary]

Sorry for bumping an old thread, but I'm running into this exact issue as well & couldn't find any other discussions or recommended workarounds.

 

[BlueSpaceCanary]

Actually, to compound the problem, the error I'm getting is also at least partially incorrect:

Code:

warning: cannot send 'foo/bar@autosnap_2021-04-15_23:05:00_yearly': not an earlier snapshot from the same fs
Replication cannot continue because existing snapshot
autosnap_2021-04-15_23:05:00_monthly is newer than
foo/bar@autosnap_2021-04-15_23:05:00_yearly, but has an older date

Code:

% zfs get -p creation foo/bar@autosnap_2021-04-15_23:05:00_yearly
NAME                                            PROPERTY  VALUE                  SOURCE
foo/bar@autosnap_2021-04-15_23:05:00_yearly  creation  1618527900             -
% zfs get -p creation foo/bar@autosnap_2021-04-15_23:05:00_monthly
NAME                                             PROPERTY  VALUE                  SOURCE
foo/bar@autosnap_2021-04-15_23:05:00_monthly  creation  1618527900

So "not an earlier snapshot from the same fs" is true, but "existing snapshot [...] is newer [...] but has an older date" is wrong: they have identical creation timestamps and identical timestamps in their names. This is on 12.0-U8, and the remote machine I'm pulling from is running FreeBSD 13.0-RELEASE-p7.