cannot receive incremental stream: most recent snapshot of backup/storage does not match incremental source

[cTurtle98]

Code:

ERROR    [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(1, 'cannot receive incremental stream: most recent snapshot of backup/storage does not\nmatch incremental source\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 164, in run_replication_tasks
    retry_stuck_replication(
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/stuck.py", line 18, in retry_stuck_replication
    return func()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 165, in
    lambda: run_replication_task_part(replication_task, source_dataset, src_context, dst_context,
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 258, in run_replication_task_part
    run_replication_steps(step_templates, observer)
... 6 more lines ...
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/ssh.py", line 151, in wait
    stdout = self.async_exec.wait()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/async_exec_tee.py", line 103, in wait
    raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: cannot receive incremental stream: most recent snapshot of backup/storage does not
match incremental source

when I try to run a manual replication on my dataset I get this error.

I am pulling from a truenas scale system to my local truenas core backups system.

I changed some things on the truenas scale and made a snapshot and now im trying to replicate and im getting this error

 

[zookeeper21]

I am having same issue. Did you figure out?

 

[ScaleyNas]

Me too! Except I'm using SCALE and I'm researching it now. Fingers crossed!

 

[sretalla]

try looking at this:
zfs list -t snap backup/storage
and compare that to the same command on the replica location... see which snapshots are different (if it's more than one) and potentially you'll have an option to delete one to make it work.

 

[john60]

What does this command do?
is backup/storage the name of the filesystem?

on my primary Truenas is get
root@truenas[~]# zfs list -t snap jjg-14TB-z2
NAME USED AVAIL REFER MOUNTPOINT

jjg-14TB-z2@manual-2021-12-15_07-21 160K - 424K -
jjg-14TB-z2@Weekly everything-2021-12-26_00-00 160K - 424K -
jjg-14TB-z2@Weekly everything-2022-01-02_00-00 184K - 424K -
jjg-14TB-z2@manual-2022-01-06_17-11 112K - 424K -
jjg-14TB-z2@auto-2022-01-06_09-36 112K - 424K -
jjg-14TB-z2@Weekly everything-2022-01-09_00-00 112K - 424K -
jjg-14TB-z2@Weekly everything-2022-03-06_00-00 128K - 424K -
jjg-14TB-z2@Weekly everything-2022-04-10_00-00 112K - 424K -
jjg-14TB-z2@manual-2022-04-12_13-55 112K - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-16_00-00 112K - 424K -
jjg-14TB-z2@Weekly everything-2022-04-17_00-00 0B - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-17_00-00 0B - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-18_00-00 112K - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-19_00-00 112K - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-20_00-00 112K - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-21_00-00 112K - 424K -
jjg-14TB-z2@daily jjg-14TB-z2-2022-04-22_00-00 112K - 424K -
jjg-14TB-z2@manual-2022-04-22_13-12 112K - 424K -
jjg-14TB-z2@auto-2022-06-27_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-06-28_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-06-29_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-06-30_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-01_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-02_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-03_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-04_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-05_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-06_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-07_00-00 128K - 464K -
jjg-14TB-z2@auto-2022-07-07_03-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-08_00-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-08_03-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-09_00-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-09_03-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-10_00-00 112K - 464K -
jjg-14TB-z2@auto-2022-07-10_03-00 112K - 464K -
root@truenas[~]#

on my backup Truenas I get

root@truenas[~]# zfs list -t snap backup-14TB-z2
NAME USED AVAIL REFER MOUNTPOINT
backup-14TB-z2@auto-2022-06-27_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-06-28_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-06-29_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-06-30_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-01_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-02_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-03_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-04_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-05_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-06_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-07_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-08_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-09_00-00 0B - 4.59T -
backup-14TB-z2@auto-2022-07-10_00-00 0B - 4.59T -
root@truenas[~]#

my replication task from my primary nas to my backup nas gives me error

[EFAULT] Active side: most recent snapshot of backup-14TB-z2 does not
match incremental source.

I'm lost for how to get my backup to get a copy of the primary nas. Any help would be appreciated.

 

[zookeeper21]

Look into this, if it works. At least it did for me. Not sure, if same issue.

SOLVED - ZFS replication on same system

So I never figured out why it used to work and suddenly stopped working, so I deleted replication task and created new one and doesn't work anymore. This happens randomly and has never figured out. If someone can point me to right direction. Here are screenshot all setting which I believe for...

www.truenas.com

 

[john60]

Look into this, if it works. At least it did for me. Not sure, if same issue.

SOLVED - ZFS replication on same system

So I never figured out why it used to work and suddenly stopped working, so I deleted replication task and created new one and doesn't work anymore. This happens randomly and has never figured out. If someone can point me to right direction. Here are screenshot all setting which I believe for...

www.truenas.com

I found a similar solution, delete the backup truenas, and start over from scratch.
Kind of a rough solution, but It worked.

 

[Apollo]

Normally, on the pool that is receiving stream, if system/user is making modification to the pool on the receiving end, it will cause ZFS to complain about it. If the pool being replicated isn't expected to deal with the data causing it to be different from the source, the use of the "-F' option on the "zfs receive ..." side can be used.
Strictly speaking, any changes done after the last snapshot on the recieving pool will be discarted in order to maintain integrity with the data previously received.

 

[zookeeper21]

Normally, on the pool that is receiving stream, if system/user is making modification to the pool on the receiving end, it will cause ZFS to complain about it. If the pool being replicated isn't expected to deal with the data causing it to be different from the source, the use of the "-F' option on the "zfs receive ..." side can be used.
Strictly speaking, any changes done after the last snapshot on the recieving pool will be discarted in order to maintain integrity with the data previously received.

You see, recently, I export two drive mirror from destination side and did last replication before doing it. Now if I import those disk to system I was replicating, I was able to import disk in mirror setup. But when I started replication again, it is asking me to start replication all over again.

Is it true or am I doing something wrong?

 

[Apollo]

You see, recently, I export two drive mirror from destination side and did last replication before doing it. Now if I import those disk to system I was replicating, I was able to import disk in mirror setup. But when I started replication again, it is asking me to start replication all over again.

Is it true or am I doing something wrong?

You need to provide more information on your setup.
Once data has been replicated, you should see the list of datasets and snapshots.
Normally, if your snapshot are present, all you need is to perform incremental replication.
If you have to start replication all over, then something isn't right.

 

[zookeeper21]

You need to provide more information on your setup.
Once data has been replicated, you should see the list of datasets and snapshots.
Normally, if your snapshot are present, all you need is to perform incremental replication.
If you have to start replication all over, then something isn't right.

Not every time. Only when I move disk to different system or if I reinstall system where it is replication. I get message snapshot doesn't match.

 

[Darkhog]

I found a similar solution, delete the backup truenas, and start over from scratch.
Kind of a rough solution, but It worked.

I hit this too. Just to clarify the "delete the backup truenas" solution for me -

System1 - TrueNas Core 12.0-U8.1, Pool1, DatasetSource
System2 - TrueNas Core 12.0-U8.1, Pool2, DatasetReplicatedData

System1 replicates DatasetSource into System2 DatasetReplicatedData. That creates a dataset DatasetReplicatedData\DatasetSource on System2.

To clear this error, on System2 I deleted the dataset DatasetReplicatedData\DatasetSource.

Running the replication again on System1 recreated the dataset DatasetReplicatedData\DatasetSource and replication started again. It will replicated everything again, which in my case is fine because I was at the very start of replicating the dataset.

Overall I find truenas replication extremely fragile. It's easy to get in a bind, and then you're screwed. Recovery is always tenuous. In this case, perhaps it is my misunderstanding of snapshots, but I thought if you created snapshot1, snapshot2, snapshot3, then you deleted snapshot 2, snapshot 3 would have everything except the changes from s1 to s2 that didn't also exist in s3. In my case, I'm loading a drive so it is likely only additive and s3 should have everything. It seems creating an option to ignore this error in that case and continue would be easier than deleting the dataset. Better yet, prevent me from deleting that snapshot because the system knows it is part of a pending replication? I hadn't intended to delete what I deleted, but the UI didn't line up right. Not sure what the right solution is, but current behavior just drains my confidence...

 

[winnielinnie]

I thought if you created snapshot1, snapshot2, snapshot3, then you deleted snapshot 2, snapshot 3 would have everything except the changes from s1 to s2 that didn't also exist in s3.

A snapshot only has what the live filesystem has at the moment of creating the snapshot. They are not intertwined with each other in terms of what (extant) data each snapshot contains.

You can have snapshot01 through snaphot99. If you delete all snapshots except snapshot50, then snaphot50 remains as it has always been. It doesn't depend on snapshot01 through snapshot49.

 

[zookeeper21]

Not sure what the right solution is, but current behavior just drains my confidence...

I felt same way. Specially when you move disk to different system or reinstall system. I keep getting error saying no snapshot exist. It will start fresh replication (don't know exact wording but it start replication all over).

 

[winnielinnie]

My hunch is that the Replication Tasks (in the GUI) has a very narrow scope for enterprise customers. This is something that iXsystems is unlikely to play around with, especially in Core.

The tooltips are ambiguous (some of which I reported as bugs that were "kind of" fixed), and the flexibility on creating replications for a personal use-case is just not that feasible with the TrueNAS GUI. The official documentation doesn't help much either.

If anything, TrueNAS could theoretically leverage syncoid, and build a GUI around it, for users who want simple "to the point" backups. I've used syncoid before and it works more intuitively as a straightforward ZFS "target -> destination" backup solution for home users. Unfortunately, there's no GUI of it in TrueNAS; let alone the program is not pre-installed.

 

[Darkhog]

A snapshot only has what the live filesystem has at the moment of creating the snapshot. They are not intertwined with each other in terms of what (extant) data each snapshot contains.

You can have snapshot01 through snaphot99. If you delete all snapshots except snapshot50, then snaphot50 remains as it has always been. It doesn't depend on snapshot01 through snapshot49.

Yes that is a better way of saying what I was trying to say. I just don't know why, if I deleted snapshot50, it wouldn't just pick up snapshot51 and keep going.

 

[Darkhog]

My hunch is that the Replication Tasks (in the GUI) has a very narrow scope for enterprise customers. This is something that iXsystems is unlikely to play around with, especially in Core.

Yeah, I'm not buying it that this is sufficient for enterprise. I'm sure this would be just as problematic for them, and they would be happy if these were more robust.

The reality is - there are a lot of tough technical problems that are mostly solved. But the UX and certain fairly common flows have silly designs that just don't make much sense. Sometimes on simple things - like why if my browser is a little narrow do you have to scroll right / left? Why is the expand UX on the right, but then the other operations are on the left? Why doesn't it keep expand state when drilling in and out? Why does it only have like 5 rows when you have a tall browser? And why is the Shell in web so narrow? Why can't you see simple activities, like when another truenas is replicating to you? These can be worked around by various ways, it just seems like lots of low hanging fruit. I haven't yet gone to 13 or Scale though, so maybe some things are addressed there. I came to truenas for ZFS appliance like behavior, but the more I do the more I think Synology probably solves a lot of this more intuitively.

Rant over, sorry. :P

 

[winnielinnie]

Yes that is a better way of saying what I was trying to say. I just don't know why, if I deleted snapshot50, it wouldn't just pick up snapshot51 and keep going.

What do you mean? If the source is missing snapshot50, you cannot do an incremental replication that start's at the destination's snapshot50. It wouldn't know the difference between the "latest snapshot" and a "snapshot that no longer exists". You always need a "common base snapshot" shared between the source and destination to do an incremental replication.

The only way around this is to use the ZFS feature called "bookmarks". But even bookmarks are sadly limited (not for technical reasons, but because the ZFS developers never got around to it.)

 

[winnielinnie]

But the UX and certain fairly common flows have silly designs that just don't make much sense.

Believe me. I know.

 

[Darkhog]

What do you mean? If the source is missing snapshot50, you cannot do an incremental replication that start's at the destination's snapshot50. It wouldn't know the difference between the "latest snapshot" and a "snapshot that no longer exists". You always need a "common base snapshot" shared between the source and destination to do an incremental replication.

The only way around this is to use the ZFS feature called "bookmarks". But even bookmarks are sadly limited (not for technical reasons, but because the ZFS developers never got around to it.)

Thanks for the response winnielinnie, I don't have a lot of experience with Truenas so some of this I may not understand. If I have a snapshot 51, then this is a problem that can be recovered from, correct? snapshot51 does not need snapshot50, as you said. I'm replicating all the snapshots, so it just needs to find the common base from which it can incrementally replicate (which in this case would be snapshot49 which is going to be less than replicating from scratch, right?).

The current solution of doing the full replication won't bring back the missing snapshot either. Software systems that are forgiving and just do the right thing are amazing user experiences.

In this case, I accidentally deleted a snapshot and it wrecked my whole replication to an offsite backup. I did not intend to delete the snapshot I deleted - it was an accident, because the snapshot UI is one of those places with lots of opportunity, and the only way to sort snapshots by date is to name them by date and then sort by name (but then the reverse sort that brings the most recent auto- snapshots to the top are under the manual- snapshots, which is why I deleted manual snapshots and accidentally an auto). It is possible that I got into this situation because it was the initial replication, which didn't have a previous common base, but when initial replications can take weeks or more, it would be great to be able to recover.