Replication fails / cannot receive incremental stream

[BlazeStar]

Using 9.2.1.9

I have two replication, one local, one remote.

Before, the local replication was running 24/7.

Doing so, the remote replication would never happen.

Therefore I reschedule the local replication to run during business hours, and the remote to run outside of business hours.

It worked fine for a few days, but now - for the remote replication task- I'm getting this error

Replication failed with cannot receive incremental stream: most recent snapshot of REMOTE-BACKUP does not match incremental source

REMOTE-BACKUP is the name of the volume, on the remote machine, which is the target of the remote replication.

 

[mpfusion]

The current replication implementation is very brittle and sensitive to stream disruptions. We do receive the same error here every other week or so. Two solutions come to mind:

1) You select “Initialize remote side for once.” This will delete the remote data and start replicating the data from scratch. Use this if you deal with a rather small amount of data and the bandwith to the remote side is sufficient.

2) Or you transfer the out of sync snapshots manually (e.g. using SSH).

Although our bandwith is more than capable for the daily diffs, we use the second approach because we have don't have a high bandwith connection to the remote side and deleting all the snapshots will take weeks to catch up (it probably never will).

Decide which one works best for you. I really hope the replicator will be rewritten soon to be more robust and will gain the ability to resume mid-way of a snapshot transfer.

 

[BlazeStar]

Can you expand on the second option?

What commands should I send to sync snapshots manually ?

Is there a command that would sync them all manually that I could include in my crontab ?

Also, after you launch these commands, do the automatic replication with FreeNAS start again normally ?


Thanks !

 

[mpfusion]

Can you expand on the second option?

What commands should I send to sync snapshots manually ?

NOTE: What follows is POSIX shell, not csh.

1) First check for the most recent snapshot for all datasets:

Code:

  for i in $(zfs list -Ho name); do
    zfs list -Ho name -r -d 1 -t snapshot "$i" | tail -1
  done

Do this on the local and on the remote side. Then you know which datasets are out of sync.

2) Then copy each diff to the remote side. Here's what I use, adapt it to your needs.

Code:

  zfs send -vRI tank/foo@auto-20141124.0600-2m tank/foo@auto-20141125.1800-2m \
    | /usr/local/bin/lz4c \
    | ssh -i /data/ssh/replication \
      -o BatchMode=yes \
      -o StrictHostKeyChecking=no \
      -q remoteserver.com \
        "/usr/local/bin/lz4c -d | zfs receive -Fd tank"

First comes the last remote snapshot, then the latest local snapshot. Compression is optional, of course.

Is there a command that would sync them all manually that I could include in my crontab ?

You could wrap the abovementioned steps in a script, but I personally don't feel comfortable doing that. That messes around behind the back of the FreeNAS replication system. You might end up with a broken replication without GUI notification or emails. Doing it manually you have to inspect the snapshots regularly anyway.

I would wait for the replicator to be rewritten instead of persisting ugly scripts in cron jobs for something as crucial as data replication.

Also, after you launch these commands, do the automatic replication with FreeNAS start again normally ?

Yes, it does.

 

[mpfusion]

Three relevant bugs that might be related:

https://bugs.freenas.org/issues/5841
https://bugs.freenas.org/issues/2360
https://bugs.freenas.org/issues/1455

But there are more. Most of them basically read: “Wait for a rewrite in 9.3-RELEASE — 10.0-RELEASE”

 

[BlazeStar]

Most of them basically read: “Wait for a rewrite in 9.3-RELEASE — 10.0-RELEASE”

Got it!

Thanks for making that clear and end my quest for an explanation.

I'll do like everyone else and ENDURE.

 

[DaveY]

I know this is an old thread, but has anyone actually gotten this to work in 9.3 or 9.10? I did the delta transfer by hand and both sides look the same as far as snapshots, but when I re-enable replication, it still complains about a mismatch

 

[depasseg]

Do you mean replication? Yes I have it working. This thread is very old and isn't relevant to the post October 9.3.1 and beyond freenas systems since the replication subsystem got a major overhaul at that time.

 

[DaveY]

@depasseg yes. I'm on the latest 9.3 Stable and I'm still getting "cannot receive incremental stream" errors every so often. So with the new overhaul, what's the proper recovery procedure without blowing away the entire filesystem/zvol and starting replication over from scratch? The above method of just manually transferring the deltas didn't seem to work.

 

[depasseg]

I don't know, but start a new thread and we'll work it out.