SOLVED Incremental replication after moving dataset to new server - had to check the full dataset name

[pvuchetich]

How do I get FreeNAS GUI to continue incremental zfs replications after moving a data set to a new server? The GUI wants to start from the beginning and resend the entire dataset.
Perhaps there is a file somewhere that has the most recent snapshot sent that I can manually edit?

Scenario:
old_server, new_server, and backup_server all all FreeNAS 11.1 U6 servers.

I have moved some datasets from old_server to new_server, with the intention of uninstalling FreeNAS from old_server.

Using zfs replication in the GUI, old_server sent incremental backups to both new_server and backup_server

I am now trying to set up replication from new_server to backup_server, but it is starting with the earliest snapshot (i.e., the entire dataset) instead of the incremental copy. This is not desirable because backup_server is remote and is relatively slow over a VPN.

I am able to manually send incremental snapshots from new_server to backup_server via the CLI, so there is not an error with the datasets or snapshots being out of sync. I think the issue is just that the GUI does not have a way to check for previous snapshots that were sent from old_server to backup_server when setting up a new replication task on new_server ("Last snapshot sent to remote side" in the GUI), so it is just starting from scratch (as one would expect when setting up a new replication task).

Note - next steps in troubleshooting would be to look more deeply into FreeNAS config files, possibly turn on debug logging for the autorepl.py, and generally poke around for where the GUI is getting its replication information...but I thought that someone on the forum might have had this sort of a change, although I didn't find a post with this exact scenario.

 

[kdragon75]

but it is starting with the earliest snapshot

My guess is that this is the middleware trying to keep track and because you went from a to b and a to c but now your going from b to c, It thinks it need to start over.

 

[pvuchetich]

This is resolved...based on the partial copies (that I manually interrupted - not any system error), I found that the replication includes the parent dataset in the target path (at least as a placeholder; I don't know if it would copy data based on my initial scenario), instead of just the specific child dataset that was requested. I knew that FreeNAS pulled in the top level dataset, but in my current configurations, I had always replicated top level datasets recursively, and not tried to replicate one level down, so I had to relearn this bit of information.

The easiest way would have been to keep the identical name for the pools/datasets on the new_server, and replicate to the same location on backup_server. That likely would have just worked.

The harder way is what I went with because I was consolidating smaller pools and better organizing the backup_sever pool:
When moving both the source data (to a new server and pool structure) as well as the target data (within the same pool, but to a server-specific dataset), additional manual steps were needed to avoid the full dataset being sent.

Illustration: on new_server
pool1/home/user1 is the new location

Originally intended to replicate dataset "pool1/home/user1" to a specific dataset with only the ending dataset name on backup_server (in FreeNAS GUI the remote volume/dataset would just be "pool2/newserver"
pool2/newserver/user1 [I used zfs rename to moved the dataset here, which wasn't where FreeNAS was looking for it]

however, the FreeNAS GUI was automatically creating the intermediate pool like this:
pool2/newserver/home
pool2/newserver/home/user1 [this is where FreeNAS was looking for matching snapshots!]


The solution I tried that worked:
1. Manually send the most recent snapshot of just the "pool1/home" dataset (not recursive) to "pool2/newserver/home" just to get it created as a replication. This dataset only contains other data sets, so the zfs send/receive takes only a few seconds.
2. Manually rename (zfs rename) the datasets on the backup server to match the new location "pool2/newserver/home/user1"
3. Set up replication again in FreeNAS GUI replication of "pool1/home/user1" using the remote location of "pool2/newserver". This was successfully sent as an *incremental* backup to "pool2/newserver/home/user1"
4. As a separate test, I disabled the replication of "pool1/home/user1", and tried both the parent level "pool1/home", also pointing to the remote location of "pool2/newserver" in the GUI, including recursive datasets. In preparation for this test, I had also renamed the "user2" dataset (zfs rename) on backup_server under the "pool2/home" dataset, and both user1 and user2 datasets were synchronized as an incremental backup.

Notes: This is not an exhaustive test - the parent dataset I am using only contains other datasets, and it has a single layer of child datasets.