Replication Task Error

[revengineer]

Hello,
Here is another noob question, this time on replication tasks. I set up daily, weekly, and monthly snapshot for my main data pool, with appropriate keep times. I want to replicate the snapshots to a backup drive. I set up a local replication task and select for "Periodic Snapshot Tasks" the daily snapshot and all worked fine for the last week. Today I added the weekly and monthly snapshot by clicking the checkmarks for the snapshots in "Periodic Snapshot Tasks"; so there are 3 checkmarks total now. When I run the task, I get the error below. If someone could tell me what I am doing wrong, I would appreciate it. Thank you!

Code:

[2020/07/01 08:40:17] INFO     [replication_task__task_3] [zettarepl.replication.run] For replication task 'task_3': doing push from 'tank1/users' to 'backup1/backup/users' of snapshot='monthly-auto-2020-07-01_00-00' incremental_base='daily-auto-2020-07-01_00-00' receive_resume_token=None
[2020/07/01 08:40:17] ERROR    [replication_task__task_3] [zettarepl.replication.run] For task 'task_3' unhandled replication error ExecException(1, 'cannot receive incremental stream: most recent snapshot of backup1/backup/users does not\nmatch incremental source\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 143, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 204, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 389, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 444, in replicate_snapshots
    run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 500, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 128, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 50, in wait
    raise ExecException(self.process.returncode, stdout)
zettarepl.transport.interface.ExecException: cannot receive incremental stream: most recent snapshot of backup1/backup/users does not
match incremental source

 

[Samuel Tai]

You'll need separate replication tasks each for the weekly and monthly snapshots. It's a 1:1 correspondence between snapshot tasks and replication tasks.

 

[revengineer]

I tried setting up a separate task for the weekly snapshots. Now I get the error message below. There is something I am not yet understanding.

Code:

[2020/07/01 09:03:34] ERROR    [replication_task__task_7] [zettarepl.replication.run] For task 'task_7' non-recoverable replication error NoIncrementalBaseReplicationError("No incremental base on dataset 'tank1/users' and replication from scratch is not allowed")

 

[Samuel Tai]

How many weekly snapshots do you have? You'll need at least 2.

 

[revengineer]

There are exactly 2 weekly snapshot, one dated 6/21/20 the other 6/28/20.

 

[Samuel Tai]

OK, the replication base is in the dailies. You could try checking the checkbox for Replicate from scratch if incremental is not possible in the replication task to get around this error, as detailed in the Guide, Section 8.6.2, for advanced replication.

 

[revengineer]

I saw that option but was scared off by the "can cause data loss" warning.

EDIT: If I use this option, will this double the size of the data set on the backup drive becuase I will have one base for the dailies and another for the weeklies?

 

[Samuel Tai]

Yes, because you'll have 2 bases.

 

[revengineer]

Thanks, that's what I thought. Do you know what the reason for the data loss warning is?

 

[Samuel Tai]

Thanks, that's what I thought. Do you know what the reason for the data loss warning is?

The warning is if you already had a successful replication on the receiver. Checking this could overwrite the replicas on the target.

 

[revengineer]

Got it! Thank you for all the help again.

 

[revengineer]

I am sorry to have to reopen this one. I royally messed up the replication task, starting to get errors on the nightly task now. So I decided to start over and deleted all snapshots on the backup target. I now want to remove the user files but cannot delete them with "rm -rf *" because the file system is read-only. Mount shows

Code:

backup1/backup/users on /mnt/backup1/backup/users (zfs, local, read-only, nfsv4acls)

I tried "mount -o remount,rw /mnt/backup1/backup/users" but then I get

Code:

mount: /mnt/backup1/backup/users: unknown special file or file system

Please let me know how to delete these files.

 

[Samuel Tai]

Try zfs destroy backup1/backup/users.

 

[revengineer]

Thank you once again. Based on your hint to destroy the dataset, I was able to do so from the GUI. I am just surprised that this is required and that there is no means, even as root, to simply delete the directories at the top level. In any case, the problem is solved for now and I am starting over the replication from scratch.