Restoring replication task isn't working

[sgt_jamez]

My intention was to backup selected datasets on a pool and push them to a remote server with the idea being to use the restore function to get the data back from the remote. The replication to the remote end works and the files are accessible on the distant end.

I wanted to test it out, so I deleted the contents of the local dataset, set up a restore task and ran it. It says SUCCESS but also says no snapshot sent yet.

When I click the SUCCESS icon I get the following text:

Code:

[2023/07/24 08:57:45] INFO     [Thread-10] [zettarepl.paramiko.replication_task__task_13] Connected (version 2.0, client OpenSSH_8.4p1)
[2023/07/24 08:57:45] INFO     [Thread-10] [zettarepl.paramiko.replication_task__task_13] Authentication (publickey) successful!
[2023/07/24 08:57:46] INFO     [replication_task__task_13] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2023/07/24 08:57:46] INFO     [replication_task__task_13] [zettarepl.replication.run] No snapshots to send for replication task 'task_13' on dataset 'main_pool/replication_target/immigration_docs'

The dataset still shows empty.

How can I restore the files from the distant end?

 

[sgt_jamez]

Continuing my attempt to restore from distant end backup, I removed all the local snapshots related to this dataset. I removed the replication task, the snapshot task, and removed the dataset as well.

I created a replication task to PULL from the remote server to local. It still fails.

Code:

[2023/07/24 13:11:48] INFO     [Thread-64] [zettarepl.paramiko.replication_task__task_16] Connected (version 2.0, client OpenSSH_8.4p1)
[2023/07/24 13:11:48] INFO     [Thread-64] [zettarepl.paramiko.replication_task__task_16] Authentication (publickey) successful!
[2023/07/24 13:11:49] INFO     [replication_task__task_16] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2023/07/24 13:11:49] INFO     [replication_task__task_16] [zettarepl.replication.run] For replication task 'task_16': doing pull from 'main_pool/replication_target/immigration_docs' to 'main_vault' of snapshot='auto-2023-05-19_23-00' incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2023/07/24 13:11:49] INFO     [replication_task__task_16] [zettarepl.paramiko.replication_task__task_16.sftp] [chan 5] Opened sftp connection (server version 3)
[2023/07/24 13:11:49] INFO     [replication_task__task_16] [zettarepl.transport.ssh_netcat] Automatically chose connect address '192.168.40.X'
[2023/07/24 13:11:49] ERROR    [replication_task__task_16] [zettarepl.replication.run] For task 'task_16' unhandled replication error SshNetcatExecException(ExecException(1, "cannot unmount '/var/lib/kubelet': pool or dataset is busy\n"), ExecException(1, "cannot send 'main_pool/replication_target/immigration_docs': Connection reset by peer\n"))
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 181, in run_replication_tasks
    retry_contains_partially_complete_state(
... 10 more lines ...
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 732, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/lib/python3/dist-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/lib/python3/dist-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/lib/python3/dist-packages/zettarepl/transport/ssh_netcat.py", line 209, in wait
    raise SshNetcatExecException(connect_exec_error, self.listen_exec_error) from None
zettarepl.transport.ssh_netcat.SshNetcatExecException: Passive side: cannot unmount '/var/lib/kubelet': pool or dataset is busy
Active side: cannot send 'main_pool/replication_target/immigration_docs': Connection reset by peer