Pull Replication destroys destination dataset and errors that it does not exist

Richard Durso

Explorer
Joined
Jan 30, 2014
Messages
70
Using TrueNAS-12.0-U1.1

Trying to configure a pull replication from an Ubuntu Docker server 20.04.1 using SSH. There is very little documentation on setting up pull replication, so I'm experimenting with it but TrueNAS keeps destroying my local destination dataset (that doesn't seem right).

First step, I setup a dataset to hold it. Pretty much all the defaults.

dataset_created.png


I then make an attempt to configure the PULL Replication, it was able to browser remote datasets to allow me to select it.
pull_replication_settings.png


Upon clicking Run Now, I get the following error:
Code:
Error
cannot open 'main/docker/docker_backups': dataset does not exist.
Logs
[2021/01/27 19:24:57] INFO     [Thread-3428] [zettarepl.paramiko.replication_task__task_17] Connected (version 2.0, client OpenSSH_8.2p1)
[2021/01/27 19:24:57] INFO     [Thread-3428] [zettarepl.paramiko.replication_task__task_17] Authentication (publickey) successful!

[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] Encrypted destination dataset 'main/docker/docker_backups' does not have snapshots or data, destroying it

[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/certbot'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/gitea'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/homeassistant'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/mosquitto'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/portainer'
[2021/01/27 19:24:59] INFO     [replication_task__task_17] [zettarepl.replication.run] No snapshots to send for replication task 'task_17' on dataset 'tank/docker_projects/traefik'
... 5 more lines ...
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 239, in run_replication_task_part
    mount_dst_datasets(dst_context, target_dataset, replication_task.recursive)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 674, in mount_dst_datasets
    dst_datasets = list_datasets_with_properties(dst_context.shell, dst_dataset, recursive, {
  File "/usr/local/lib/python3.8/site-packages/zettarepl/dataset/list.py", line 30, in list_datasets_with_properties
    output = shell.exec(args)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/zfscli/exception.py", line 29, in __exit__
    raise DatasetDoesNotExistException(exc_val.returncode, exc_val.stdout) from None
zettarepl.transport.zfscli.exception.DatasetDoesNotExistException: cannot open 'main/docker/docker_backups': dataset does not exist


It destroys where I'm trying to place snapshots from the remote and then gives an error "dataset does not exist".. well, yeah, you destroyed it.

When I go look, sure enough, its gone.
dataset_gone.png


I'm sure I'm doing something wrong, but TrueNAS destroying the local dataset can't be right thing to do.
 

Richard Durso

Explorer
Joined
Jan 30, 2014
Messages
70
As a follow up I tried:
  • Setting the local dataset as readonly, didn't matter still got deleted.
  • Tried selecting "Full File System Replication" but yet it insists on snapshot schema (why? its supported to replicate everything). Produced exact same error as above.
  • I tried schema `autosnap_%Y-%M-%d_%H:%m:00_hourly` as well, same error and dataset destroyed.
As reference, these examples of remote datasets I'm trying to replicate:

Frequently ones:
Code:
tank/docker_projects@autosnap_2021-01-28_00:15:05_frequently                   0B      -      548K  -
tank/docker_projects@autosnap_2021-01-28_00:30:00_frequently                   0B      -      548K  -
tank/docker_projects@autosnap_2021-01-28_00:45:00_frequently                   0B      -      548K  -
tank/docker_projects@autosnap_2021-01-28_01:00:00_frequently                   0B      -      548K  -
tank/docker_projects/certbot@autosnap_2021-01-28_00:15:05_frequently           0B      -      956K  -
tank/docker_projects/certbot@autosnap_2021-01-28_00:30:00_frequently           0B      -      956K  -
tank/docker_projects/certbot@autosnap_2021-01-28_00:45:00_frequently           0B      -      956K  -
tank/docker_projects/certbot@autosnap_2021-01-28_01:00:00_frequently           0B      -      956K  -
tank/docker_projects/gitea@autosnap_2021-01-28_00:15:05_frequently             0B      -      196K  -
tank/docker_projects/gitea@autosnap_2021-01-28_00:30:00_frequently             0B      -      196K  -
tank/docker_projects/gitea@autosnap_2021-01-28_00:45:00_frequently             0B      -      196K  -
tank/docker_projects/gitea@autosnap_2021-01-28_01:00:00_frequently             0B      -      196K  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_00:15:05_frequently  2.49M      -     19.7M  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_00:30:00_frequently  2.58M      -     19.7M  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_00:45:00_frequently  2.28M      -     19.8M  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_01:00:00_frequently     0B      -     19.8M  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_00:15:05_frequently         0B      -      376K  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_00:30:00_frequently         0B      -      376K  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_00:45:00_frequently         0B      -      376K  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_01:00:00_frequently         0B      -      376K  -
tank/docker_projects/portainer@autosnap_2021-01-28_00:15:05_frequently       176K      -      344K  -
tank/docker_projects/portainer@autosnap_2021-01-28_00:30:00_frequently       176K      -      344K  -
tank/docker_projects/portainer@autosnap_2021-01-28_00:45:00_frequently       176K      -      344K  -
tank/docker_projects/portainer@autosnap_2021-01-28_01:00:00_frequently         0B      -      344K  -
tank/docker_projects/traefik@autosnap_2021-01-28_00:15:05_frequently          92K      -      476K  -
tank/docker_projects/traefik@autosnap_2021-01-28_00:30:00_frequently          92K      -      476K  -
tank/docker_projects/traefik@autosnap_2021-01-28_00:45:00_frequently          92K      -      476K  -
tank/docker_projects/traefik@autosnap_2021-01-28_01:00:00_frequently           0B      -      476K  -
tank/docker_projects/trilium@autosnap_2021-01-28_00:15:05_frequently           0B      -      196K  -
tank/docker_projects/trilium@autosnap_2021-01-28_00:30:00_frequently           0B      -      196K  -
tank/docker_projects/trilium@autosnap_2021-01-28_00:45:00_frequently           0B      -      196K  -
tank/docker_projects/trilium@autosnap_2021-01-28_01:00:00_frequently           0B      -      196K  -
tank/docker_projects/unifi@autosnap_2021-01-28_00:15:05_frequently          1.76M      -     30.6M  -
tank/docker_projects/unifi@autosnap_2021-01-28_00:30:00_frequently          1.96M      -     30.7M  -
tank/docker_projects/unifi@autosnap_2021-01-28_00:45:00_frequently          1.99M      -     30.8M  -
tank/docker_projects/unifi@autosnap_2021-01-28_01:00:00_frequently             0B      -     30.8M  -


Hourly ones:
Code:
$ 
zfs list -r tank/docker_projects -t snapshot | grep hourly
tank/docker_projects@autosnap_2021-01-28_00:00:00_hourly                       0B      -      548K  -
tank/docker_projects@autosnap_2021-01-28_01:00:00_hourly                       0B      -      548K  -
tank/docker_projects/certbot@autosnap_2021-01-28_00:00:00_hourly               0B      -      956K  -
tank/docker_projects/certbot@autosnap_2021-01-28_01:00:00_hourly               0B      -      956K  -
tank/docker_projects/gitea@autosnap_2021-01-28_00:00:00_hourly                 0B      -      196K  -
tank/docker_projects/gitea@autosnap_2021-01-28_01:00:00_hourly                 0B      -      196K  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_00:00:00_hourly         0B      -     19.8M  -
tank/docker_projects/homeassistant@autosnap_2021-01-28_01:00:00_hourly         0B      -     19.8M  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_00:00:00_hourly             0B      -      376K  -
tank/docker_projects/mosquitto@autosnap_2021-01-28_01:00:00_hourly             0B      -      376K  -
tank/docker_projects/portainer@autosnap_2021-01-28_00:00:00_hourly             0B      -      344K  -
tank/docker_projects/portainer@autosnap_2021-01-28_01:00:00_hourly             0B      -      344K  -
tank/docker_projects/traefik@autosnap_2021-01-28_00:00:00_hourly               0B      -      476K  -
tank/docker_projects/traefik@autosnap_2021-01-28_01:00:00_hourly               0B      -      476K  -
tank/docker_projects/trilium@autosnap_2021-01-28_00:00:00_hourly               0B      -      196K  -
tank/docker_projects/trilium@autosnap_2021-01-28_01:00:00_hourly               0B      -      196K  -
tank/docker_projects/unifi@autosnap_2021-01-28_00:00:00_hourly                 0B      -     30.5M  -
tank/docker_projects/unifi@autosnap_2021-01-28_01:00:00_hourly                 0B      -     30.8M  -
 

Richard Durso

Explorer
Joined
Jan 30, 2014
Messages
70
I've gotten the pull replication to work, I had my "%M" and "%m" mixed up.

wrong: `autosnap_%Y-%M-%d_%H:%m:00_hourly`

corrected: `autosnap_%Y-%m-%d_%H:%M:00_hourly`

TrueNAS created the new dataset and replicated the snaphots from the Ubuntu 20.04.1 docker server. I'm still not sure TrueNAS should have deleted a dataset I created just because no snapshots were found to replicate.

zfs_dataset_populated.png


Code:
root@truenas[/data]# zfs list -r main/docker/docker_backups
NAME                                       USED  AVAIL     REFER  MOUNTPOINT
main/docker/docker_backups                 153M  25.1T      524K  /mnt/home/docker/projects
main/docker/docker_backups/certbot        1.04M  25.1T      932K  /mnt/home/docker/projects/certbot
main/docker/docker_backups/gitea           300K  25.1T      172K  /mnt/home/docker/projects/gitea
main/docker/docker_backups/homeassistant  68.9M  25.1T     20.3M  /mnt/home/docker/projects/homeassistant
main/docker/docker_backups/mosquitto      2.77M  25.1T      352K  /mnt/home/docker/projects/mosquitto
main/docker/docker_backups/portainer      2.57M  25.1T      320K  /mnt/home/docker/projects/portainer
main/docker/docker_backups/traefik        1.73M  25.1T      460K  /mnt/home/docker/projects/traefik
main/docker/docker_backups/trilium         300K  25.1T      172K  /mnt/home/docker/projects/trilium
main/docker/docker_backups/unifi          74.8M  25.1T     35.1M  /mnt/home/docker/projects/unifi


Code:
root@truenas[/data]# zfs list -t snapshot main/docker/docker_backups   
NAME                                                              USED  AVAIL     REFER  MOUNTPOINT
main/docker/docker_backups@autosnap_2021-01-28_00:00:00_yearly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_00:00:00_monthly     8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_00:00:00_daily       8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_00:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_01:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_02:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_03:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_04:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_05:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_06:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_07:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_08:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_09:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_10:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_11:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_12:00:00_hourly      8K      -      524K  -
main/docker/docker_backups@autosnap_2021-01-28_13:00:00_hourly      0B      -      524K  -
 
Top