Issue with Replication task - don't copy all files

[Adamlorens]

Hi All,

I have 2 Truenas servers: Main(Truenas) and Backup(TrueBackup).

I've scheduled snapshots on Truenas every 15min, and Replication task on TrueBackup every 30 min.

Unfortunately new files are not showing in TrueBackup evern 1 day later.

Below logs from TrueBackup

[2022/11/22 10:40:00] INFO [Thread-16] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.8-hpn14v15)
[2022/11/22 10:40:00] INFO [Thread-16] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2022/11/22 10:40:00] INFO [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/11/22 10:40:00] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing pull from 'Main/Main_storage' to 'HT_Backup/Backup' of snapshot='auto-2022-11-22_10-30' incremental_base='auto-2022-11-22_10-00' receive_resume_token=None encryption=False
[2022/11/22 10:40:00] INFO [replication_task__task_1] [zettarepl.paramiko.replication_task__task_1.sftp] [chan 5] Opened sftp connection (server version 3)
[2022/11/22 10:40:00] INFO [replication_task__task_1] [zettarepl.transport.ssh_netcat] Automatically chose connect address '192.168.1.224'

Thanks in advance!!

 

[Patrick M. Hausen]

You made sure the destination dataset on the backup NAS is read-only? You cannot have a replication and mount the destination read/write.

 

[Adamlorens]

You made sure the destination dataset on the backup NAS is read-only? You cannot have a replication and mount the destination read/write.

Thanks! Is it enough to restric my ( only user) right to read only? if not where I can do it?

 

[Patrick M. Hausen]

That's not enough. The destination dataset must not be mounted read/write. The replication works on the block level. As soon as you change anything in the destination dataset it won't get updated anymore, only the snapshots.

You can enforce read-only mode for the destination in the replication task settings. You might have to start over. Or possibly roll back to the latest snapshot after manually setting read-only on the destination.

 

[Adamlorens]

Huge thanks!!

so I will:
1. remove current pool on TrueBackup
2.disable samba
3. create new replication task

so how I can check or even double check that replication tast is working and copy all files?

 

[Patrick M. Hausen]

You do not need to remove the pool. Just rename the current dataset and trigger the replication task again. You can also share via Samba to check the contents as long as the dataset is read-only on the ZFS level.

Otherwise simply check Storage > Snapshots - if all snapshots are present, all data is present.

 

[Adamlorens]

You do not need to remove the pool. Just rename the current dataset and trigger the replication task again. You can also share via Samba to check the contents as long as the dataset is read-only on the ZFS level.

Otherwise simply check Storage > Snapshots - if all snapshots are present, all data is present.

So probably last question, how I can make dataset read-only on the ZFS level?

 

[Patrick M. Hausen]

In your replication task:

 

[Adamlorens]

OK I've already had this, so now I can share via samba to double check?

 

[Adamlorens]

OK still have the problem snapshot on Main Truenas is different than snapshot in TrueBackup:

 

[Patrick M. Hausen]

Then check your replication task schedule, the status, etc.

 

[Adamlorens]

Then check your replication task schedule, the status, etc.

Status is finished Logs:

[2022/11/22 12:00:00] INFO [Thread-8] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.8-hpn14v15)
[2022/11/22 12:00:00] INFO [Thread-8] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2022/11/22 12:00:00] INFO [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/11/22 12:00:00] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing pull from 'Main/Main_storage' to 'HT_Backup/Backup' of snapshot='auto-2022-11-22_12-00' incremental_base='auto-2022-11-22_11-30' receive_resume_token=None encryption=False
[2022/11/22 12:00:00] INFO [replication_task__task_1] [zettarepl.paramiko.replication_task__task_1.sftp] [chan 5] Opened sftp connection (server version 3)
[2022/11/22 12:00:00] INFO [replication_task__task_1] [zettarepl.transport.ssh_netcat] Automatically chose connect address '192.168.1.224'

 

[Patrick M. Hausen]

What does zfs list -t snap show on both systems? Any hints?

I'm a bit at a loss of ideas, now.

Edit: I only run push tasks. So the snapshot being created triggers the replication. That works. Never tried pull myself.

 

[Adamlorens]

What does zfs list -t snap show on both systems? Any hints?

I'm a bit at a loss of ideas, now.

Edit: I only run push tasks. So the snapshot being created triggers the replication. That works. Never tried pull myself.

I removed some lines which are the same to avoid long reply

TrueNas main:

Last login: Mon Nov 21 13:23:31 on pts/1
FreeBSD 13.1-RELEASE-p1 n245406-814eb095751 TRUENAS

TrueNAS (c) 2009-2022, iXsystems, Inc.
All rights reserved.
TrueNAS code is released under the modified BSD license with some
files copyrighted by (c) iXsystems, Inc.

For more information, documentation, help or support, go here:
http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# zfs list -t snap
NAME USED AVAIL REFER MOUNTPOINT
Main/.system/samba4@wbc-1667999215 140K - 300K -
Main/.system/samba4@wbc-1668432937 140K - 308K -
Main/.system/samba4@wbc-1669103963 240K - 592K -
Main/Main_storage@auto-2022-11-15_05-37 0B - 7.33G -
Main/Main_storage@auto-2022-11-15_05-38 0B - 7.33G -
Main/Main_storage@auto-2022-11-15_06-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-15_06-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-15_06-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-15_07-00 0B - 7.33G -

Main/Main_storage@auto-2022-11-16_08-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_08-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_08-30 100K - 7.33G -
Main/Main_storage@auto-2022-11-16_08-45 508K - 7.33G -
Main/Main_storage@auto-2022-11-16_09-00 72K - 7.33G -
Main/Main_storage@auto-2022-11-16_09-15 64K - 7.33G -
Main/Main_storage@auto-2022-11-16_09-30 720K - 7.33G -
Main/Main_storage@auto-2022-11-16_09-45 848K - 7.33G -
Main/Main_storage@auto-2022-11-16_10-00 976K - 7.33G -
Main/Main_storage@auto-2022-11-16_10-15 1.08M - 7.33G -
Main/Main_storage@auto-2022-11-16_10-30 396K - 7.33G -
Main/Main_storage@auto-2022-11-16_10-45 916K - 7.33G -
Main/Main_storage@auto-2022-11-16_11-00 516K - 7.33G -
Main/Main_storage@auto-2022-11-16_11-15 260K - 7.33G -
Main/Main_storage@auto-2022-11-16_11-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_11-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_12-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_12-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_12-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_12-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_13-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_13-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_13-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_13-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_14-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_14-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_14-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_14-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_15-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_15-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_15-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_15-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_16-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_16-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_16-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_16-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_17-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_17-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_17-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_17-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_18-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_18-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_18-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_18-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_19-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_19-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_19-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_19-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_20-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_20-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_20-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_20-45 112K - 7.33G -
Main/Main_storage@auto-2022-11-16_21-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_21-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_21-30 56K - 7.33G -
Main/Main_storage@auto-2022-11-16_21-45 64K - 7.33G -
Main/Main_storage@auto-2022-11-16_22-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_22-15 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_22-30 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_22-45 0B - 7.33G -
Main/Main_storage@auto-2022-11-16_23-00 0B - 7.33G -
Main/Main_storage@auto-2022-11-17_06-00 0B - 7.35G -
Main/Main_storage@auto-2022-11-17_06-15 0B - 7.35G -
Main/Main_storage@auto-2022-11-17_06-30 84K - 7.35G -
Main/Main_storage@auto-2022-11-17_06-45 0B - 7.35G -
-
Main/Main_storage@auto-2022-11-17_22-15 0B - 7.35G -
Main/Main_storage@auto-2022-11-17_22-30 0B - 7.35G -
Main/Main_storage@auto-2022-11-17_22-45 88K - 7.35G -
Main/Main_storage@auto-2022-11-17_23-00 7.29M - 7.35G -
Main/Main_storage@auto-2022-11-18_06-00 0B - 64.0G -

Main/Main_storage@auto-2022-11-20_22-45 0B - 64.0G -
Main/Main_storage@auto-2022-11-20_23-00 0B - 64.0G -
Main/Main_storage@auto-2022-11-21_13-30 7.35M - 64.3G -
Main/Main_storage@auto-2022-11-21_14-00 256K - 66.4G -
Main/Main_storage@auto-2022-11-21_14-30 96K - 66.5G -
Main/Main_storage@auto-2022-11-21_15-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_15-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_16-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_16-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_17-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_17-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_18-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_18-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_19-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_19-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_20-00 200K - 66.5G -
Main/Main_storage@auto-2022-11-21_20-30 204K - 66.5G -
Main/Main_storage@auto-2022-11-21_21-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_21-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_22-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_22-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-21_23-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_06-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_06-30 208K - 66.5G -
Main/Main_storage@auto-2022-11-22_07-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_07-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_08-00 504K - 66.5G -
Main/Main_storage@auto-2022-11-22_08-30 144K - 66.5G -
Main/Main_storage@auto-2022-11-22_09-00 152K - 66.5G -
Main/Main_storage@auto-2022-11-22_09-30 224K - 66.5G -
Main/Main_storage@auto-2022-11-22_10-00 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_10-30 0B - 66.5G -
Main/Main_storage@auto-2022-11-22_11-00 216K - 66.5G -
Main/Main_storage@auto-2022-11-22_11-30 1.55M - 66.6G -

TrueNas Backup:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@TrueBackup[~]# zfs list -t snap
NAME USED AVAIL REFER MOUNTPOINT
HT_Backup/.system/samba4@wbc-1668582168 120K - 292K -
HT_Backup/.system/samba4@wbc-1668588705 120K - 292K -
HT_Backup/.system/samba4@wbc-1668590904 120K - 292K -
HT_Backup/.system/samba4@wbc-1668598332 120K - 292K -
HT_Backup/.system/samba4@wbc-1668598951 120K - 292K -
HT_Backup/.system/samba4@wbc-1668599098 120K - 292K -
HT_Backup/.system/samba4@wbc-1669039747 120K - 292K -
HT_Backup/Backup@auto-2022-11-15_05-37 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_05-38 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_06-00 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_06-30 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_06-45 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_07-00 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_07-15 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_07-30 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_07-45 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_08-00 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_08-15 0B - 5.16G -
HT_Backup/Backup@auto-2022-11-15_08-30 0B - 5.16G -

HT_Backup/Backup@auto-2022-11-17_20-30 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_20-45 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_21-00 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_21-15 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_21-30 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_21-45 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_22-00 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_22-15 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_22-30 0B - 5.17G -
HT_Backup/Backup@auto-2022-11-17_22-45 88K - 5.17G -
HT_Backup/Backup@auto-2022-11-17_23-00 4.39M - 5.17G -
HT_Backup/Backup@auto-2022-11-18_06-00 0B - 47.2G -
HT_Backup/Backup@auto-2022-11-18_06-15 0B - 47.2G -
HT_Backup/Backup@auto-2022-11-18_06-30 0B - 47.2G -

HT_Backup/Backup@auto-2022-11-20_22-30 0B - 47.2G -
HT_Backup/Backup@auto-2022-11-20_22-45 0B - 47.2G -
HT_Backup/Backup@auto-2022-11-20_23-00 0B - 47.2G -
HT_Backup/Backup@auto-2022-11-21_13-30 4.48M - 47.5G -
HT_Backup/Backup@auto-2022-11-21_14-00 144K - 48.2G -
HT_Backup/Backup@auto-2022-11-21_14-30 88K - 48.2G -
HT_Backup/Backup@auto-2022-11-21_15-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_15-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_16-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_16-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_17-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_17-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_18-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_18-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_19-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_19-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_20-00 192K - 48.2G -
HT_Backup/Backup@auto-2022-11-21_20-30 192K - 48.2G -
HT_Backup/Backup@auto-2022-11-21_21-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_21-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_22-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_22-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-21_23-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-22_06-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-22_06-30 192K - 48.2G -
HT_Backup/Backup@auto-2022-11-22_07-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-22_07-30 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-22_08-00 412K - 48.2G -
HT_Backup/Backup@auto-2022-11-22_08-30 144K - 48.2G -
HT_Backup/Backup@auto-2022-11-22_09-00 152K - 48.2G -
HT_Backup/Backup@auto-2022-11-22_09-30 224K - 48.2G -
HT_Backup/Backup@auto-2022-11-22_10-00 0B - 48.2G -
HT_Backup/Backup@auto-2022-11-22_10-30

 

[Patrick M. Hausen]

This looks completely erratic on the backup side. Can you try push instead?

 

[Adamlorens]

This looks completely erratic on the backup side. Can you try push instead?

It's also weird for me, maybe something wrong on main TrueNas?

cannot change:

 

[Patrick M. Hausen]

The push task must be configured on the main TrueNAS ...

Main NAS - Snapshot - Push Task --> Backup NAS.

 

[Adamlorens]

The push task must be configured on the main TrueNAS ...

Main NAS - Snapshot - Push Task --> Backup NAS.

I will try do it tomorrow, thanks a lot for you support!

 

[Adamlorens]

The push task must be configured on the main TrueNAS ...

Main NAS - Snapshot - Push Task --> Backup NAS.

Hi!
Still have the same issue, snapshots on Main are different than od backup, no mather is it push or pull.

 

[Patrick M. Hausen]

Sorry, never seen that with my systems. Out of ideas. Open an issue in JIRA.