RSYNC Task Interrupted - Now What?

[Thousandbuckle]

Hello all, I have a question which might be simple but I dont seem to know the answer.

I had created and RSYNC Task in TrueNAS Core 13.0-U5.3 to RSYNC around 16TB of data from one TrueNAS server to another on the same local network. The job got interrupted by a power outage today and when the server came back up I restarted the Task that I had created in Tasks-RSYNC (see image below). During the initial transfer it had copied around 14.5 TB to the backup server according to the backup server dataset size and I was getting around 907 Mb/s on the network transfer rate. The RSYNC Task has been running now since I restarted it for about 3 hours and I see very little data traffic only around 1 Mb/s or less.

Is the RSYNC task taking a long time to re-compare the source and the target data before it resumes copying the data again

 

[winnielinnie]

Is the RSYNC task taking a long time to re-compare the source and the target data before it resumes copying the data again

Most likely, yes.

To further add salt to your wound, a powerloss means you've lost all metadata in RAM, so it needs to retrieve (from disk) all the metadata represented by 16TB worth of files. This, itself, is a slow process.

A question arises: Why are you using Rsync to transfer from TrueNAS to TrueNAS? Why not ZFS replication?

 

[Thousandbuckle]

I thought RSYNC would be better option as I dont want to take up space on the source server for snapshots unless I am not understanding ZFS snapshots.

 

[winnielinnie]

Snapshots don't take up extra space. They only "hold" blocks that were deleted/modified.

If you've got a live filesystem with 10 TB of data, then you take a snapshot, the snapshot will hold zero bytes of storage. If later on you delete a few MB from your live filesystem, it means you don't actually "free up" those few MB, since they still exist in the snapshot.

In ZFS, the only time any space is "freed up" in a pool is when there are no longer any pointers to the blocks, either from a snapshot or live filesystem. (As long as there is even a single reference to the block, then it still consumes space on the pool.)

 

[Thousandbuckle]

So if my Snapshot Task had a Lifetime of 2 weeks, that would mean every 2 weeks when it expired and a new one was taken it would release the old reference data and free up those blocks?

 

[winnielinnie]

Not exactly. The only time any space would be freed up is if all references to the particular block(s) are gone. As long as any snapshot still points to the block(s), then such space won't be freed up. (It's quite common for multiple snapshots to point to the same blocks.)