rsync off-site backup question

Status
Not open for further replies.

kompact

Cadet
Joined
Feb 6, 2017
Messages
3
Hello all! This is my first post, so please bare with me. I have tried googling and searching this forum and FreeBSD documentation as much as I could and I got quite far, but sometimes that doesn't really help grasp a concept.

Anyway, here's what I'm trying to do:

I have a 4TB mirrored Volume (2x WD Red) and a 4TB striped Volume (1x WD Green). The mirrored volume called "kompact_nas" will be my primary "working" storage and I'd like to clone it regularly to the striped volume called "kompact_backup".

On kompact_nas, I have created 3 datasets, 2 of them shared via SMB and one via AFP for timemachine backup.

I tried copying the whole thing from kompact_nas to kompact_backup with the following rsync command:

rsync -avxPHAXW -h /mnt/kompact_nas /mnt/kompact_backup

This didn't really copy anything as far as I'm concerned, so I went ahead and started copying one of the datasets instead, like this:

rsync -avxPHAXW -h /mnt/kompact_nas/Time\ Machine /mnt/kompact_backup/Time\ Machine

I repeated the process with the remaining datasets and it seemed to work. I even browsed some directories on the copied volume to make sure the files are there, which they are.

So my questions are:
Am I doing this right, or is there a right way to do this?
Will I be able to recover my volume from a backup like this?
Specifically, I'd like to know if I have to create the corresponding datasets on /mnt/kompact_backup first, so I can also see them in the "Storage" section, or is the rsync I did enough?
And finally: would there be an easier way to rsync the whole volume without having to start the process again for 3 separate datasets?

I hope this makes sense at all and I'm looking forward to your responses! Thanks! :)
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
First, keep your rsync commands simple, at most you want [a]rchive, [v]erbose and [r]ecursive. If you want to see the progress, include --progress:

rsync -avr --progress /source /destination

The other way to do it would be sending/receiving snapshots. You could set this up by setting up a replication task to itself: http://doc.freenas.org/9.10/storage.html#replication-tasks
 

kompact

Cadet
Joined
Feb 6, 2017
Messages
3
Wow, that was fast. Thanks! I will use simpler commands next time, but will I be able to work with what I have already copied so far (it took quite a while to sync the datasets and I'd like to avoid having to do it all over again)? Regarding the (non-existent) datasets on the target - am I good with copying one by one as I did or do I have to create the datasets on the target first? I'd prefer to stick with rsync if possible.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Welcome to the forums!

To copy the entire volume, I believe you can simply invoke rsync with your kompact_nas pool as the source and the kompact_backup pool as the target, like this:
Code:
rsync -rltgoDhv /mnt/kompact_nas/ /mnt/kompact_backup

Note that placing the trailing slash on the source and leaving it off the target are critically important.

This approach will copy every dataset in the kompact_nas pool, including the syslog and home directories, which may not be what you want. For more fine-grained control, write a shell script that only copies the three datasets you mentioned:
Code:
#!/bin/bash
rsync -rltgoDhv --delete-during --inplace /mnt/kompact_nas/Time\ Machine/ /mnt/kompact_backup/Time\ Machine
rsync -rltgoDhv --delete-during --inplace /mnt/kompact_nas/SMB_share1/ /mnt/kompact_backup/SMB_share1
rsync -rltgoDhv --delete-during --inplace /mnt/kompact_nas/SMB_share2/ /mnt/kompact_backup/SMB_share1

The '--delete-during --inplace' options tell rsync to delete any files on the target that don't exist on the source, so that the target dataset will be a perfect copy of the source.

Here are two rsync-related threads that may help you out, in them I explain why I use the -rltgoDhv options with rsync:

https://forums.freenas.org/index.ph...rmissions-support-for-windows-datasets.43973/
https://forums.freenas.org/index.php?threads/easy-rsync-task-creation.46373/

Good luck!
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
So my questions are:
Am I doing this right, or is there a right way to do this?
Will I be able to recover my volume from a backup like this?
Specifically, I'd like to know if I have to create the corresponding datasets on /mnt/kompact_backup first, so I can also see them in the "Storage" section, or is the rsync I did enough?
And finally: would there be an easier way to rsync the whole volume without having to start the process again for 3 separate datasets?
There is always more than one way to do something like this... :)
Using the second approach I outlined above, you'll be able to recover whatever datasets you copy to the backup pool.
Yes, I would create the datasets on the target pool - so that FreeNAS knows about them.
The backup process will take a long time to complete when you run it the first time. After that it will be much faster, as rsync is intelligent and knows to copy only new or modified files.
 

kompact

Cadet
Joined
Feb 6, 2017
Messages
3
Thank you all very much! This is what I was looking for.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Note that rsync only transfers filesystem contents, not structure. If you create the datasets ahead of time on the destination, the transferred files will end up in the right place, but any new dataset you create only on the source will appear as a regular directory on the destination. To preserve the full filesystem structure, use ZFS snapshot replication.
 
Status
Not open for further replies.
Top