Rsync --link-dest Backups

[jackydany]

Hi,

thanks to you folks, i have my rsync up and running.
Its backing up from my Freenas by rsync-module to a remote OpenMediaVault server. Pushing.
Also i am backing up my webserver via ssh with ssh-keypair

Now i want to configure incremental backups.
Therefor i search the net and found out, rsync has a nice option called link-dest for this case.

What i want to do:

5 rsync tasks.
1. task full backup on monday to /mnt/data/Backup/backup0
2.-5. task incremental backup tuesday to friday to /mnt/data/Backup/backup1-4

My question is:

I dont know how hardllinks are really handled.
If i OVERWRITE backup0 (on next monday) will the files that backup1-4 are referenced to still work?
If so, will backup1 be as big as backup0 used to be before? and so on....


monday 1.1.15 -> backup0 (size 10mb) full
tuesday 2.1.15 -> backup1 (size 5kb) inc referenced to backup0
wednesday 3.1.15 -> backup2 (size 10kb) inc referenced to backup0
thursday 4.1.15 -> backup3 (size 15kb) inc referenced to backup0
friday 5.1.15 -> backup4 (size 20kb) inc referenced to backup0

monday 8.1.15 -> backup0 (size 10mb) full
what about backup1 to 4 now?
Are they still working? Or are they broken because i overwrote the backup0 ?
If working... is backup1 now 10mb + 5kb? I know, IF this is working, the files changed from monday to tuesday are only available with date-tuesday. Monday is gone for good :D

Hopefully you understand my question and where my concerns are.

Thank you very much

 

[fracai]

Hardlinks don't really have an "original" file. Both links point to the same data. You can use rsync to "move" files by using link-dest set to your source directory and also using remove-source-files. The files you want to move are hardlinked and the originals deleted, leaving only one copy. It's much faster than standard rsync and allows you to merge directories, unlink mv.

Anyway, rather than five rsync tasks I'd have one that performs the rsync task, and one that moves / deletes the folders. Basically, increment the number assigned to each backup (0 -> 1, 1 -> 2, ...) and then rsync from source to backup0, with link-dest set to backup1. This is what the "rsnapshot" tool does. You might be able to just copy over the rsnapshot script; I think it's a single perl file, if not you can use a jail.

If you manually delete one of the backup directories, the hardlinks will be removed, but the other copies will still point to the data. This is pretty similar to how ZFS works with snapshots. One difference is that 'du' handles hardlinks a bit nicer. If you run 'du' on all your backup directories, you'll see only the space taken by each backup. In other words, backup0 would look like a full backup and every other backup would be deltas. More specifically, you'll only see disk space counted once. This is a bit better than counting ZFS snapshot size, where disk usage can be hidden when multiple snapshots contain the same data (it will appear to not be counted at all in the snapshot used space column).

So, use "rsnapshot" if you can, but if you can't you can get by with one task. something like:

Code:

rm -rf backup5 && mv backup{4,5} && mv backup{3,4} && mv backup{2,3} && mv backup{1,2} && mv backup{0,1} && rsync --link-dest backup1 source backup0

 

[jackydany]

Hi,

I am again working on a good solution for my backup plan.

Regarding to what you supposed, i will NOT create a rsync task but just a normal task, correct?

In this cron job-task i will use a .sh script which is on my local freenas system.
Lets call the script backup.sh

My intention now is to do a backup every day and keep the last 20 backups. Using -link-dest to save only the files which differ from the last backup.
I will also include the whole rsync call in the script, so i will not have a "move script" to move backup0 to backup1 and one rsync task. If one of them fails, the other will be "corrupted" (not the files will be corrupted but my backup-schedule is not correct anymore).

My settings:
Local Freenas = freenas
Mount on the Freenas where the backup should be copied to = /mnt/daten/Backup/vServer/

Remote Server = Server
Source = / (root)

Using User root with ssh key.
Using Pull (from Remote Server to Freenas System, started and scheduled on Freenas)

Code:

rm -rf backup.20 && mv backup.{19,20} && mv backup.{18,19} && mv backup.{17,18} && mv backup.{16,17} && mv backup.{15,16} && mv backup.{14,15} && mv backup.{13,14} && mv backup.{12,13} && mv backup.{11,12} && mv backup.{10,11} && mv backup.{09,10} && mv backup.{08,09} && mv backup.{07,08} && mv backup.{06,07} && mv backup.{05,06} && mv backup.{04,05} && mv backup.{03,04} && mv backup.{02,03} && mv backup.{01,02} && mv backup.{00,01} && rsync -avcEAXzhb root@Server:/ /mnt/daten/Backup/vServer/ --link-dest=/mnt/daten/Backup/vServer/backup.01/ --delete --backup-dir=/mnt/daten/Backup/vServer/DELETED/`date +%Y-%m-%d`/

Is this one big command correct? :D

It will first remove the last backup backup.20
Then it will move every other backup to n+1
Then it will run a rsync task as archive, verbose, preserving group, owner, times, acl, extend attr etc, human readable, compression AND backup the deleted files to backupdir on the Freenas to /mnt/daten/Backup/vServer/DELETED/"the actual DATE" but only copy the files which are new (new to checksum) because of link-dest to backup.00

I will schedule this task every 1 day at 2' am

Is this correct? Its a little bit confusing :D

Thanks in advance!

Jack

 

[fracai]

I'm not too familiar with the "backup-dir" option, but your command doesn't actually copy anything into "backup.00". I'd think that would need to be your destination (...root@Server:/ /mnt/daten/Backup/vServer/backup.00/...).

Actually, on my reading of what backup-dir does, when combined with link-dest, won't there never be anything to delete? I'd have to experiment with that.

Plus, if you're putting this in a script I'd recommend splitting things to separate lines to keep things readable. And also use a loop rather than specifying each backup manually. You can check the exit status of each command to still abort if there's a failure somewhere.

Before too long we'll have re-implemented all of rsnapshot ;-)

Code:

#!/bin/bash
DEST="/mnt/daten/Backup/vServer/"
CURRENT_BACKUP=20
while [ "$CURRENT_BACKUP" -gt "0" ]
do
    rm -rf "${DEST}/backup.${CURRENT_BACKUP}" || exit
    let BACKUP_TO_MOVE=CURRENT_BACKUP-1
    mv "${DEST}/backup."{${BACKUP_TO_MOVE},${CURRENT_BACKUP}} || exit
    CURRENT_BACKUP=$BACKUP_TO_MOVE
done
rsync -avcEAXzhb root@Server:/ "$DEST"/backup.0/ --link-dest="$DEST"/backup.1/ --delete --backup-dir="$DEST"/DELETED/$(date +%Y-%m-%d)/

As always, "I haven't tested this". (I should add that to my signature...)