Backup one dataset to another dataset on same FreeNAS machine?

[TXAG26]

I have a FreeNAS machine setup with the following:
-4x 2TB HDD's setup as ZFS RaidZ2 (3.6TB total)
-1x 4TB HDD

My plan is to have the 4x HDD RaidZ2 dataset backup to the single 4TB HDD, but I'm not sure how to get this setup. It doesn't need to be a continuous backup/mirror. I'd just like a way to set it up so it runs maybe once a day, or even once a week, and just simply keeps an updated copy of the Raidz2 data on the single 4TB HDD.

I looked at the options in Rsync and Replication, and both of those look like they only work for backing up data between two different physical FreeNAS machines.

Any help would be greatly appreciated!

 

[joeschmuck]

Well I'd think you could create a cron job to run every so often as determined by you. This could be every 10 minutes or every day. Rsync can be used to copy files locally. Take a look at this page: http://www.freebsd.org/cgi/man.cgi?....0-RELEASE+and+Ports&arch=default&format=html

 

[TXAG26]

Is there any way to set this up within the FreeNAS GUI web interface? To be perfectly candid, I've never tinkered with FreeNAS behind the curtains so to speak.

 

[joeschmuck]

I have no clue to be honest, I've never used Rsync. Looking at the GUI it wasn't obvious to me. Until someone tells you the exact thing to type you could create a Virtual Machine of a FreeNAS 9.1 machine with three hard drives, one 4GB for the FreeNAS OS, one 10GB for the main drive, and one 10GB for the backup drive. Next install FreeNAS to the VM and run it. You will need to configure it like a real machine, transfer some data to one of the drives, then try to use rsync commands using the link I gave you as a guide. Once you can copy your data to the backup drive then add some more data and see if you can make that copy without problem. Eventually you should have some simple command that you type in to make the rsync work. You would take that and in the GUI create a CRON job, make the time maybe 5 minutes for testing purposes and drop in the rsync line. Now that will be executed every 5 minutes. Start adding and changing data and see if it get copied to the backup drive. This is how I would do it and I don't know much about FreeBSD, just enough to get into trouble.

Whatever you do, before you execute a command on your real machine, BACKUP YOUR DATA! Off the machine. If you make a mistake and copy the wrong direction, well you won't be happy if you deleted something you can't recover that you really wanted to keep.

Good Luck!

EDIT: Creating a VM is easy. You can use VMWare Player (one I have used) or Virtual Box, both are free.

 

[TXAG26]

Thanks for your help, I'll give that a try and see if I can get something figured out. I guess, worst case, I could always spin up another instance of FreeNAS on this same machine and get them to talk to each other via the ESXi virtual switch. I'd rather not do this as it doesn't seem necessary. Plus, I'd have to move the 4TB HDD off of the LSI 2308 that I have passed through to ESXi via Vt-d and plug it instead onto the motherboard chipset SATA ports.

 

[pirateghost]

You can use rsync in the GUI or you can set up cron job with rsync params
If you use RSYNC, in the GUI, the 'remote machine' would be localhost

 

[TXAG26]

Is it really as simple as just typing in 'localhost' for the remote target??? Wow, this should work great if that's the case!

 

[pirateghost]

yeah. just use localhost instead of another machine name. i have configured a few rsync jobs like this in the past. i usually just use cron jobs for it though so i can fine tune the times that it runs.

here is a cronjob i configured for my freenas box to backup my shared xbmc profile from one dataset to another:

Code:

rsync -rvuAth --log-file=/mnt/lun1/xbmc.log /mnt/lun1/xbmc/* /mnt/lun2/xbmc/

 

[joeschmuck]

See, I knew someone would come to the rescue. That helps me too should I choose to do something like that.

 

[Dusan]

It is also the same with replication. Just use localhost as the destination. Replication is also faster than rsync. While rsync has to check every file and compare at least the size & modification time, replication knows "instantly" which blocks changed between snapshots. I use replication to backup the most important data to an removable drive (actually two of them that I rotate for offsite backup).

 

[TXAG26]

Excellent! Thanks for the quick schooling on Rsync and Replication. Can't wait to give these a try this weekend.

 

[joeschmuck]

Please post your results since I'm interested on how you do this.

 

[TXAG26]

Here's where I'm at:
I have two volumes: "fs1" and "fs1backup". I want all of the files on "fs1" to backup/copy over to "fs1backup" (which is on another HDD inside the same machine).

1) I enabled snapshots on fs1 (see first pic for config).
2) I then setup a Replication Task (see second & third pics)

I have both volumes shared and mounted as drives in Windows, yet the replication doesn't seem to be working as no files are showing up on "fs1backup".

 

[Dusan]

Check /var/log/messages for any error reported by the replication. Did you set up the replication ssh keys as described in the manual? Does a pool named zpool1backup contain a dataset named fs1backup?

 

[TXAG26]

Replication doesn't appear to be working, logs are saying something about a broken pipe? I'm not a plumber, so maybe someone knows what this means!


Sep 22 20:04:02 FreeNAS autorepl.py: [tools.autorepl:266] Destroying remote zpool1backup/fs1backup/fs1

Sep 22 20:04:02 FreeNAS autorepl.py: [common.pipesubr:71] Executing: /usr/bin/ssh -c arcfour256,arcfour128,blowfish-cbc,aes128-
ctr,aes192-ctr,aes256-ctr -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 localhost /sbin/zfs destro
y -rRf zpool1backup/fs1backup/fs1

Sep 22 20:04:02 FreeNAS autorepl.py: [common.pipesubr:71] Executing: (/sbin/zfs send zpool1/fs1@auto-20130921.2158-2w | /bin/dd
obs=1m | /bin/dd obs=1m | /usr/bin/ssh -c arcfour256,arcfour128,blowfish-cbc,aes128-ctr,aes192-ctr,aes256-ctr -i /data/ssh/replicat
ion -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 localhost "/sbin/zfs receive -F -d zpool1backup/fs1backup && echo Succeed
ed.") > /tmp/repl-37649 2>&1

Sep 22 20:04:02 FreeNAS autorepl.py: [common.pipesubr:57] Popen()ing: /usr/bin/ssh -c arcfour256,arcfour128,blowfish-cbc,aes128
-ctr,aes192-ctr,aes256-ctr -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 localhost "zfs list -Hr -
o name -t snapshot -d 1 zpool1backup/fs1backup/fs1 | tail -n 1 | cut -d@ -f2"

Sep 22 20:04:02 FreeNAS autorepl.py: [tools.autorepl:325] Replication of zpool1/fs1@auto-20130921.2158-2w failed with dd: stdou
t: Broken pipe 2048+0 records in 0+0 records out 0 bytes transferred in 0.012457 secs (0 bytes/sec) dd: stdout: Broken pipe 4087+20
records in 1+0 records out 1048576 bytes transferred in 0.012503 secs (83866565 bytes/sec) warning: cannot send 'zpool1/fs1@auto-201
30921.2158-2w': Broken pipe

 

[Dusan]

Is your SSH service running?

 

[TXAG26]

Is your SSH service running?

Yes, SSH is "enabled" in Services using Port 22 and the "Login as Root with Password" box and the "Password Authentication" box are both CHECKED. Everything else is unchecked/blank.

 

[Dusan]

I already asked, but I have to ask again. Did you set the replication SSH key for the root user?
See Configure PULL here: http://doc.freenas.org/index.php/Replication_Tasks
Do the both PULL & PUSH actions on the same computer.
There is also a troubleshooting section at the end of the wiki page.

 

[TXAG26]

I already asked, but I have to ask again. Did you set the replication SSH key for the root user?
See Configure PULL here: http://doc.freenas.org/index.php/Replication_Tasks
Do the both PULL & PUSH actions on the same computers.
There is also a troubleshooting section at the end of the wiki page.

I completely missed that part of your question. I just ran:

ssh -vv -i /data/ssh/replication localhost

and it asked for a password, so I'll have to dig deeper to see what's causing that. It looks like the key isn't setup correctly if it asks for a password. In the Replication setup, I did copy the key into both of the boxes, and initially, I thought that might have been what you were asking about. In 9.1.1 it actually wouldn't let you finish adding the Replication Task until you did that, but I think the key may be needed somewhere else too according to the doc. I'll dig into the SSH and key issue today and see what I can find. Thanks for your help!!!

 

[Dusan]

You need to copy the replication key (View public key button on the replication tab) to the root user's public ssh key field (Account -> Users).