relative beginner needs help on setting up a backup plan

[johnnychicago]

Hello --

So I've been setting up Freenas on an existing machine. Coming from Linux, I've used LVM on software raid for the best part of a decade. This is a home setup.

I am getting along reasonably well with the basics, no pressing need for support there. The documentation is quite exhaustive, so I am doing ok.

What I am after is how to set up my backup solution. I am lazy, so the backup just has to work automated. Here's what I used to do: a second machine, not raided, just LVM'ed (to save on drives), on the network, and I rsynced my mounts daily on to that one, using hard links to the previous day's directory. Keeping the daily directories for two months, sunday's for half a year, and first sunday of the month for two years. I found that quite nice to be able to get files from long ago, and I've done a total restore from it, and was happy with how that went too.

Starting new, I was figuring to do it better and put the backup box on the internet (precisely: at my parent's). I was figuring I'd 'prime' it with initial backups on the LAN and then move it out. My daily changes are small enough to make this work, it would take maybe an hour every night - two if I do housekeeping.

I am currently experimenting with periodic snapshots and replication tasks. That goes pretty easily, my test backup machine gets the snapshots pretty instantly. Cool stuff.

The bit I am currently missing is the organisational one.

* How can I choose which snapshots are being backed up? For convenience I'd like to have regular snapshots during the day, but would really only need to back up one nightly.

* How am I going to purge unwanted snapshots from the remote machine? As far as I can see currently, I can either remove them based on the main NAS deleting them (e.g. after 2 weeks), or they will stay forever, in very fine granularity. Or I can delete them manually. Is this a problem that has been solved? I figure a small shell script would allow me to take out according to some granularity like the above mentioned, but I'm new around here, and it will take a bit of a learning curve to come up with such script.

* For added complexity, I'd like the backup machine to sleep when not needed. I could wake it up from the BIOS at midnight, and I understand there's ways to upgrade a dynamic DNS entry. However currently I am not quite sure how to approach putting it back to sleep. I figure no established SSH connection for 20 minutes or so would be a decent criteria.

* for the main NAS (as well as the backup, come to think of it) is there a smart way to create shares with the snapshots? I'd imagine something that would give me a share with every existing snapshot being a subfolder named for the snapshot date, and I could just mount the whole shebang and look for a given file in all its versions in the different snapshot folders.

I would enjoy getting some directions - I feel somewhat insecure, since I'm quite set in my ways on my previous environment, and this feels new and not yet comfy to me. Am I having a good approach here, or are these things done differently? I'm quite ok with changing my ways, my problem is that I am currently not quite sure what to search for.

Thanks for reading.

 

[depasseg]

First thing is separate snap shots and replication. Create multiple snapshot tasks on different schedules (hourly during the day, once at night, every week, every month, etc). Then choose the retention accordingly. This is your roll back possibility.

Now consider replication. Replication runs once per minute unless you change when it is allowed to run. This will establish an ssh connection to do the check, so your sleep idea might take some tweaking. When replication runs, it will make the destination the same as the source, snapshot wise, and then prune according to what you've configured. I would let dailies and weeklies expire and delete. Maybe keep all monthlies.

 

[johnnychicago]

@depasseg, I've figured out the bit about running different snapshots for daily/weeky/monthly, instead of selectively deleting snapshots. Obviously two snapshots taken at different schedules but at the same time are pretty much free of charge. I like the approach of leaving the daily/weekly on the main box and letting the pruning happen there and propagate to the backup, while letting the monthlies go stale. I can live with deleting those manually every now and then. Good setup, thanks!

So I'll start looking at the backup really just as insurance in case of a big booboo - like major hardware trouble, fire, theft. Currently I am using the backup to look up older versions of files, but I will be able to do that on snapshots.

Any insights on making snapshots easily available? As said, I am currently just having the users read-only mount shares on my backup server to get to old files. I can see doing that on the main box, since the backup will be inconveniently far away anyway. However I currently do not see how I can make the snapshots accessible in an easy and automated fashion. Manually cloning and creating a share for each snapshot does not sound like fun. Is there a better way to enable this kind of functionality?

 

[depasseg]

If you are using Windows, there is a previous versions tab on the file properties tab. There is a setting in freenas somewhere (sorry I'm mobile ATM) to provide the snapshot data to that previous version tab. I don't know any simple way to mount the snapshots in an easy to navigate manner.

 

[johnnychicago]

Thanks for the tip on the previous versions, but Windows is a bit of a rarity here. SWMBO's laptop and an XP VM to humor some legacy software is all the Windows in use.

I've played around a bit with the replication, and switching off the receiving host seems to be taken in good stride: Freenas complains, but as soon as the remote is back, it seems to pick up where it left. I wanted to investigate on how to capture the recipient having caught up and then shut it down.

As for the snapshots, I was googling around a bit (I just couldn't imagine that nobody ever wanted to mount all the snapshots), and in a non freenas related forum I found a tip on checking the .zfs/snapshot folder in the dataset. That one seems to do exactly what I am after, it shows one folder for each snapshot, and I've just tried sharing it read-only.

 

[David3D]

If you are using Windows, there is a previous versions tab on the file properties tab. There is a setting in freenas somewhere (sorry I'm mobile ATM) to provide the snapshot data to that previous version tab. I don't know any simple way to mount the snapshots in an easy to navigate manner.

Also looking for a similar management solution.

My current knowledge has me queuing a Free File Sync job to investigate files that have changed and sync them back to the external drive hooked up to a Windows server. I'm still very fuzzy on the details of snapshots, replication and how they work together. I have a lot of reading to do.

I am very happy to hear you can configure snapshots to interact with the "previous versions" tab within Windows! Of course I'm anticipating it would be quite hassle to get it working properly. Once done though, it would be stupid-proof for everyone else.

 

[depasseg]

I haven't played with it (shadow copies/previous versions), but this is the relevant section of the manual: http://doc.freenas.org/9.3/freenas_sharing.html#configuring-shadow-copies

 

[johnnychicago]

Drats, I've misunderstood things when it comes to pruning old snapshots. I really seem not to be able to do that differently on the backup machine compared to the life one. I can only autodelete on life and let the backup follow. That is inconvenient, since after a while, it will start getting expensive on space in the life.

 

[depasseg]

Think of snapshots separate from the replcation and figure out a couple schedules that work for you. Figure out the pruning, etc on the live system.

Then turn your attention to the replication tasks to the backup system. At this point you have one more option to consider, which is whether or not to ever delete a snapshot. You might still choose to not delete the monthlies, but auto-delete the others.

 

[johnnychicago]

And that's where I think it's not working - or I am not understanding it correctly.

On the replication, the one thing I can tune is to not ever delete snapshots on the backup. That applies to all of them, not just the monthlies. I'm just running hourlies that die after 2 days and dailies that keep for 2 weeks to test it - and when I let them go stale on the replication, they all do. So effectively that would give me all hourlies forever on the backup.

 

[depasseg]

OMG, I'm an idiot. I'm sorry. You are right, replication is by dataset (and all corresponding snapshots that go with that dataset). I can't think of a way to only do that with one set of snapshots.

 

[johnnychicago]

OK, so at least I understood that bit :)

My next thing to try out would be setting up regular snapshots on the backup box - I wonder if the replication sender will purge those, too - or if it will only remove those it originally sent.

 

[johnnychicago]

OK, I am not really getting far with this - I've set up snapshots on the backup machine, but they seem not to be doing what I expected.

For this test, I used hourly and daily snapshots on the main box, replicating them to the backup, and on the backup I've added snapshots holding longer, up to a year. I've not yet exactly figured out what happened, but it seems clear that the longer snapshots on the backup box seem either not to get created at all, or purged pretty much immediately. Short ones seem to work ok.

I am not quite sure what's going wrong here, but I am currently still assuming it's my fault. So, to clarify, what is the way to hold snapshots on the backup that are older than the ones on the main? This is somewhat important to me, and currently I can't really imagine nobody else caring about this at all?

Do I really have to go back to rsync to do long term backups? I was so looking forward to having everything in ZFS and just replicated around!