Backup to external drive

[Argentic]

Hi community,

I'm still up setting my system and need some help. I want to backup the data from the FreeNAS to external drives, preferably directly connected, without another PC in between. I intend to rotate two drives, with the least amout of manual interaction. Ideally that would be just changing the drive.

1) USB/eSATA:
I have read, that there are problems with USB with automatic mounting/detecting? Is that right? Does it work without problems with eSATA? Which one can I just plug in and then the scheduled backup works?
I have already USB 3.0 drives, but I can get eSATA cases.

2) Configuration:
How can I configurate it? Is there an option in the GUI for that? Or a plugin? Or do I have to use scripts ...? :-/

3) Features/Strategy:
The simplest strategy would be a weekly full backup and rotate for the next one.
Better would be something like this:
- Full backup after (weekly/monthly) rotation (should automatically detected that full backup is necessary)
- Daily incremental backup (for usabillity with hardlinks or a similar method (snapshots?), so recovery is easy)
- A notification, if a backup failed (email)
Hm ... It would need to delete the disk after a rotation before the new full backup.

Is there a solution for this? Or one that comes near to it?

 

[Nick2253]

There's nothing built in that does what you want to do. Your going to need to do your own scripting to accomplish what you want.

In my view, there are two ways to do this: use rsync to copy the files to the drive, or format it as ZFS and replicate the dataset(s). More information on these options can be found in the documentation and elsewhere on the forums. It will be up to you to script those solutions in a way that meets all the rest of your requirements.

 

[anodos]

3) Features/Strategy:
The simplest strategy would be a weekly full backup and rotate for the next one.
Better would be something like this:
- Full backup after (weekly/monthly) rotation (should automatically detected that full backup is necessary)
- Daily incremental backup (for usabillity with hardlinks or a similar method (snapshots?), so recovery is easy)
- A notification, if a backup failed (email)
Hm ... It would need to delete the disk after a rotation before the new full backup.

Is there a solution for this? Or one that comes near to it?

There's no simple way to meet these requirements. Technically, you could install the bacula file daemon / client in a FreeNAS jail and add your storage to the jail via the webgui. Then install the bacula storage daemon / server on other hardware (system requirements are fairly minimal). Backup speed would be limited to 1gbps, but that probably plenty in most cases. The only catch is that you have to configure bacula ;)

Alternatively, you can theoretically create devfs rules to pass your USB devices to the bacula jail directly, and install the bacula-sd in the jail as well (so that you have both the file and storage daemons in the same jail). The only catch then is that you only have to properly configure your devfs rules and bacula.

Of course I'm using 'only' in this context like a marathon is 'only' a run. ;)

 

[Robert Trevellyan]

Does it work without problems with eSATA?

Yes, generally, but make sure to use a suitable controller, avoid port multipliers and keep the cables within spec.

 

[Argentic]

Of course it isn't that easy ...

Well, let's get through the possibilities. What are common rotation backup solutions?
Sorry for these noobish questions, but I dont have much experience in backup-systems, let alone with FreeNAS.

The (little expanded) setup: FreeNAS, USB-(or eSATA)-Drives, with or without another PC for the task.
Important would be the features from above, that it does consistent backups on rotating drives. Not just the changed files from the last backup, but checking the difference to the drive, because of the rotation.
There should be a notification on errors (or successes?) too.

About a direct attached backup drive, which should be removed (for rotation) (from another thread):

If you want to take the backup offsite, you should definitely detach the pool using the Detach Volume button in GUI, then shut the box down and remove the drives. In this scenario, automated replication isn't going to be very useful.

Does that mean I would/should do that at every rotation?

use rsync to copy the files to the drive,

Would that work with a direct attached drive? Or do I have to setup a client on a PC?
With either of these, would the rest work as wanted?

or format it as ZFS and replicate the dataset(s)

Does this work with a single drive? Could it detect read errors and bit rot, because of the ZFS?

Technically, you could install the bacula file daemon / client in a FreeNAS jail and add your storage to the jail via the webgui. Then install the bacula storage daemon / server on other hardware (system requirements are fairly minimal). [...]

Of course I'm using 'only' in this context like a marathon is 'only' a run. ;)

I hoped for a simpler solution ... I would prefer not to pour hours and hours into this task.


What are the solutions with a PC in between?
Again rsync?
A backup program?
Over SSH, mounting the data via CIFS, other?

I am still searching for a backup programm for the PCs in the network. Which works with hardlinks too. Should I backup the FreeNAS via CIFS I could probably use the same software for it. That is if it works correctly with rotating drives.

 

[depasseg]

If it were me, and I absolutely had to have 2 separate external, offline storage backups here's how I would do it:

Main pool called "tank", first backup called "backup-A" and second backup called "backup-B"

Create a snapshot schedule that meets your needs (hourly, retain for a week; weekly, retain for 4 weeks; monthly retain for 12 months).
Then create 2 replication tasks- source dataset: tank; destination dataset: backup-A; destination host: 127.0.0.1 (and a second task with a destination of backup-B)
Leave one drive attached for a week, at the end of the week, detach backup-A, attach backup-B
You will likely receive failure errors since one of the datasets won't be avaialble, so you can use the GUI to "detach" the offline drive, but that requires you to re-add the replication task when you "attach" it.

 

[Nick2253]

Of course it isn't that easy ...

Well, let's get through the possibilities. What are common rotation backup solutions?
Sorry for these noobish questions, but I dont have much experience in backup-systems, let alone with FreeNAS.

The backup rotation is entirely up to your needs for both data retention, and data access. There are too many variables for us to advise you properly. You need to understand why you are trying to backup (what you are trying to accomplish), and do some reading on different backup schemes to come to your own conclusion on this.

The (little expanded) setup: FreeNAS, USB-(or eSATA)-Drives, with or without another PC for the task.
Important would be the features from above, that it does consistent backups on rotating drives. Not just the changed files from the last backup, but checking the difference to the drive, because of the rotation.
There should be a notification on errors (or successes?) too.

The features that you want will be up to you to implement. Again, there is nothing built in to FreeNAS that does what you want, so you'll have to script/program it.

If you are willing to bring in another PC to do the task, then your job gets much easier. There's a lot of software out there (especially for Windows) that will take the data from your NAS via the network, incrementally write it to your hard drive, notifying you of errors, etc. It won't be as fast as direct connect, but that may not even be necessary depending on the amount of data you have, and the frequency of backup that you want.

Would that work with a direct attached drive? Or do I have to setup a client on a PC?
With either of these, would the rest work as wanted?

You could use rysnc to copy files locally. rysnc will also give you options for incremental and error checking. You need to read up on rysnc: https://www.freebsd.org/cgi/man.cgi...ath=FreeBSD+8.0-RELEASE+and+Ports&format=html

Does this work with a single drive? Could it detect read errors and bit rot, because of the ZFS?

Yes, there's nothing about ZFS that requires multiple drives. However, with one drive, you have no redundancy. ZFS will still checksum your data, so it will know if the data is bad, but with one drive, you won't have another copy to restore from.

I hoped for a simpler solution ... I would prefer not to pour hours and hours into this task.

Frankly, FreeNAS is not really designed to support what you want, so there won't be a simple solution with FreeNAS. If you want to do backup the "FreeNAS Way", you would use ZFS replication to move data between live pools, and employ snapshots to handle point-in-time recovery.

 

[Argentic]

Thanks depasseg, this method should work for either solution. Regardless if I will realise it directly attached or with another software and a PC in between.


@Nick2253
I've read some and I think the rotation is a good solution for the given circumstances.

I will look, if there is a good software, which helps me with the task. I put that in another thread, because this is more about possibilities and general questions, while the new one here is about recommendations for a software alone.
I will try the easier way first (bringing another PC into the task), but the rest is good to know too and maybe it is helpful for other people.
I like the idea, that a backup on a ZFS-drive would detect reading errors even at no redundancy, because if the rotation cycle is short it is very probable, that you have another copy of the file on the other drive. But for now I'm with the simpler solution happy.

Thanks to everyone so far. :)

 

[robsonj]

Sorry to hijack the thread, but I have a similar requirement to the OP.

I have a main store attached to a windows box which routinely gets robocopied over the network to my newly built FreeNAS machine (currently still populating it with data). I would then like to have an externally attached (via eSata) drive which is a backup of my most important FreeNAS datasets (photos & family movies).

So if I read correctly. I create a second pool on that eSata attached drive and save snapshots of those datasets to it using replication? I'm happy to leave the eSata drive attached all of the time as its purpose is to ensure I always have three copies (plus an offsite).

Again, sorry for hijacking the thread, but I haven't really found a good clear answer or tutorial on how to do this.

Cheers
Jonathan

 

[Robert Trevellyan]

I create a second pool on that eSata attached drive and save snapshots of those datasets to it using replication?

Correct.

 

[robsonj]

Can I save snapshots from multiple datasets to one 'backup' pool. Or do I have to replicate the dataset configuration on the 'backup' pool with my main pool?

Sorry I would try this myself, but my tobo backup drive is currently holding a second copy of my data while it is copying to the FreeNAS box currently, after which point it can be repurposed as the backup - 3rd copy drive.

Cheers
Jonathan

 

[depasseg]

If you are going to use a single pool as a target for multiple replication sources, you should create a separate dataset on the target for each source. And you should configure the source replication to point to the destintion pool/dataset-name.

 

[Argentic]

Hi, I'm back.
Now I tend to the replication solution. It seems very nice and I tested it a little (with just one replication task).
The problems I see are:

1. The permanent error message, due to one missing drive and volume (because of the rotation)
2. The problem of discouraged hot plugging, which apparently can lead to problems -> see this thread

Is there any solution for the first point? Custom rules, so it doesn't trigger an error message, as long there is one of the two volumes/drives available wont be possible, I guess.
The alternative would be exporting one drive and importing the other AND disabling one replication task and re-enabling the other, because both things would trigger an error message, right?
Or creating scripts, so it is semi-automated. Via Cron Jobs (using "Run Now") or Init/Shotdown Scripts? Would that be possible?

I guess the second point should be discussed in the according thread, so everything is in the same place, right?

 

[depasseg]

The only way to avoid the error is to "detach" the pool. You might want to make each backup drive it's own pool, that way you can detach the whole pool and import the second pool. You could probably use the api to script the detach and possible recreation of the replication task (it might go away when you detach).

 

[Argentic]

Ok, so I have to look into scripting.
I've found the Useful Scripts section in the forum, but where do I find more resources, if it isn't enough?
A tutorial on these scripts in general would be nice (this could be also not FreeNAS specific, right?) and sites/documentation where I can find the needed commands, like mounting, importing, manipulating replication (and probably snapshots and scrub) tasks, getting the status of different stuff (like is the replication running or is the drive/volume idle so I can export it?)
I know to program, but don't know FreeNAS or FreeBSD (or Unix based systems) . But with the right documentation and resources I should be able to do it or estimate if it is too much work/takes too much time.

 

[Bidule0hm]

As always, the answer is in the documentation: http://doc.freenas.org/9.3/freenas_api.html :)

 

[Argentic]

As always, the answer is in the documentation: http://doc.freenas.org/9.3/freenas_api.html :)

Good to know, but you are talking about python scripts.
So there are multiple possibilities. Python scripts, shell scripts, what else? What would be probably enough for my scenario?

Also is there a way to see the commands from GUI actions? When I create e.g. a replication task, can the executed command be found in any kind of log? I could use that as a starting point for the individual tasks.

 

[Bidule0hm]

The API is a web API so as long as your language can do GET/ POST/PUT/... requests it's fine.

Well, FreeNAS is open source so of course you can see what the code do, just read the sources :)