Manual Replication Tasks

[Roger Wilco]

Hi,

I would like to propose a "Manual Replication Tasks" feature, what do you think about the following:

Feature request:
FreeNAS should have a configurable one-click solution to replicate datasets.

Motivation (not ordered by importance)
Currently datasets can only be replicated periodically, which is not always the best solution:

- Datasets used for "dumping data" like photos, do not need to be snapshot and replicated periodically - it would be sufficient, to initiate replication after new data has been added .Though this is more or less cosmetics, as empty snapshots do not hurt, it is neither nice nor necessary.

- Some datasets cannot (should not) be replicated periodically (e.g. datasets used for backups like Time Machine). If a time machine backup is in progress and a snapshot is taken, the data will most probably be garbage. It would need to make sure, that no backup is in progress (probably shutting down AFP) before taking the snapshot (see feature request #7308).

- The manual replication tasks functionality would make (partial) disaster recovery quite simple - if something happens to the main box, one can replicate dataset by dataset back from the backup server - I do not want to copy files back and forth, I would prefer clicking one button (more or less).

- I prefer to backup really important data (kids photos and alike) on a third offline box (lightning strike) which is not connected to the power line (do not have an offsite backup possibility). The "manual replication" feature would make things easier - a button click on the main box and new photos would replicate to the offline box. This is not possible with the current periodic snapshots+replication tasks, as each failed replication produces a warning, which is fine but not in this case - I do not dump my photos every day but maybe once a moth.

- Usability. I do not deal with ZFS on a daily base and in case I have to restore a dataset from the backup (or hopefully not the offline) box, I do not want to look up the ZFS CLI parameters first.


All these examples can of course be solved by scripts, but if one thinks about problems that may arise (replication failure due to network issues, launching the script before a replication has been finished,...) one comes to the conclusion, that it needs a mechanism like FreeNAS already has implemented for replication tasks (holds + tags). So what implementing things twice?

User Interface:
I would suggest a "Manual Replication Tasks" button in UI->Storage, which would open a page similar to the existing "Replication Tasks" page - a table with all configured man. rep. tasks. The only difference would be an additional "REPLICATE" button in the last column of each configured task. One would of course not need the Begin/End/Enabled columns.
There would also be ADD / EDIT / DELETE buttons, exactly like on the existing page.

Metadata:
The naming of the snapshots could be similar to the existing ones, something like "manual-20150321.0230-2w"

Snapshot handling:
Not sure - when replicating from a backup, it would certainly be nice to have all existing snapshots replicated back. So there might be some sort of selection of the snapshot...


What do you think about this proposal?

Regards,
rw

 

[DrKK]

you can replicate data sets any time you want to with zfs send, right at the command line.

Just make a snapshot, and do a zfs send.

?

or am I missing something.

 

[Roger Wilco]

@DrKK

Hi,
Well, I think you are missing something :)
It's partially my fault, as I was talking about incremental ZFS sending but never wrote it in my post.

Of course one can use zfs snapshot, send, receive on the CLI to replicate a dataset to somewhere else.
It is in a way a convenience feature (like e.g. the "Create Snapshot" in UI->Storage), but not only.

As soon as you have to do things more than once, you'll start scripting the whole thing (that's actually what I did, too).

But with incremental send there are things to consider which make your script bigger and bigger:
- Has the previous run succeeded or do I have to send the last snapshot again?
- How do I find the last successfully sent snapshot?
- How do I find pending snapshots (i.e. those which have not been successfully sent and need to be resent)
- Can I send, or is currently a replication of the dataset in progress?
- I want to get a mail, in case something fails.
- ...

Then you dump your shell script and start over in python.
Then you'll start asking yourself why wasting time, as the periodic snapshot + auto replication features have all of the needed functionality already implemented.
Then you'll take a look at the autosnap.py and autorepl.py scripts and come to the conclusion, that it would be only minor changes to create the Manual Replication Task feature.

And that's where I am now - I do not want to reinvent the wheel and write the same thing one more time.

I hope I could clarify a bit what I mean...

Bye,
rw

 

[survive]

Hi guys,

I personally don't see anything wrong with this idea, assuming I understand it in fact I rather like it.

I can think of a few time I would have liked to have been able to just replicate a dataset to another via the GUI. A button in FreeNAS that would automate a one-off replication would be great.

-Will

 

[SilverJS]

Seconded.

 

[Roger Wilco]

Created feature request #9077