How to replicate to multiple destinations?

[depasseg]

I'm at a loss figuring this out.

I'd like to replicate (either the pool or all it's datase's) to another FreeNAS machine as well as a different pool on the same server.

I got an error message when trying to configure replication of the same dataset to 2 different locations. So I created a snapshot and replication task at the pool level, so the individual datasets can replicate locally and the pool replicates to the other FreeNAS.

This sorta works, except the Pool only replicates the dataset structure, but not their contents. I've tried both without and with recursive snapshots and replication options. I can't get anything to work.

Is there any way to get data replicated to 2 locations?

 

[cyberjock]

You cannot replicate from one place to more than one other place. Likewise you cannot replicate from one place to another place if the other place is going to attempt to replicate to a 3rd place.

FreeNAS places holds on the snapshots themselves at various stages to prevent them from expiring. For this reason you can't have one dataset's snapshots going to more than one place and any given place is either receiving or sending snapshots (never both).

 

[depasseg]

So does that only apply to freenas created snapshots? Would I be able to script a daily snapshot and replication separate from what the GUI provides without affecting the freenas snapshots and replication?

 

[Bidule0hm]

But the replication process outputs a standard flux, no?

 

[cyberjock]

So does that only apply to freenas created snapshots? Would I be able to script a daily snapshot and replication separate from what the GUI provides without affecting the freenas snapshots and replication?

If you plan to write code to manage your own snapshotting and replication, then sure. The killer is all those edge cases (what if a snapshot should expire but hasn't replicated, what if a snapshot should expire but there is no newer snapshot that has replicated, etc.)

It's doable. Not something you'll be able to just recode with the same kind of control and power, but it is doable.

The replication and snapshoting is something like 500 lines of code in FreeNAS. So you'll be trying to do the same thing with how many less lines? ;)

 

[depasseg]

My thought was to script a manual snapshot, replicate it, and then delete it. Something like this which has the added benefit of keeping 2 snapshots so I can do incrementals.
https://github.com/leprechau/zfs-replicate

 

[cyberjock]

That may or may not work on FreeNAS/FreeBSD. It doesn't say if it's designed for Linux, FreeBSD, Solaris, etc.

Just as a warning, you can't "just delete it". You have to keep the source and destination in-sync for replication to work. You can't replicate from A -> B and then delete it from A. If you do, then the next snapshot you make on A will not be in sync with B, and you will have to re-replicate all of the data all over again.

 

[depasseg]

Looking through the scripts, it's using zfs commands (and "find"), so fingers are crossed that it works.

As for the delete, yeah, I wasn't a fan of re-replicating, but since this script (and I found a couple others that work the same), keep at least the last couple snapshots so that incremental replication is possible.

 

[Ericloewe]

Likewise you cannot replicate from one place to another place if the other place is going to attempt to replicate to a 3rd place.

There go my grandiose plans for a hierarchy of backup servers.

Any word on making this a bit more flexible with FreeNAS 10?

 

[depasseg]

You mean something like this baked in: http://www.znapzend.org/

 

[cyberjock]

Looking through the scripts, it's using zfs commands (and "find"), so fingers are crossed that it works.

As for the delete, yeah, I wasn't a fan of re-replicating, but since this script (and I found a couple others that work the same), keep at least the last couple snapshots so that incremental replication is possible.

So how are they keeping the last couple of snapshots from being removed due to them expiring? My first guess would be ZFS holds, which is exactly what FreeNAS uses and is exactly why it doesn't work on FreeNAS.

Likewise, FreeNAS 10 is supposed to allow for multiple destinations and such. But I have no control over if/when it gets implemented, but I'd say that when FreeNAS 10 comes out you should be able to prove if this feature is or isn't in FreeNAS by looking at the manual.

 

[depasseg]

It looks like zfs-replicate just keeps an array of how many snapshots it's taken. If the counter exceeds the configured amount, then it destroys the oldest snapshot. I don't think zfs-replicate supports multiple destinations (since the config file only allows one host). However, it looks like zrep does. zrep also seems to be a little more mature. Either way, I'm just looking for a second set of snapshots to go to a replication target. zrep seems to be working so far. the only drawback is that it didn't handle recursive datasets, so I had to initialize replication manually on each of my dataset/fs individually. The nice thing is that it's a simple command to update all of them after they are initialized, which I can run via cron.

 

[cyberjock]

I hope it doesn't sound like I'm saying you are a fool for doing what you are doing. But there's lots of edge cases and situations that people don't think of, and it sucks when your replication suddenly stops working after weeks or months and you don't find out until your primary server goes down that you see your backups are 3 months old. :P

 

[depasseg]

Nope, I appreciate being called out.

This is the second replication, (well third if you include Crashplan). FreeNas is replicating to another direct attached pool on the same server. I didn't use znapzend (co-developed by Tobias Oetiker). I went withZrep. Zrep is running zfs snapshots and sends/receive to a second local server, and so far I'm comfortable with the risk. I've read through the entire shell script (1000 lines). It's definitely something that I would love to get away from using, because it's another thing to keep track of (if I create a new dataset, I need to add the replication task separately in zrep). But it should do until FreeNAS10. The only thing I see it adding is some additional properties to the datasets:

 

[pbrown]

... I don't think zfs-replicate supports multiple destinations (since the config file only allows one host). However, it looks like zrep does. zrep also seems to be a little more mature. Either way, I'm just looking for a second set of snapshots to go to a replication target. zrep seems to be working so far. the only drawback is that it didn't handle recursive datasets, so I had to initialize replication manually on each of my dataset/fs individually. The nice thing is that it's a simple command to update all of them after they are initialized, which I can run via cron.

Update for the sake of web search posterity :)
zrep does now support automatic recursion for a dataset, if you explicitly tell it to do that.
Details are at
http://www.bolthole.com/solaris/zrep/zrep.documentation.html#recursion

 

[depasseg]

I just got your github update regarding this zrep capability.

FreeNAS added the ability to manage multiple replication destinations a couple months ago (as well as multiple snapshot schedules). I've been using it and it seems to work pretty well.

 

[RyanJennings]

FreeNAS added the ability to manage multiple replication destinations a couple months ago (as well as multiple snapshot schedules). I've been using it and it seems to work pretty well.[/QUOTE]

Could you tell me how you are managing multiple destinations? Are you just creating another replication task and selecting the same dataset again or is there a way to specify more than one destination in a single replication task. Thanks for the help.

 

[depasseg]

I just created separate replication tasks.

I don't think you can set multiple destinations in a single task.

 

[Todd Ladouceur]

I just created separate replication tasks.

I don't think you can set multiple destinations in a single task.

What version of Freenas are you using?

 

[depasseg]

FreeNAS-9.3-STABLE-201602031011