Dealing with large Snapshots and Offsite Replication

[Matt Tyree]

What's the best way to handle large snapshots when you're replicating offsite?

I dumped about 75GB of 4K video onto my server yesterday and the snapshot upload has been running for over 14 hours now. As far as I can tell, there is no way to see what percentage of completion it's at.

When there are large changes like that, what are the options for the replication process? Just let it run? Is there a time limit on the rep task? What happens when the next task needs to run?

Thanks!

 

[Stux]

Well, the only alternative is to bring the offsite onsite and finish the replication over the LAN.

If you ssh in and type "top" you can see the progress but I think it's only accurate for replications less than 4GB. pity.

Someone posted a script which will email you replication progress.

There is no time limit and the same replication task will not double up, but others will run simultaneously.

 

[depasseg]

What's the best way to handle large snapshots when you're replicating offsite?

Nothing special, just let them run. I would schedule frequent snapshots so there isn't a huge backlog.

there is no way to see what percentage of completion it's at.

You are right, there isn't a percentage. If I'm curious, I track the remote dataset size and compare to the source.

Just let it run?

Yessir!

Is there a time limit on the rep task?

No, it will run until complete.

What happens when the next task needs to run?

It will queue behind the running task.

 

[Matt Tyree]

Well, the only alternative is to bring the offsite onsite and finish the replication over the LAN.

Yeah, that's what I figured would be the only fix for the issue. Not very feasible as the box isn't very easy to get to in the offsite location. Guess, I'll just let it run.

 

[Matt Tyree]

I would schedule frequent snapshots so there isn't a huge backlog.

My backups from my desktop to the NAS only run once a day, so I set up the replication to run every night. Otherwise, there's no new data on the NAS to snapshot and/or replicate more often.
If I ever start using the NAS for other storage beyond just backups, I'll change the snapshot/rep schemes accordingly.

 

[Stux]

My backups from my desktop to the NAS only run once a day, so I set up the replication to run every night. Otherwise, there's no new data on the NAS to snapshot and/or replicate more often.
If I ever start using the NAS for other storage beyond just backups, I'll change the snapshot/rep schemes accordingly.

If the problem is that the transfer is affecting your network, then you can always throttle it in the replication setup dialog. One would hope that your network connection can keep up with the average rate of change.... for the offsite backup... if it can, then the best thing is just to forget about it ;)

 

[Matt Tyree]

If the problem is that the transfer is affecting your network, then you can always throttle it in the replication setup dialog. One would hope that your network connection can keep up with the average rate of change.... for the offsite backup... if it can, then the best thing is just to forget about it ;)

Agreed. And yes, the network should do just fine. I'm on cable modem and the up speed is my pinch-point at 11-12Mbps. If only I had a balanced line and could get the 180Mbps down AND up! Planning to move next year where I can get 20Mbps up. Any little bit helps! Of course, that also puts me 100s of miles away from the offsite box, though. :-/

One other question about the snapshots on both boxes....Should the "Used" column match on both sides?