remote ZFS replication and bypassing the slow link

[technopop]

Two 36 drive 4u rack servers running FreeNAS that are 50Km apart.

I have a 10Mbit link between FreeNAS boxes and 15TB of additions to sync which would take around 150 days to do.

A car (or even on foot) is definitely the faster pipe.

Is there any way I can utilize that faster pipe to run the replication of the big snapshot (without actually moving one storage server next to the other)?

 

[MatthewSteinhoff]

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." -- Andrew S. Tanenbaum

How is your data provisioned/billed at each end? Can you temporary add bandwidth? Once a circuit is in place, some providers will allow you to increase and decrease capacity for short term events. Buy a month of 100Mbit (or faster) service. If you have a good relationship with your data center, you may even be able to get the capacity for free on a limited basis(*).

You might not even need to upgrade capacity at both ends. In many cases, bandwidth is not symmetrical. Might just need to add upstream bandwidth on the sending side; the receiving side may have plenty.

Cheers,
Matt

(* We had a situation where we needed extra bandwidth temporarily to bring in data from a new client. Their IT guy - who we were replacing - didn't want to provide any assistance and offered to burn a thousand DVDs and mail them to us COD. Our regular DC tech dragged over an ethernet cable, said 'you didn't get this from me', we brought up their MASSIVE bandwidth on a unused NIC, pulled down the new client's data then unplugged the DC's connection. Super fast. Hours instead of days. Finished up the final sync on our existing connection. A big middle finger to their guy who was giving us a hard time.)

 

[technopop]

One end is at a datacenter. I can have Gigabit if I want.

The other end is a 10Mbit up only. Residential/Business DSL.

 

[Chris Moore]

The other end is a 10Mbit up only. Residential/Business DSL.

15TB of additions to sync

You could connect something like an 8 or 10 TB drive and copy as much as you can, make the trip, then come back and copy the rest for a second trip. Unless you get a multi-drive enclosure and setup a pool of drives, you won't get 15TB in one shot... It isn't super hard to add a single drive as a separate pool. I have a 4-drive backup pool in my main NAS that was once in an external enclosure and may be again one day...
Do you have any unused drive bays to put it in?
If the servers are new enough to have USB 3.0 ports, you might be able to use that, but I have not tried it.

 

[MatthewSteinhoff]

The other end is a 10Mbit up only. Residential/Business DSL.

UGH! Yeah. You're screwed unless you can get more bandwidth to the house. Even high speed cable is probably capped at 25Mbit upstream. Assuming you could get that delivered. Even then you've only cut your sync to 75 days. Time to load up the hardware and road trip.

You've got 'Replication Stream Compression' set to 'plzip', right? If you have highly compressible data, that might help in your case.

Cheers,
Matt

 

[MatthewSteinhoff]

The biggest problem in this case is not the painfully slow transfer but an interrupted replication. ZFS replication won't resume (yet, coming soon) on a carrier drop. You could be 149 days into the transfer, lose your internet connection and have to start all over again.

Cheers,
Matt

 

[Stux]

The interesting question is if you replicate to one location and then replicate to another, is it the equivalent of replicating to first to last?

Ie a -> b -> c === a -> c

If so, you might be able to replicate to an external. Then replicate that to the server.

You can definitely save a replication stream to disk then replay that later.

 

[Ericloewe]

The biggest problem in this case is not the painfully slow transfer but an interrupted replication. ZFS replication won't resume (yet, coming soon) on a carrier drop. You could be 149 days into the transfer, lose your internet connection and have to start all over again.

Cheers,
Matt

I'm pretty sure it's in FreeNAS just not hooked up to the GUI yet.

The interesting question is if you replicate to one location and then replicate to another, is it the equivalent of replicating to first to last?

Ie a -> b -> c === a -> c

Yeah. It should be

 

[Evi Vanoost]

You could do a manual hard-disk transfer. I'm assuming you have the systems already replicated and need to do an incremental zfs send. Let's say you have 10x2TB hard drives that you can fill, drive over and unload (you could do this even over 2 or 3 trips with less hard drives or buy bigger drives, whatever works). You do need 2x the amount of space (for your 15TB, you need 30TB available) on your destination for this to work.

zfs send -i $INCREMENTSNAP $NAME@$CURRENTSNAP | \
$MBUFFERPATH -o /mnt/externalhd/snapshot_stream -D 1800M -A 'date > /mnt/externalhd/timestamp && read -p "Press Enter to Continue" < /dev/tty'

To get mbuffer, simply create a simple jail (named build or so), install mbuffer (pkg install mbuffer) and then from the host system you can call /mnt/pool/jails/build/usr/local/bin/mbuffer

Every time you see: "Press enter to continue", you disconnect the external hard drive, connect the new one and do "touch /mnt/externalhd/snapshot_stream" (the paths have to remain consistent). At the other end, you can copy each of the hard drive contents onto the array (copy them in order, hence the timestamp file), then cat all the stream-files together in order and use it as input to zfs recv.

I am working on a script right now that does this to stream "zfs send" using rclone (much faster than file-by-file) and it seems to be working okay.