zfs pool replication via sneakernet

[cfcaballero]

Hello:

I have two FreeNAS servers (9.10) in different countries, let's call them A&B.

I want to have them replicate each other, and I can run that with test servers just fine.

But B is new/empty and A has 6 TB of data and a slow internet connection (just 1Mb/s upload), so it is really a non-starter to do the first replication via SSH, and I cannot bring server B to location A or vice-versa.

I saw this post and understand that process fine and have tested with test-data on server B.

https://www.cod3r.com/2015/12/freenas-internal-backups/

My question is then, can I use that process to indirectly replicate pool A to server B, a sort of transitive replication?

I have a single drives (shingled archive drive and USB JBOD array) large enough to hold the entire pool from A that I can carry from location A to B. If I "ingest" the pool from A into server B using localhost replication with removable/external drives, as in that post, will I then be able to replicate new periodic snapshots from A to B?

Obviously (despite my searching), I don't understand what goes over the wire in replication over SSH, is it file data ala rsync, or disk sectors, etc.?

I'm aware that I could use the same transportable single drive to rsync the files from A to B via the drive, but again, would that result in incremental snapshots successfully replicating changes in A to B? Especially since I eventually want changes replicated in both directions.

Thanks in advance for any advice.

 

[danb35]

https://www.cod3r.com/2015/12/freenas-internal-backups/

That link doesn't work for me. But you should be able to do a sneakernet replication. zfs send outputs a datastream, which can be treated as a file (everything's a file in Unix-land). The most common thing to do with that datastream is to pipe it to zfs recv, but you can just print it to your screen, pipe it through more, send it to paper tape, or (probably most usefully in this context) redirect it to a file. That file can be on any filesystem, as long as FreeNAS can read from, and write to, it. So, the process would go like this:

• Create a snapshoot
• zfs send (options) pool/dataset@snap > /path/to/backup/disk/file.zfs. The .zfs extension is arbitrary.
  
• Take backup disk to the other server
• zfs recv (options) pool/dataset < /path/to/backup/disk/file.zfs

 

[cfcaballero]

Many thanks, danb35!!! I hadn't realized the send/receive commands could put the pools in replicable state.

 

[cfcaballero]

Updating this in case anyone else needs this kind of setup ("seeding" a replication with a physical drive, whether external or internal).

Following the instructions in the link in the OP does the trick for setting up the transportable external pool. Once you move it to the target system and import the volume (very important - NOT the disk), then you setup the local replication in reverse, let it run and get up to date, then delete or disable it.

Now you can setup the remote replication, and (this is VERY important) as long as you have at least two overlapping snapshots on the two systems, you will be fine. In other words, it can NOT be the case that the newest snapshot on the target system is also the oldest snapshot on the source system. If that is the case, it seems to want to send and/or compare the entire volume/dataset.

I also learned that if your source system has multiple datasets that you do recursive snapshots for, you will have to create a "fake" periodic snapshot task just for the datasets you want to replicate (and leave it disabled) in order for the datasets to appear on the dropdown list for the replication task.

Also, make sure that even if your replication is for a single dataset that you specify ONLY the volume for the remote, or it will create a different dataset on the remote/target.

 

[cfcaballero]

Forgot to add, I only wound up getting this working on FreeNAS-11.0-U4. I forget what the problem was with 9.10.