how to use zfs send | zfs receive to back up my current pools to a 10TB transfer disk?

[devnullius]

After I've noticed that cp isn't a fool-proof way of making a backup of your zfs pools' data, I was pointed to zfs send and received.

I looked around but I sure would like some help: how would one backup all /mnt/POOL/* data to /mnt/TRANSFERDISK?

I'm planning to add disks to my existing ZFS so I first need to back everything up. I also think I might be able to do it with zipping and rarring and catting etc. but if there's a specialized tool for this... That probably works better, no?

Devvie

 

[gpsguy]

Search the forum for a backup script used by @Arwen.

Note: You can't add disks to an existing vdev, but you can add another vdev and use it to extend your existing piool.

 

[Arwen]

The resource I wrote to backup a FreeNAS server to a locally attached disk is here;

https://forums.freenas.org/index.php?resources/how-to-backup-to-local-disks.26/

I wrote it with the intent of monthly backups, not one time. My script currently uses Rsync. Using ZFS send / receive is a better choice for a one time copies, but I don't have the syntax memorized.

 

[colmconn]

These are the commands I used in a situation similar to yours. Please don't use them blindly. Read the man pages for zfs and understand what the effect of each of the options is.

Note that the commands below are for an incremental send of data that was changed/added/deleted between the creation of the pool@migration_base snapshot and the subsequently created pool@migration_base_20160706 snapshot.

Create a snapshot

1. zfs snapshot -r pool@migration_base_20160706

Get details about what data will be transferred by a zfs send before actually sending the data

1. zfs send -Rvn -i pool@migration_base pool@migration_base_20160706

Send an incremental data to a target pool

1. zfs send -Rv -i pool@migration_base pool@migration_base_20160706 | zfs receive -Fd tank

 

[devnullius]

great feedback - thank you guys. Will give this another try tomorrow :) (but if I see that last reply's instructions... Sigh! Xd)

 

[devnullius]

I find it so complicating...

I now found this article http://jrs-s.net/2016/09/15/zfs-snapshots-and-cold-storage/ and started with that.

First thing I notice is that I don't have any snapshots to begin with... I have 1 Pool with 2 Setnames in it.
zfs list -rt all Mount/Setname gives just 1 result, no mentions of snapshots.

If that has consequences let me know please, while I continue to try to understand it all :(

Thank you!

 

[devnullius]

Quote:
"By contrast, if you use syncoid to replicate directly to a second server or pool, you never need more space than you took up in production, you no longer have the nail-biting period when you delete old fulls, and you no longer have to worry about going agonizingly through ten or a hundred separate incremental restores to get back where you were – you can delete snapshots from the back or the middle of a working dataset on another pool without any concern that it’ll break snapshots that come later than the one you’re deleting, and you can restore them all directly in a single operation by replicating in the opposite direction."

Isn't syncoid the best way for FreeNAS to backup a ZFS data pool to a single backup disk?

 

[devnullius]

Q: would rsync also work if I did have snapshots...? https://arstechnica.com/information...on-to-the-cloud-is-finally-here-and-its-fast/

Post-answer: as far as I can tell, yes... (???) It's just a LOT slower depending the circumstances when you are updating an existing backup of your pools...

 

[devnullius]

https://forums.freenas.org/index.ph...to-script-replication-of-zfs-snapshots.60313/

Let me side-track with a little investigation into this "syncoid" script ;-)

 

[devnullius]

Since I'm still on an OLD version of FreeNAS - for now I'll go for the rsync option and be done with it.

Then I'll look into latest firmwares etc and slowly start installing the latest FreeNAS. I believe some tools have been released by now that might make my life simpler.

I'll leave this note to myself: https://www.svennd.be/zfs-snapshots-of-proxmox-using-sanoid/ and http://unicolet.blogspot.nl/2016/05/from-0-to-zfs-replication-in-5m-with.html and https://arstechnica.com/information...using-the-zfs-next-gen-filesystem-on-linux/3/

 

[devnullius]

Would it be to be expected that the ZFS Pool's 'du -hs' gives 1.7T and the Pool's rsync copy gives 1.8T? Before I destroy the existing Pool :)

I should be safe and complete with my rsync copy, right?
​

 

[toadman]

Probably, but impossible to say since we don't know what commands you used or what was done to the pool during and after the sync. I suppose you could hash everything on both sides as a check if you want to be sure.

Yes, I think zfs send|receive would be simpler. :)

 

[devnullius]

just the regular rsync as advised on countless of fora. And it ain't really impossible... You could just follow the link I gave ;p https://arstechnica.com/information...on-to-the-cloud-is-finally-here-and-its-fast/

Well if zfs send / receive is easier... wtf do I do when I don't have any @snapshots but only data?

 

[wblock]

Make a snapshot of the data.

 

[toadman]

wtf do I do when I don't have any @snapshots but only data?

Yep, as @wblock saud, make a snapshot, then send it. @colmconn already posted an example of how to do it above.

Of course the limitation is the receiving disk must be formatted in zfs and have a pool present. You can't for example zfs send a snapshot to a disk formatted in NTFS.

 

[devnullius]

(sigh) I can make a snapshot first... Well that's fair enough :) I'll go through it once more

 

[danb35]

You can't for example zfs send a snapshot to a disk formatted in NTFS.

Maybe not to NTFS, but you can store a snapshot as a file on any filesystem the OS can write to. zfs send pool@snapshot > /path/to/somefile works just fine. You won't have direct access to the data in that state, though, but it can be useful for transporting data.

I'll go through it once more

It really isn't difficult:

• zfs snapshot -r POOL@snapname
• zfs send -Rv POOL@snapname | zfs recv -F TRANSFERDISK

Of course, it'd be much easier if the GUI provided a simple way to do local replication...

 

[toadman]

Maybe not to NTFS, but you can store a snapshot as a file on any filesystem the OS can write to. zfs send pool@snapshot > /path/to/somefile works just fine. You won't have direct access to the data in that state, though, but it can be useful for transporting data.

Awesome. I stand corrected. I never thought of doing that. Learn something new every day!

Yes, to get to the data it appears you would have to still zfs recv the snapshot back into a pool somewhere.

 

[danb35]

Yes, to get to the data it appears you would have to still zfs recv the snapshot back into a pool somewhere.

Indeed. Sending the output to a file doesn't give you a live filesystem to work with, but it lets you store it on any media and filesystem you want. It's Unix, after all--everything is a file. You could even send the output through jive if you wanted, though that would be pretty silly.

 

[toadman]

Of course, it'd be much easier if the GUI provided a simple way to do local replication...

The GUI does allow using "localhost" as the remote system to replicate to. But I've never tried it. None the less that would still be based on a periodic snapshot.

It would be a nice feature to have a "one time" replication option to a specific target, including a local pool and the file option you mentioned above. But I suppose one can just use the command line. :)