zfs send with replication

[Binary Buddha]

So, I'm trying to figure out the difference between

Code:

zfs send -R

and

Code:

zfs send

. According to the man page:

Generate a replication stream package, which will replicate the specified filesystem, and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved.

I've used both and they both seem to do that if you do

Code:

zfs snapshot -r

.

 

[depasseg]

In simple terms -R includes the snapshots of child datasets, otherwise they won't be sent.

If the thing you are running snapshot -r on doesn't have any child datasets, then you won't see a difference.

 

[Binary Buddha]

In simple terms -R includes the snapshots of child datasets, otherwise they won't be sent.

If the thing you are running snapshot -r on doesn't have any child datasets, then you won't see a difference.

So, if I have a dataset with child datasets and I pipe zfs send -R to a file would that be an equivalent to a full backup? From my understanding, zfs snapshots are intended for a "quick recently happened" restore from backup and not a "real" backup.

 

[Robert Trevellyan]

A snapshot is not a backup. A replicated snapshot is a backup.

pipe zfs send -R to a file would that be an equivalent to a full backup?

Yes, if it's on a different storage device.

 

[Binary Buddha]

Yes, if it's on a different storage device.

So, if I do a zfs send -R (backup A) of "snapshot A" and then I do another snapshot (snapshot B) would a piped zfs send -I to a file between snapshots A and B be considered an "incremental backup"? And so on and so forth, acknowledging that I would need the file from snapshot A; or the like, and the "incrementals" in between to restore.

 

[depasseg]

So, if I do a zfs send -R (backup A) of "snapshot A" and then I do another snapshot (snapshot B) would a piped zfs send -I to a file between snapshots A and B be considered an "incremental backup"? And so on and so forth, acknowledging that I would need the file from snapshot A; or the like, and the "incrementals" in between to restore.

Not really, if -I is used, all snapshots between snapA and snapB are sent and they will be there completely. When you replicate a snapshot, the entire set of data is replicated for the files contained in that snapshot. If you are using -I, then it will also copy the relevant snapshot data.

 

[Binary Buddha]

Not really, if -I is used, all snapshots between snapA and snapB are sent and they will be there completely. When you replicate a snapshot, the entire set of data is replicated for the files contained in that snapshot. If you are using -I, then it will also copy the relevant snapshot data.

AAAANNNNDDD.... you lost me...

 

[depasseg]

Do some more reading on how ZFS works.

You appear to understand backups, and snapshots work sorta like incremental backups, however when you go to replicate, YOU can't just must the incremental changes. The snapshot will also contain the original data.

Think of snapshots as just your data at a specific time. When you replicate, your data at that place in time is copied. If you choose multiple snapshots (like using -I) then all the snapshots between the two will also get replicated.

 

[Binary Buddha]

Do some more reading on how ZFS works.

You appear to understand backups, and snapshots work sorta like incremental backups, however when you go to replicate, YOU can't just must the incremental changes. The snapshot will also contain the original data.

Think of snapshots as just your data at a specific time. When you replicate, your data at that place in time is copied. If you choose multiple snapshots (like using -I) then all the snapshots between the two will also get replicated.

LOL... I'm asking here all the reading I've done still doesn't make sense.

So, I thought that snapshots were just a "diff" of the original data. And incremental snapshots (zfs send -I) were a time based grouping of those diffs. And when you say "replicate" I assume you mean zfs send -R, right?

New file gets made data written to disk, snapshot is taken recording this new "original" data. Changes are made to the file, the "new" data as result of the change is written to somewhere else and not overwriting the original data. Another snapshot gets taken and it records this "new" data referencing the old data to fill in. And another (third) snapshot is new data referencing the previous snapshot; and inherently the original data from the first snapshot. Replicating (-R) the snapshot sends all the diffs along with the original data? And not using replication (-R) doesn't send the original data?

 

[depasseg]

A snapshot contains the diff, but when you replicate, you need to start with all the data. So during replication ZFS will apply the "diffs" and then replicate all that data. If you use the -I option, then ZFS replication will also include the "diffs" for the snapshots after the first one.

IOW, there is no way that I know of to ONLY replicate just the diffs, you must have at least one full set of your data (with the appropriate diffs incorporated) and then just the diffs will replicate.

 

[Binary Buddha]

So ZFS replication is only really useful if you want to send the child datasets with permissions right?


Sent from my iPhone using Tapatalk

 

[depasseg]

So ZFS replication is only really useful if you want to send the child datasets with permissions right?

No. I don't know how you go that idea.

ZFS replication is only useful of you want a replica of 1 or more datasets at either 1 or many snapshots (points in time). If you want the ability to go back in time, you use snapshots. If you want the ability to survive the physical destruction of your primary server, you use replication.

 

[Binary Buddha]

So using zfs send without -I and -R would result also in a full backup of the current files, but without the child datasets and snapshots?


Sent from my iPhone using Tapatalk

 

[depasseg]

For your hypothetical, please type out the exact command to avoid any uncertainty.
And why are we discussing using the command line? Is there something shorebird you are trying to do? Why not use the GUI?

 

[Binary Buddha]

For your hypothetical, please type out the exact command to avoid any uncertainty.
And why are we discussing using the command line? Is there something shorebird you are trying to do? Why not use the GUI?

zfs send some_dataset@some_snapshot > some_file

I'm a Unix guy. Command-line is less questions than trying to understand what point and click is doing exactly. No specific goal that I'm trying to accomplish. Just trying to learn how the zfs features work in comparison to other filesystem. Although leveraging zfs capabilities to optimize traditional backup schemes would be useful. Rsync and tar takes longer and have larger sizes than zfs snapshots to a non-zfs medium. IMO rsync is good for backups but lacks incremental control; always doing a full backup. Tar was made for a now becoming obsolete technology; tape drives, and takes forever on large amounts of data that can no longer fit on a single tape. The cost to implement a full tape backup of 8tb+ is impractical for home use. However, zfs snapshots to a file that's on a shared "unlimited" web host is a bit more practical. ... For example.


Sent from my iPhone using Tapatalk

 

[depasseg]

I can understand wanting to learn, and you are right, the CLI is a better place to learn what's going on. I'd suggest reading up on ZFS to gain more of that insight, I bet someone probably did what you might be thinking of.

As for the command, the reason for asking you to write it out, was to see that when running zfs send, it's sending that single snapshot. -R would send that same snapshot from all child datasets.

 

[droeders]

IMO rsync is good for backups but lacks incremental control; always doing a full backup.

If you like rsync but want incremental backups with it, checkout rsnapshot:

http://rsnapshot.org/

I've been using it for years, and love it.

 

[Binary Buddha]

If you like rsync but want incremental backups with it, checkout rsnapshot:

http://rsnapshot.org/

I've been using it for years, and love it.

Still takes forever due to having to build the file list.


Sent from my iPhone using Tapatalk