I've been having all sorts of problems with replication (especially when recursion is used). This mostly comes down the the replication scripts being quite fragile with respect to changes in the auto snapshot structure. Let me be clear that I'm looking at this from an server/business/enterprise point of view, not as a home NAS.
I'm aware of ticket #1399 (http://support.freenas.org/ticket/1399), but before I start adding my two cents about the lack of robustness in my specific cases, I want to get an idea of how the developers think it should work, so I don't mistake a "feature" for a bug. ;^)
Replication means different things to different people, and I haven't seen a design document that spells out the desired outcome of successful replication, and especially how different error scenarios should be handled. For example, ZFS datasets can be destroyed intentionally (whether maliciously or not) or by accident. Is the goal of the replication system to seamlessly send these dataset changes to the target system, or should it throw up a warning and fail? Both are arguably correct actions.
In other words, is the replication intended to be "back up?" In my world, replication and back up are two related but ultimately different operations. I see replication as a way to quickly get running again when a whole system goes down. It's not intended to protect against selective accidental or malicious data loss where only a portion of the data is harmed.
In my view, a separate back up system is better to protect against those contingencies. One option is file level rsync (where the source is a snapshot) and the destination is an independent ZFS file system with its own snapshot schedule. Another option is Bacula and the like. This guards against the destruction of entire ZFS datasets, which can/should? be propagated with replication (because the system doesn't know whether that 'zfs rollback' or 'zfs destroy' was accidental, intentional and benign, or intentional and malicious).
(I don't think ZVols can be rsynced, but that's a different concern and outside the scope of this discussion.)
Example problem: In testing, I have two more or less identical systems configured with replication. In the simple case, replication does work, as long as you don't make changes to the auto snapshots. Now if I rollback a snapshot on the source machine, this breaks replication because the source and destination no longer match up. In my case, it seemed like rolling back the replication target to the same snapshot did not allow replication to sync up and proceed again. Theoretically this should have fixed the problem, but I think the handling of recursive snapshots is very broken for child datasets, which is what I was working with.
Rolling back a snapshot is quite a common action, as this is one of the features that makes ZFS so useful. So the question becomes, should something like a rollback (or dataset destruction) cause the replication to throw up errors and fail or should it transparently send those changes to the replication target?
Requiring manual intervention for every rollback or 'zfs destroy' is going to be quite annoying, especially when the majority of these actions are intended by the operator, and the chance that the change was accidental/malicious is tiny in comparison.
[EDIT: Seems like the replication does handle data set destruction properly (at least in the simple cases), and sends it straight through to the replication target as expected. The problems I ran into that made me think otherwise were probably due to the snapshots not being fully recursive on the source side. There's at least one ticket open in Trac regarding this.]
In conclusion, I'd like to see a clear design statement that spells out how the replication is supposed to work and how it's supposed to act under different circumstances. If nothing else, it will make it easier for us determine whether what we're seeing is a bug.
[EDIT: So after even more testing of simple cases it seems the biggest replication issue has to do with the snapshots not matching up on the two sides. What complicates matters a little is that snapshots which should be recursive on the source side sometimes are not. This seems to happen more frequently when attempting to snapshot at intervals of under 1 hour. Sticking to 1hr or longer intervals seems to make replication much more stable.]
On a related note, the Replication Tasks section for the 8.2 User Guide (http://doc.freenas.org/index.php/Replication_Tasks) states ". . . THERE IS A REPLICATION BUG IN BETA2 . . ."
Does anyone know specifically if there's a ticket that describes this bug? Obviously replication is very important to me and I want to correctly document the problems I'm seeing, but I don't want to create redundant tickets. That line is really unclear.
[EDIT: Answering my own question: Looking through some more tickets, it seems like this probably refers to the way the replication target host key is not being stored properly, requiring the manual addition to root's .ssh/known_hosts file. This wasn't one of the problems I'm concerned about because the work-around is simple.]
I'm aware of ticket #1399 (http://support.freenas.org/ticket/1399), but before I start adding my two cents about the lack of robustness in my specific cases, I want to get an idea of how the developers think it should work, so I don't mistake a "feature" for a bug. ;^)
Replication means different things to different people, and I haven't seen a design document that spells out the desired outcome of successful replication, and especially how different error scenarios should be handled. For example, ZFS datasets can be destroyed intentionally (whether maliciously or not) or by accident. Is the goal of the replication system to seamlessly send these dataset changes to the target system, or should it throw up a warning and fail? Both are arguably correct actions.
In other words, is the replication intended to be "back up?" In my world, replication and back up are two related but ultimately different operations. I see replication as a way to quickly get running again when a whole system goes down. It's not intended to protect against selective accidental or malicious data loss where only a portion of the data is harmed.
In my view, a separate back up system is better to protect against those contingencies. One option is file level rsync (where the source is a snapshot) and the destination is an independent ZFS file system with its own snapshot schedule. Another option is Bacula and the like. This guards against the destruction of entire ZFS datasets, which can/should? be propagated with replication (because the system doesn't know whether that 'zfs rollback' or 'zfs destroy' was accidental, intentional and benign, or intentional and malicious).
(I don't think ZVols can be rsynced, but that's a different concern and outside the scope of this discussion.)
Example problem: In testing, I have two more or less identical systems configured with replication. In the simple case, replication does work, as long as you don't make changes to the auto snapshots. Now if I rollback a snapshot on the source machine, this breaks replication because the source and destination no longer match up. In my case, it seemed like rolling back the replication target to the same snapshot did not allow replication to sync up and proceed again. Theoretically this should have fixed the problem, but I think the handling of recursive snapshots is very broken for child datasets, which is what I was working with.
Rolling back a snapshot is quite a common action, as this is one of the features that makes ZFS so useful. So the question becomes, should something like a rollback (or dataset destruction) cause the replication to throw up errors and fail or should it transparently send those changes to the replication target?
Requiring manual intervention for every rollback or 'zfs destroy' is going to be quite annoying, especially when the majority of these actions are intended by the operator, and the chance that the change was accidental/malicious is tiny in comparison.
[EDIT: Seems like the replication does handle data set destruction properly (at least in the simple cases), and sends it straight through to the replication target as expected. The problems I ran into that made me think otherwise were probably due to the snapshots not being fully recursive on the source side. There's at least one ticket open in Trac regarding this.]
In conclusion, I'd like to see a clear design statement that spells out how the replication is supposed to work and how it's supposed to act under different circumstances. If nothing else, it will make it easier for us determine whether what we're seeing is a bug.
[EDIT: So after even more testing of simple cases it seems the biggest replication issue has to do with the snapshots not matching up on the two sides. What complicates matters a little is that snapshots which should be recursive on the source side sometimes are not. This seems to happen more frequently when attempting to snapshot at intervals of under 1 hour. Sticking to 1hr or longer intervals seems to make replication much more stable.]
On a related note, the Replication Tasks section for the 8.2 User Guide (http://doc.freenas.org/index.php/Replication_Tasks) states ". . . THERE IS A REPLICATION BUG IN BETA2 . . ."
Does anyone know specifically if there's a ticket that describes this bug? Obviously replication is very important to me and I want to correctly document the problems I'm seeing, but I don't want to create redundant tickets. That line is really unclear.
[EDIT: Answering my own question: Looking through some more tickets, it seems like this probably refers to the way the replication target host key is not being stored properly, requiring the manual addition to root's .ssh/known_hosts file. This wasn't one of the problems I'm concerned about because the work-around is simple.]
