Replication Task "Full Filesystem Replication" does not send intermediate snapshots?

[winnielinnie]

Up until I upgraded to TrueNAS 12.0-U3, whenever I ran a manual Replication Task with the option "Full Filesystem Replication", everything would be sent to the destination, including intermediate snapshots since the last replication.

Apparently, using "Full Filesystem Replication" has always used -i instead of -I, which neglects intermediate snapshots.

I just noticed now when I ran a manual Replication Task (same options), no intermediate snapshots were transferred.



TABLE 1: Expected behavior.

SOURCE	DESTINATION (after replication task completes, as expected!)
manual-replication_2020-12-01 <-- base for "incremental"	manual-replication_2020-12-01
auto_2020-12-15	auto_2020-12-15
auto_2021-01-01	auto_2021-01-01
auto_2021-01-15	auto_2021-01-15
auto_2021-02-01	auto_2021-02-01
auto_2021-02-15	auto_2021-02-15
manual-replication_2020-03-01 <---------- replicate this	manual-replication_2020-03-01

TABLE 2: Actual result.

SOURCE	DESTINATION (after replication task completes, they're missing!)
manual-replication_2020-12-01 <-- base for "incremental"	manual-replication_2020-12-01
auto_2020-12-15
auto_2021-01-01
auto_2021-01-15
auto_2021-02-01
auto_2021-02-15
manual-replication_2020-03-01 <---------- replicate this	manual-replication_2020-03-01

 

[winnielinnie]

UPDATE: It appears I wasn't the only one who noticed, but apparently this has existed even before 12.0-U3.

https://jira.ixsystems.com/browse/NAS-109476

The bug reporter is correct in that a "Full Filesystem Replication" which claims...

"Completely replicate the selected dataset. The target dataset will have all of the properties, snapshots, child datasets, and clones from the source dataset."

...should in fact do as it claims. Unfortunately, for every subsequent run, the Replication Task (with the option enabled "Full Filesystem Replication") will skip intermediate snapshots.



When I used to write my own command-line script, I invoked -I (instead of -i) to make sure all snapshots (including intermediate ones) would be sent to the backup-pool.



I humbly request that others vote for this issue, as it affects the user's expectations of a "true" full backup from a live pool to a backup pool when using "Full Filesystem Replication".

 

[winnielinnie]

Has anyone else noticed this when using a Replication Task with the option "Full Filesystem Replication"? Isn't your expectation by selecting this option that it will do a true full filesystem backup?

When you read a tooltip that claims "Completely replicate the selected dataset. The target dataset will have all of the properties, snapshots, child datasets, and clones from the source dataset." Don't you expect it to backup all your child datasets, snapshots, and clones?

According to the bug report, this can only be fixed in SCALE, but not in Core. So I'm scratching my head even more confused, since I was able to emulate the expected behavior ("Full Filesystem Replication") by using a shell script.

If you look at the tables above, Table 1 is what a user expects when running such backups, yet if they do not double-check their backup pool, they might be unaware that in reality they're getting Table 2's results.

 

[petervg]

hmm... one year later and I see exactly the same issue... checked the box "(almost) full replication", added naming sceme "auto-%Y-%m-%d_%H-%M" and checked all periodic replication tasks, but almost none of the automated snapshots are replicated.

Don't wanna throw any stones, but after a bad experience with a severe bug in the bhyve hypervisor that IX didn't want to fix and this issue on top, I'm starting to question a bit as to why truenas is considered to be such a great and reliable tool... I mean.. a simple "full backup" should not be that hard to do, right?

 

[petervg]

Filed a bug report NAS-116045

 

[urza]

hmm... one year later and I see exactly the same issue... checked the box "(almost) full replication", added naming sceme "auto-%Y-%m-%d_%H-%M" and checked all periodic replication tasks, but almost none of the automated snapshots are replicated.

Don't wanna throw any stones, but after a bad experience with a severe bug in the bhyve hypervisor that IX didn't want to fix and this issue on top, I'm starting to question a bit as to why truenas is considered to be such a great and reliable tool... I mean.. a simple "full backup" should not be that hard to do, right?

You are not alone feeling like this. The feedback between users and developers feels to be badly broken.

I tried to accomplish the same thing - daily backups from mainpool to backuppool on the same machine on truenas scale. I am unable to get it work, such a basic and critical functionality for a NAS system !!! and all issues here are at least a year old and tickets are without much activity... makes me question the choice of TrueNAS Scale.

 

[petervg]

Well, the 5th of may I filed a bugreport in Jira, but uo until today the status is "Unresolved, moved to backlog". If it wasn't for ZFS implemented in Truenas, I would have been on the lookout for a better system...

 

[winnielinnie]

To be honest, and based on the older bug report, this is likely never going to be changed for TrueNAS Core (any version), and they might never get to it with SCALE.

Enterprise / paying customers come first, and if the tools are "good enough" for them in the meantime, the developers probably feel it's not worth the risk of trying to introduce a new feature and/or fix an existing behavior.

Here's a direct quote from the original bug report:

"[T]he limitation is in the amount of changes we had to make in the TrueNAS replication engine (zettarepl) in order to support replication of snapshots without parsing their names. We will not risk integrating so many changes into stable version of TrueNAS."

---

If it wasn't for ZFS implemented in Truenas, I would have been on the lookout for a better system...

What's interesting is that the ZFS utility named "syncoid" is a great CLI tool with sufficient options and flexibility, and makes for a quick way to do manual replications/backups with a single command.

It wouldn't be a far-stretch to build a GUI around syncoid for the sole purpose to address the use-case of simple manual full backups.

I would imagine there could be an entry in the GUI under "Tasks" (maybe right below "Replication Tasks"), which is named "Manual Backups". From here you simply configure a backup that can be invoked manually (with the press of a button) to backup a dataset(s) to a destination (doesn't matter if the destination is local, USB, remote server, etc.) It would run "syncoid" in the background. (This isn't much different than the assortment of user-friendly backup tools for Windows. You create a backup policy, plug in your USB drive or point it to an SMB share, and then click "Backup Now!")

Obviously there would be safety checks, such as "Is a backup currently running?" and "Is the destination available?", in which it would abort with a warning message.

What I wrote above is not just a pipe dream. I've successfully used syncoid with Linux boxes (Linux to Linux) to test this out. However, it was command-line only (no user-friendly GUI.) When I tried Linux to TrueNAS, my TrueNAS server crashed and reported an unhealthy pool. (Still not sure why.)