Issue with multiple replication tasks for the same data

[hexlabsnick]

Hi,

A bit of a TrueNAS noob but am trying to learn quickly.

I have two 112 TB RAID z2 pools in one system. One pool is for production data and id like the other one to be for backup (only until my 2nd system arrives, then the backup pool will move to that system).

Right now I have the following replication/snapshot tasks setup to be stored on the production pool,

Hourly snapshot, kept for 1 week
Daily snapshot, kept for 15 days
Weekly snapshot, kept for 4 weeks
Monthly snapshot, kept for 2 months

These snapshots are running without issue.

They were actually setup by the system as part of the replication tasks I created. I would like to replicate all of these snapshots to the backup pool, while also extending the retention out a bit on the backup pool. The ideal retention for these snapshots on the backup pool is,

Hourly snapshot, kept for 1 week
Daily snapshot, kept for 30 days
Weekly snapshot, kept for 8 weeks
Monthly snapshot, kept for 6 months

I started this config out with only the daily snapshot/replication and in order to get it to work I had to check "Synchronize Destination Snapshots with Source" because there were some old snapshots on the backup pool. When I did this it removed everything from the old pool and started fresh. This check box has since been unchecked and this daily snapshot/replication task continues to run without issue.

I wanted to try to start up the replication task for the hourly snapshot, but every time it runs I get this error,

Error
No incremental base on dataset 'prodpool1' and replication from scratch is not allowed.
Logs
[2021/06/02 23:00:00] ERROR [replication_task__task_2] [zettarepl.replication.run] For task 'task_2' non-recoverable replication error NoIncrementalBaseReplicationError("No incremental base on dataset 'prodpool1' and replication from scratch is not allowed")

I don't want to select the "Synchronize Destination Snapshots with Source" option again because I think that will remove the daily snapshots taking me back to step 0.

What am I missing here? I would like to be able to have these four tasks all running at the same time doing their own thing, does that mean I have to have 4 full copies of the data on the backup pool? Does all of this need to be combined into one replication task somehow?

Here is what the replication task for the hourly job looks like,

Here is what the replication task for the daily job looks like,

'

I am sure I am just doing something wrong, but all of my googling hasn't helped me figure out the issue yet, so hopefully someone on here can set me in the right direction.

Let me know if you guys need more info.

Thanks,
Nick

 

[sretalla]

If all of your replication is for the same dataset(s), then you only need to replicate on the most frequent schedule of snapshots (all the others in between will be picked up in the incrementals and will appear on the other side).

 

[hexlabsnick]

If all of your replication is for the same dataset(s), then you only need to replicate on the most frequent schedule of snapshots (all the others in between will be picked up in the incrementals and will appear on the other side).

Hi,

Thanks for the info.

Wouldn't that mean there is no way to have snapshots kept for different schedules on the backup server? My hopeful goal was to have snapshots kept for a specific length on the primary server and then for a longer length on the backup server.

Thanks!

 

[Basil Hendroff]

does that mean I have to have 4 full copies of the data on the backup pool?

No. The snapshots are copied.

Does all of this need to be combined into one replication task somehow?

Yes. Edit the replication task. Click on the Periodic Snapshot Tasks dropdown and select each snapshot task that applies.

 

[Stick]

What did you end up doing here? I'm in the exact same boat, and as I understand it, the initial question / concern was never answered.

Example:
I have a production box with a dataset where I'm taking the following snapshots:
15min interval - 2 day retention
1 hour interval - 1 week retention
1 day interval - 2 month retention

I am replicating to a backup box, where I would like to keep the retention for the 15min and 1hour snapshots, but retain the daily snapshots for a year. I thought this could be accomplished by creating two replication tasks, one that contains both the 15min and 1hr snapshots with retention same as source, and a second replication task that contains the once per day snapshots and specifies a retention of 1 year. However, when I do I get the same error as the OP because each replication task seems to want its own dataset on the destination box.

- If I put all the snapshots in a single replication task, I can't specify a different retention policy for some snapshots, which is really what I'm after.
- If I hold ALL the snapshots for a year, I have WAY too many tiny snapshots accumulated, in this example a year's worth of 15min interval (no thanks).
- If I use two different replication tasks, I seemingly have no choice but to duplicate the dataset on the destination box.

Like the OP, I feel I must be missing something here, as I would think this would be a common setup for someone to want to have.