11.3 Pull replication - Naming schema requirement issues / limits

[SMnasMAN]

TLDR: with the new PULL replication task in fn 11.3 ,im restricted to only backing up one "type or set" of snapshots, because of restrictions related to the requirement of defining a snapshot Naming Schema to pull only snaps that match , not all snaps from xyz dataset on remote FN source.

I have 2x physical Freenas boxes. lets call them:
BackupFN (has the PULL replication task)
and
SourceFN (has the datasets and snapshots i want to PULL replicate into "BackupFN")

(im using pull replication for security reasons vs using Push replication as i had been previously)

When setting up a Pull Replication task, you must define a Naming Schema (FN 11.3 u3.2 docs link, and see image below)


On the SourceFN side, like most users, i have a few Periodic Snapshot tasks setup for "Dataset A" such that it auto creates snapshots with names like this:
auto-20200628.0300-1m
auto-20200715.0100-1d
auto-20200722.1800-4w
(and so on, just a few examples , see image below).


The problem im having is that because PULL type replication requires defining a snapshot Naming Schema (and no wildcards apparently) , im only able to backup one "type or set" of snapshots,
ie ONLY this works:
auto-%Y%m%d.%H%M-4w
(which if set will ONLY backup the -4w snapshots , not the -1m or -1d snaps).

ideally what i wish would work is auto-%Y%m%d.%H%M-* (or .* ) but this does not work
or even auto-*

I need Pull replication to backup all snapshots of a given dataset (just like push replication does).
(hopefully im missing something, as i would need to make around 15x Pull Replication task, one for each Periodic snapshot tasks on the SourceFN side, and going forward if i make any new Periodic snapshot tasks, i would have to go back to BackupFN and create a new / corresponding Pull Replication task)

Does anyone have any solution to this? or how are you all using Pull Replication?
thanks!

edit; this image below is the crux of the issue, if you see "0 snapshots found" nothing will happen when you run the Task (understandably). But also apparently the Recursive checkbox does nothing (when using PULL) and the only way to get "N Snapshots found" is to exactly define one set of snapshots, ie see these examples:

 

[SMnasMAN]

well, was somewhat hopeful when i found this bug report, (user says he is able to enter multiple naming schema's, separated by a space, but then gets an error at task creation):

https://jira.ixsystems.com/browse/NAS-106767

I tested it, and while i get the same bug/error (cant add replication task), the gui does make it appear as if you can enter multiple naming schema, like this: auto-%Y%m%d.%H%M-16h auto-%Y%m%d.%H%M-2w (note the space as a separator/delimiter)
(which files 30x snaps to replicate, whereas each of them separately only find 16x and 14x snaps)

Im assuming im missing something here, because it does not make sense that only PULL replication requires you to specify a naming schema for the snaps to PULL / replicate in (rather, it should pull in ALL snaps of a pool/dataset, but as an option you can choose to restrict it by adding a naming schema mask, if you would like to restrict it). clearly the gui/middleware is able to list all the snaps of a given pool/dataset, so why not PULL replicate all of them in?
tks

 

[SMnasMAN]

am i one of the few using the new PULL replication tasks (as opposed to the standard Push type) ?

 

[styno]

am i one of the few using the new PULL replication tasks (as opposed to the standard Push type) ?

I feel you, I was struggling with this myself and was planning to solve this with a -custom snapshot schedule & naming convention.
In the end I basically gave up on it, stopped worrying about it and created a -hourly and replicated all of the hourly snapshots with a longer retention period on the backup device.

 

[SMnasMAN]

I feel you, I was struggling with this myself and was planning to solve this with a -custom snapshot schedule & naming convention.
In the end I basically gave up on it, stopped worrying about it and created a -hourly and replicated all of the hourly snapshots with a longer retention period on the backup device.

hey styno, i did the exact same thing as well! (to a T !) , im still wondering why the "naming schema" field is required and not an option. (as the default should be to pull replicate all snapshots of a given dataset or pool recursively. This is how it was with push replication tasks pre-11.3).

However, this does seem to be working wo error for the past week or two ive had it running- although the setup does take quite some time (and is a bit complicated) as ive ended up with 9x Pull replication tasks as opposed to 1 or 2x recursive ones.

One other interesting bit of info, I also did a 2nd test setup with pull replication running on FN 11.3 u4 , and the source as a FN 11.2 u6 box- this also works! which is pretty neat as i didnt plan on upgrading my large, main FN box from 11.2 u6 to 11.3 u4 , just yet. (i still want to see 11.3 proven a bit more before putting it into live production).

(the main boxes im speaking about in my opening post, are both 11.3u4 boxes, btw)
thanks

 

[MrNaz]

Wow. Very surprised to find that this is the case. How is it at all sensible to have to PULL snapshots by name? That seems like it would be the exception, not the rule. If I want to replicate site A to site B, I want to be able to tell site B to just pull everything. Only baking up some named subset of snapshots doesn't strike me as the most common case at all, so even if it was changeable, it would make an odd default. That it's the ONLY option blows my mind.

 

[Darkhog]

I'm having the same issue. I don't understand why it doesn't allow you to query the system you are pulling from to even see the snapshots that exist. Is there a way to query these snapshots so you can see the pattern? I'm thinking this is what would be used if I am replicating offsite, then I want to do a restore - I would do a PULL replication from the offsite backup, correct?