Need to one-way sync (mirror) one folder hierarhcy to another on different datasets, triggered by source change (not polling)

[Koguni]

Need to continuously monitor and one-way sync (mirrored) one folder hierarchy to another on different datasets.

Want it to be done by custom script, triggered by source change (added, removed or changed files and directories), but not the periodic polling.

Found no inotify_wait command line tool, no inotify python3 library installed, also no pip installed.

I know the "Do not change base system" rule, but how could I get the thing done?

 

[ChrisRJ]

I know that polling has a bad reputation. But firstly, the impact is in my experience often much smaller than most people think. In addition, I have seen a number of push implementations that under the covers still use a polling mechanism. They just present it the higher layers as a push.

Of course, this is a general statement. Without more information about the specifics of your data, change rate, etc. it is difficult to have a propert verdict.

 

[Patrick M. Hausen]

There is no inotify on FreeBSD. Possibly some equivalent based on kqueue. You will probably have to search a bit.

 

[Koguni]

There is no inotify on FreeBSD. Possibly some equivalent based on kqueue. You will probably have to search a bit.

Base FreeBSD 13.x distribution have such a tools, but they're missing from TrueNAS distribution:

fanotifywait(1)

www.freebsd.org

Seems that inotify and fanotify were added to kernel recently.

 

[Koguni]

I know that polling has a bad reputation. But firstly, the impact is in my experience often much smaller than most people think. In addition, I have seen a number of push implementations that under the covers still use a polling mechanism. They just present it the higher layers as a push.

Of course, this is a general statement. Without more information about the specifics of your data, change rate, etc. it is difficult to have a propert verdict.

Polling adds dilemma between replication delay and datasets rescan overhead.

I've added task to cron which starts rsync time to time, but I want to start sync immediately on change while avoiding unnecessary load.

 

[sretalla]

How about a replication task based on a very frequent snapshot task with the setting to allow empty snapshots turned off...? Load should be minimal as it's just checking the txg number on the dataset.

You would just need to keep the snapshot retention under control to not have millions of snapshots.