Non-root replications failing after upgrade to TruNas

[keboose]

I upgraded my main NAS and a remote machine both from FreeNas 11.3 to TrueNas 12.0. After the upgrade, the replication (Push) from the remote machine to my NAS fails every time:

* Replication "Main Pool Replication" failed: cannot mount 'BigNas/remotebak/backup/data': Insufficient privileges
cannot mount 'BigNas/remotebak/backup/users': Insufficient privileges..

Where BigNas/remotebak/backup is the destination for the replication task on the remote machine. So the remote NAS's pool looks like:

Code:

Tank
    ↳ data
    ↳ users
    ↳ VM_ZVol

The remote NAS uses a dedicated user I created on my main NAS, with UID 1007. After the replication began failing, I tried these things:

• manually unmount the BigNas/remotebak/backup dataset via root. This seems to work the FIRST replication, but it continues to fail after the first.
• make UID 1007 the owner of the dataset mount point AND owner of the folders once they are mounted. (same effect as setting the owner permissions for the dataset in the web GUI, was all owned by root:wheel before upgrade.) This did not solve the problem. After the single successful replication, all the mount points are owned by root:wheel again. What's up with that?
• Change the owner of the Tank and all child datasets on the remote NAS from root to a local user with UID 1007, to match my main NAS. Also had no effect.

Note that the ZVol dataset for the VM is on a separate replication task, which completes successfully every time.

I'm not experienced by any means with ZFS or replication, is there something I'm messing up? I know that replication not using the root account on both ends is kind of a permissions nightmare, but I'm not sure what else I can change to make this work

 

[keboose]

As an update, this still fails. I even went so far as to delete the BigNas/remotebak/backup/data dataset and let the replication rebuild it, but the same problem remains: cannot mount, Insufficient Privileges.

The situation has changed slightly, it seems that the replication sometimes completes successfully when I start the replication MANUALLY: If I log into the GUI, and hit the start button on the failed replication task, sometimes it will start and complete, other times it will fail. This only started happening in the last few hours after re-building. I may be able to narrow down the issue once I get more data points.

 

[keboose]

Okay, I think the manual replication was a fluke; the behavior is back to what it was: insufficient privileges error on source, mount points owned by root on destination. It seems that it is the START of the replication process on the source that resets the mount point owner, as I will reset the owner back to my normal user account on the destination NAS, then run the replication on the remote NAS, then it will immediately fail, and the permissions will reset. What's interesting is the SECOND time I reset permissions and run the replication, it works! (Then back to failing on the automatic scheduled replications.)

I think the problem is that I don't know what is happening behind the scenes during a replication that could cause the permissions to reset. What part of the replication process requires permission resets like this? I want to reiterate that all the datasets (and indeed the entire pool) on the source (remote NAS) are NOT OWNED BY ROOT, so this should not be a permissions issue from that perspective.