Issues with dataset encryption and replication between Ubuntu and TrueNAS systems

[Kebba]

Hello

I am having trouble with my setup. I think I might be doing something unusual but not extraordinary. But I am running into issues and I can't find what I need by searching.

In short, I am trying to backup datasets from my Ubuntu Server machine to my TrueNAS system, and from this replicate to a remote backup on another TrueNAS system. This has worked, but I am running into trouble when I start to use encrypted datasets. I will describe how my replication job are made now at the end of the post. It might be a false start anyway

This is my setup, including naming conventions to be used in the rest of the post:

Machine #1
Name: Blackbox
OS: Ubuntu 20.04
ZFS pool: zfs-utils 0.8.3
User: Dedicated user "freenas" that is allowed to zfs send

Show : Blackbox HW

MB: SuperMicro X10SLM+-LN4F
CPU: Intel Xeon E3-1220 v3
RAM: 2*8GB
Pool: 1 pool of 6 2TB disks in mirrors[

Machine #2
Name: Home_Nas
OS: TrueNAS 12.0-U1
User: root

Show : Home_NAS HW

CPU/Mobo:Asrock C2550D4I
RAM:32GB
Pool: RaidZ2 6*3TB

Machine #3
Name: Remote_NAS
OS: TrueNAS 12.0-U1
Users: Dedicated user "Kebba_replication". zfs permissions on MainRaidZ/Kebba_Replication: create,destroy,diff,mount,readonly,receive,release,send,userprop

Show : Remote_NAS HW

Not sure fo the moment. Running ontop of Proxmox on a machine with an 8-thread Xeon and 32GB or RAM

What I want to achieve:

Blackbox is going to take over the heavy lifting at home. I therefore want to have a lot of data be created on this filesystem. I have snapshots being managed by sanoid on this machine. Backups shall be replicated (pulled) to Home_NAS and selected datasets will be replicated to Remote_NAS (pushed). I want to use encrypted datasets for most datasets. Remote_NAS should not have keys to decrypt the backups (preferably). Same goes for Home_NAS if possible.

Visualization:

Blackbox -> (Pulled) Home_NAS -> (Pushed) Remote_NAS.

What works now?
I can pull snapshots from blackbox into a dedicated dataset vol1/Backup/blackbox-backup/blackbox-lockbox on Home_NAS. This works recursively with child datasets as well. I have also confirmed that I can restore these to blackbox. A restore require that I push the snapshots using CLI on Home_NAS

Code:

zfs send -vR vol1/Backup/blackbox-backup/blackbox-lockbox/example_dataset@latest_snapshot --raw | ssh freenas@blackbox zfs receive tank/temp/testrestore

I get a lot of permission errors when replicating (the user freenas is not allowed to set dataset properties and can't mount filesystems) but the snapshot get pushed anyway. I can then log into blackbox, set the correct properties, load the keys and mount the datasets where they should be.

So what do I need help with?
Issue #1
It is harder than it should to replicate Blackbox -> Home_NAS. Datasets must live under an encrypted parent /tank/Backup/blackbox-lockbox or the replication job fails with a "Unable to send encrypted dataset to existing unencrypted or unrelated dataset" error message. Creating an encrypted root does solve this problem. I am pulling the dataset with the "Full filesystem replication" option enabled. I am not sure if this is an advisable way to do it. It works, but is maybe not as elegant as I wish and I am slightly worried I am shooting myself in the foot by doing this. It is possible to achieve this without a root dataset if both target and source are running TrueNAS 12. This issue is not high priority for me if the "shoot myself in the foot" part is just paranoia.

Issue #2
I can't push the backed up datasets pulled from blackbox to Remote_NAS from Home_NAS. It will fail with a few different error messages depending on what I try. Examples:

"Unable to send encrypted dataset to existing unencrypted or unrelated dataset"

"dataset key must be loaded"

And a combination of different permission errors. (With "Full filesystem replication" enabled)


Are there any known solution to what I want to achieve or are there showstoppers that make it so I must reconsider some parts of the concept?

Thank you for your time

Example of pull job from Blackbox to Home_NAS (this works)

Show : Working pull from blackbox to Home_NAS

"Direction" - Pull
"Transport" - SSH
"Source" - tank/source_dataset_example
Full filesystem replication: Yes
Naming Schema: autosnap_%Y-%m-%d_%H:%M:%S_hourly/daily/monthly

Destination" - vol1/Backup/blackbox-backup/blackbox_lockbox