SOLVED New Encryption broke replication

[Sprint]

Hi All

So allow me to explain my setup, and what's changed.

I have a Primary NAS at home and a secondary offsite. I replication my datasets snapshots to the offsite each night and this has been working great for some time. The two pools in question (local and remote) were both encrypted using FreeNAS, so now show under TrueNas as "Legacy Encryption". I decided I wanted to beef up my encryption, (also because in my lab, I discovered TrueNAS Scale can't import pools with Legacy encryption, and this is something I want to be able todo down the line). I moved all the data off my Legacy Encrypted pool (to another pool which will get the same treatment later), tore the pool down, and rebuilt it using the new encryption methods. I then copied back the dataset, re-abled the Replication job to the offsite NAS, and left it to run.

This morning, I woke up to the following alert. Replication "Taskname" Failed: cannot receive incremental stream: cannot perform raw receive on top of existing unencrypted dataset.

What's confused me is that the remote side is encrypted, but has this broken because the encryption technology is now totally different? I should have anticipated this i guess? Is there a way to fix this without having to start from scratch? It would take 3 months to resync all the data if I have to rebuild the pool at the other end, plus would leave me exposed if a disaster were to happen. While the remote side is only 60 miles away, and I would just normally go and retrieve it, rebuild the pool, resync it all locally over 10Gb before re-deploying it, with the UK in lockdown at the moment, that's not an option, so I'm looking for alternative options

Both machines are running TrueNas 12.0.U1 (latest).
Primary Nas: VM running in ESX, with 8 cores assigned, 128Gb Ram assigned, 3xHBAs, ASRock 4x m2 PCIE 16x card with Optane drive attached being used as SLOG/L2ARC.
Primary Pool (unencrypted) 8x8Tb WD Gold + Optane Slog/L2ARC
Secondary Pool (Newly encrypted and discussed above) 8x8Tb WD Reds
other SSD pools that aren't relevant

Remote side
TrueNas 12.0.U1 running on bare metal
10Core Xeon, 256Gb ram
1x HBA
8x8Tb WD Reds
200Mbps VPN back to primary site.


Thanks all in advanced
Rich

 

[winnielinnie]

The issue is that GELI (legacy encryption) does not apply encryption at the ZFS/dataset level. For all intents and purposes, once the underlying devices are unlocked, ZFS views the datasets as non-encrypted datasets.

You won't be able to send a raw encrypted stream to the destination like this. As it stands now, you can have either plain ZFS datasets residing on top of lower-level GELI encrypted devices, or native ZFS encrypted datasets residing on top of standard non-encrypted devices. But not both together.

 

[Sprint]

The issue is that GELI (legacy encryption) does not apply encryption at the ZFS/dataset level. For all intents and purposes, once the underlying devices are unlocked, ZFS views the datasets as non-encrypted datasets.

You won't be able to send a raw encrypted stream to the destination like this. As it stands now, you can have either plain ZFS datasets residing on top of lower-level GELI encrypted devices, or native ZFS encrypted datasets residing on top of standard non-encrypted devices. But not both together.

So as I feared, the fact that TrueNAS doesn't "see" it as encrypted at the other end, (because the GELI method is done at a lower level (if I've understood that correct)), understandably, it refuses todo it... which makes sense I guess...

That'll teach me for not testing that element of it in my lab first :) Looks like I'll be rebuilding the remote pool and re-replicating the datasets over.

Thanks for the clarification :)

Sprint

 

[winnielinnie]

So as I feared, the fact that TrueNAS doesn't "see" it as encrypted at the other end, (because the GELI method is done at a lower level

That's exactly correct!

Yeah it's a weird time to transition encryption methods, especially in regards to large amounts of data.

Are you replicating only specific datasets, or pretty much everything in the pool, from local to remote?

I haven't done this myself, but I believe it is possible to decrypt a stream on-the-fly, and have it recv'd and saved non-ecrypted on the remote server. In your case "non-encrypted" means "non-encrypted, but on top of lower-level GELI encryption."

 

[Sprint]

Decided to move this post to a new thread, as its a different question to whats above. Thread located here https://www.truenas.com/community/threads/cant-zfs-send-between-encrypted-pools.90532/

Thanks