Restored pool from TrueNAS 12.0-U4.1 fails to decrypt

[MNeverOff]

Hey all,

After moving I'm now reassembling my NAS (Build first in July 2021 using TrueNAS Core) in a VM encironment. I have successfully imported it but only 2/3 drives are ok (third got corrupted, I replaced and resilvered it successfully). However, after successful import it prompts me to decrypt it. Name of the pool is Backup.

I have the source files stored away securely: dataset_Backup_keys.json that contains a 64 char key, I see the timestamps and the fact that they were created the same time as the pool. However, whatever I try, on True NAS 13.0-U3.1, I can't decrypt it using that key. Uploading file, doing the key entry, doing the CLI `zfs mount -l Backup` all result in Incorrect Key Provided / Provided key is invalid error.

I'm heartbroken as it's my only hope to restore some very old photos (cloud backup was deleted and the local copy suffered from an NTFS partition table corruption so 1-2-3 got kinda ruined). I have since tried to resilver and repair the pool (it was degraded but is now stable and still not working) and use a ZFS 12.0-U4.1 to decrypt - all to no avail.

My current running theory, after reviewing the key files, is that I have encrypted a System Dataset Pool and that somehow overriden the encryption on the Backup pool? I am now trying to restore the initial VM that I have had before with VMWare (current one is Hyper-V). Do I have any hope there and is that's something that TrueNAS would do?
Is there any way for me to use the System Dataset Pool to then uncover the Backup Pool Dataset Encryption keys?

Thank you and apologies if any details are amiss. I have reviewed other posts on this form but most of them deal with geli encryption, whereas this is ZFS encryption.

 

[artlessknave]

My current running theory, after reviewing the key files, is that I have encrypted a System Dataset Pool and that somehow overriden the encryption on the Backup pool?
I do not believe you can encrypt the system data-set on it's own, but it does sound like you managed to change the encryption settings without saving the new key, or saved the wrong key to begin with.

unfortunately, if you don't have the decryption key, you essentially have a bunch of random bits; if you have no backups, then you have no data.

you should generally only use encryption if you actually need it.

if anything, you should get rid of the virtualization and see if you can import the pool into a baremetal install of TrueNAS, however, depending on how you attached the disks in hyperV, this might have mixed results. if you were doing any janky disk passthrough or windows obfuscation of the disks, I wouldnt expect it to import at all, so probably not helpful.

as you have given none of the required post information, I can't think of anything else.

 

[MNeverOff]

Fair's fair, here's a full breakdown:

• Motherboard: AsRock X570 Taichi
• AMD 5800X3D
• 64GB G.SKILL 3600-CL16 DDR4 RAM
• Boot drive (host) - 970 Evo M.2 1TB, HDDs - Seagate IronWolf ST4000VN008-2DR166 4TB x 4, connected over an ACHI controller to the motherboard
• Boot drive (first client VM) - virtual VMWare drive on a host boot drive, 3xIronWolfs mounted over iSCSI and formed up in a pool. 16Gb virtual RAM, 8 virtual processors. Three pools: Apps pool (virtual disk on boot drive), TrueNAS pool (system dataset) and Backup pool (3x4tb drives in the zfs version of raid5) - all three encrypted (I see the key files in my storage)
• Boot drive (second VM, recreating) - virtual Hyper-V drive on a boot drive, 3xIronWolfs mounted as Physical Drives (assuming iSCSI too). 16GB virtual RAM, 8 virtual processors, two pools: boot-pool (default) and Backups in the same arrangement how before, but only Backups is encrypted
• NICs - all on-board (Intel® I211AT with a separate Intel® 802.11ax WiFi Module)

 

[MNeverOff]

I do not believe you can encrypt the system data-set on it's own, but it does sound like you managed to change the encryption settings without saving the new key, or saved the wrong key to begin with.

So, to address this - which TrueNAS operations would cause an encryption key to change? Because I can see, based on the creation date of the key files, that I first encrypted the Backup pool and then created and encrypted TrueNAS (presumably a System Dataset Pool) and an Apps pool, both of which were .vdmk virtual drives on VMWare with physical files on the host system boot drive.

Would encrypting a system datased pool do something like this? I never remember decrypting the Backup Pool during over a year of its operation, meaning there should've been some automated way of doing so, which leads me to believe that I've inadvertently set up some new encryption keys?

 

[jgreco]

After moving I'm now reassembling my NAS (Build first in July 2021 using TrueNAS Core) in a VM encironment. I have successfully imported it but only 2/3 drives are ok (third got corrupted, I replaced and resilvered it successfully). However, after successful import it prompts me to decrypt it. Name of the pool is Backup.

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your virtualization setup.

Because this includes a virtualization component, please also review the virtualization guidelines located at

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

and please explain any differences between your setup and the recommended method.

 

[artlessknave]

I never remember decrypting the Backup Pool during over a year of its operation

if you encrypted it with the key method, it would be automatically encrypted on startup. only decryption using a passcode needs to be unlocked manually. if you then didnt save that exact key, then the data would be unrecoverable.

do you have the old boot disk, by chance? because if you do, you might be able to put the pool back there to get the key exported

you should always backup your keys regularly, not just once on creation, for reasons like this. hopefully, once you answer jgreco's explanation request, he might have some idea's.