Recovering accidentally removed truenas config and keys

[mangof60]

Normally, if you go through multiple updates/upgrades, then you will get more selection amongst your list of Boot environment.
However, seeing what you have, Default would have been a good ooption, but I douobt it contains much data as it doesn't use a lot of space. You can try the "Default" and see if you can reboot and have the drives already imported.

Otherwise, I would try checking if you have any snapshot that could exist on your boot drive. (I might have been wrong about the creation of snapshots during update/upgrade.
However the Boot environment seem to be stored under "freenas-boot/ROOT" folder structure.
If you have snapshots, which you can list with:

Then you might be able to clone the most recent one to look for the keys and such.

Excuse my ignorance, I'm not sure where to run that zfs command. I sshed into root@truenas.local and I can't find any files or folders named freenas-boot

 

[Apollo]

Excuse my ignorance, I'm not sure where to run that zfs command. I sshed into root@truenas.local and I can't find any files or folders named freenas-boot

While within SSH CLI, you just need to run:

zpool list

What I get on my end is:

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
TANK 36.2T 24.8T 11.5T - - 43% 68% 1.00x ONLINE /mnt
freenas-boot 111G 6.39G 105G - - 4% 5% 1.00x ONLINE -

freenas-boot is the pool for the boot drive.

Maybe yours shows up as a different name, but I doubt.

 

[winnielinnie]

What I think happened is that sometime between 2022-12-14, I reconfigured the encryption and didn't copy the keyfile, and then all this time I thought the 'old' keyfile was my current one and didn't think to make another copy.

You might have screwed yourself.

Try restoring to the "Now/Reboot" environment. Then reboot. If that doesn't work, try the "default" environment and reboot.

Did you ever backup your config file at any point? (It will be in a .tar file, wherever you chose to save it.)

 

[joeschmuck]

Did you have a backup of your files? One thing TrueNAS / ZFS does well is encryption to keep the files safe from prying eyes. Unfortunately if you cannot recover the key, I'd say all your data is unrecoverable. I truly hope you do however recover the key.

-rw------- 1 www www 32 Jun 1 15:06 pwenc_secret

Did you reset the configuration today 2 Jun or the day before on 1 Jun? I'm just curious why the file date is this. My file date is back in October 2022, but then again, I do not use encryption. While the answer is not going to help you, it just appears to be a discrepancy I'm trying to understand.

 

[mangof60]

You might have screwed yourself.

Try restoring to the "Now/Reboot" environment. Then reboot. If that doesn't work, try the "default" environment and reboot.

Did you ever backup your config file at any point? (It will be in a .tar file, wherever you chose to save it.)

I did back it up, but the title is "truenas-20221214181628.tar" which is the same date as the boot env from my screenshot, and if my theory is right (that I changed the encryption later), then that environment is outdated for sure. I feel like I did screw myself

While within SSH CLI, you just need to run:

What I get on my end is:


freenas-boot is the pool for the boot drive.

Maybe yours shows up as a different name, but I doubt.

Here is my console output:

Code:

root@truenas[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool  39.5G  3.04G  36.5G        -         -     1%     7%  1.00x    ONLINE  -
nas        7.27T   735G  6.55T        -         -     3%     9%  1.00x    ONLINE  /mnt
root@truenas[~]# zfs list -t snapshot -r boot-pool
NAME                                           USED  AVAIL     REFER  MOUNTPOINT
boot-pool/.system/samba4@wbc-1685657469        152K      -      324K  -
boot-pool/.system/samba4@wbc-1685657474        152K      -      324K  -
boot-pool/.system/samba4@wbc-1685749278        136K      -      308K  -
boot-pool/ROOT/13.0-U3.1@2022-07-11-03:16:03  3.79M      -     1.50G  -
boot-pool/ROOT/13.0-U3.1@2022-12-14-16:12:05  3.99M      -     1.50G  -

The nas pool is what I'm trying to unlock, and boot-pool is the name of the boot pool. Maybe I have a different version? Version: TrueNAS-13.0-U3.1

Did you have a backup of your files? One thing TrueNAS / ZFS does well is encryption to keep the files safe from prying eyes. Unfortunately if you cannot recover the key, I'd say all your data is unrecoverable. I truly hope you do however recover the key.

Did you reset the configuration today 2 Jun or the day before on 1 Jun? I'm just curious why the file date is this. My file date is back in October 2022, but then again, I do not use encryption. While the answer is not going to help you, it just appears to be a discrepancy I'm trying to understand.

I reset it last night on June 1.


Btw, I do have freenas-v1.db and pwenc_secret from my 2022-12-14 config backup I mentioned above. Depending how the key is generated, could I just use that to regenerate the key?

 

[winnielinnie]

and if my theory is right (that I changed the encryption later)

If you did this, but never backuped up the keyfiles, then the data is gone forever. The other chance is a config file + secret seed, which you said you don't have any that are more recent then December 14 of last year (implying you changed the encryption key after that date.)

 

[winnielinnie]

Assuming you didn't use passphrase for the encryption key, but truly a keyfile / HEX string, these are your remaining options, in this order:

1. Find a backup of this new keyfile. If it doesn't exist, your likelihood of accessing your data takes a steep dive...
2. ...perhaps you copy-pasted the keystring somewhere (notes, LastPass, KeyPassXC, text file, etc)...
3. ...find a copy of the complete config DB + secret seed that was generated on a date after you changed the encryption key...
4. ...maybe you did switch to a passphrase instead? You can check with zfs get keyformat nameofdataset ...
5. ...resort to a backup of your datasets, and chalk this up as a painful lesson learned.

 

[winnielinnie]

Did you reset the configuration today 2 Jun or the day before on 1 Jun?

It could be a time-zone difference?

What I'm guessing happened is this:

Sometime after December 14, 2022, the encryption key was changed on the dataset. No backup was made of this keyfile.

During this time, as long as they do not export the pool, the data is always accessible and the "scrambled" form of the keystring exists within the freenas-v1.db database file, which is decoded using the complementary pwenc_secret upon every reboot of the server. (From a user's perspective, "everything is working fine". However, this is deceptive, since the moment the pool is exported and/or the config is factory reset, then the data is gone forever. This is because no backups of the new keyfile exists.)

Then around June 1, 2023, the config was factory reset.

And that's where the illusion of "everything is fine" shatters.


If the pool is exported, then attempting to re-import it will prompt the user to upload the keyfile. (But this keyfile doesn't exist! It was never backed up.) The same issue happens when the config is factory reset. Attempting to redo your configuration (which includes re-importing the pool) will bump into the same issue: "Please upload your keyfile to unlock your datasets."

 

[joeschmuck]

Tell you what, let's make sure there are no "save your butt" files on the system...
1. At the command prompt type cd / , only because I like to do this step.
2. At the command prompt type find / -name "*.db" | grep "configs" | more and note that if you do have a lot of access-able data, this could take a while to go through the files.
3. If you are really lucky some files will be displayed that you can use. If there are any, post the output for our viewing pleasure.

EDIT:
Also search for the pwenc_secret file as well using find / -name "pwenc_secret" however I suspect it will only show up the one time where you indicated before.

 

[mangof60]

@winnielinnie I think you're exactly right.
I remember setting the encryption in December, then I remember later messing with the encryption settings and I strongly recall setting a passphrase (which I know exactly what that would be) because I did the same on a Windows machine when I set bitlocker (and I was able to recover that with the passphrase after a drive failure), but I could be misremembering that I did that since truenas really wants a keyfile instead of a passphrase.

I also have two keyfiles saved, one which says "nas":"{key}" and one which says "nas-backup":"{key}", and I have a pool called nas with a dataset called backups. I remember encrypting only backups at first, then switching to encrypt the whole nas pool. This is what led me to believe I had the right key, because I thought that "nas":"{key}" was the key for the new encryption I set up.


I'm currently scrubbing every hard drive and flash drive I have to see if I have any keyfiles that I can try, but if we're right that I never backed it up then it won't matter anyway.

I still have a small hope that some data forensics can uncover the deleted .db and pwenc_secret files if they haven't been overwritten yet, but I'm not that optimistic for that. It would be a cool story if it works though.
I did learn for the future though, that I should back up the boot drive offsite (even if it wasn't my mistake, an equipment failure would have had the same result)

 

[mangof60]

Tell you what, let's make sure there are no "save your butt" files on the system...
1. At the command prompt type cd / , only because I like to do this step.
2. At the command prompt type find / -name "*.db" | grep "configs" | more and note that if you do have a lot of access-able data, this could take a while to go through the files.
3. If you are really lucky some files will be displayed that you can use. If there are any, post the output for our viewing pleasure.

I'll do this search and report back. I made a copy of the drive and searched for any sqlite files, but there was only one that matched the format of the config db so I believe that's just the new factory one.

 

[mangof60]

Tell you what, let's make sure there are no "save your butt" files on the system...
1. At the command prompt type cd / , only because I like to do this step.
2. At the command prompt type find / -name "*.db" | grep "configs" | more and note that if you do have a lot of access-able data, this could take a while to go through the files.
3. If you are really lucky some files will be displayed that you can use. If there are any, post the output for our viewing pleasure.

EDIT:
Also search for the pwenc_secret file as well using find / -name "pwenc_secret" however I suspect it will only show up the one time where you indicated before.

Code:

root@truenas[/]# find / -name "*.db" | grep "configs"
/var/db/system/configs-a6924d0eea7543099bc278a3576ed212/TrueNAS-13.0-U3.1/20230603.db

This is the only result I have, interesting that it's dated for today as well

Code:

root@truenas[/]# find / -name "pwenc_secret"
/data/pwenc_secret
root@truenas[/]# ll /data/pwenc_secret
-rw-------  1 www  www  uarch 32 Jun  1 15:06 /data/pwenc_secret

And here is the pwenc_secret


Btw, it appears my timezone is incorrect. I am in central time, but the dates are reported in PDT

Code:

root@truenas[/]# date
Sat Jun  3 11:16:41 PDT 2023

 

[joeschmuck]

That is unfortunate results but we did expect them. I was hoping to "pull a bird out of a hat".

As I said earlier, there is a link in my signature to Multi-Report script. This will automatically email you your config and key file if you set it up properly. I get an email every Monday with these files. You can also setup for the attachment to be sent encrypted as well. Just use a password you will be able to remember.

Cheers,
Joe

 

[mangof60]

That is unfortunate results but we did expect them. I was hoping to "pull a bird out of a hat".

As I said earlier, there is a link in my signature to Multi-Report script. This will automatically email you your config and key file if you set it up properly. I get an email every Monday with these files. You can also setup for the attachment to be sent encrypted as well. Just use a password you will be able to remember.

Cheers,
Joe

I'll set up that script on my new system. I'll probably take this opportunity to rebuild the whole system since I've wanted to for a while but I didn't want to take it down. Since the data is dead there's no point keeping it up now
Thanks for the help

 

[winnielinnie]

I also have two keyfiles saved, one which says "nas":"{key}" and one which says "nas-backup":"{key}", and I have a pool called nas with a dataset called backups. I remember encrypting only backups at first, then switching to encrypt the whole nas pool. This is what led me to believe I had the right key, because I thought that "nas":"{key}" was the key for the new encryption I set up.

What do you mean by this? You can't change a non-encrypted dataset to an encrypted dataset. Encryption is decided at the time of the dataset's creation. (Yes, even the "root dataset", which shares the same name as the pool.)

Do you mean that you created a new pool with the "encryption option enabled" (which essentially creates an encrypted root dataset with a randomly-generated keystring), and then later created a child dataset underneath that you chose to break inheritance to become its own "encryptionroot"? Then even later you chose to inherit (again?) its encryption properties? And then even after this you decided to change the encryption key of the root dataset?


To clear some things up, there is one other option:

Code:

zfs list -r -t filesystem -o name,encryption,encryptionroot,keyformat nas

You might (MIGHT) have a shot at bypassing the issue of being unable to unlock the root dataset, yet still access the child datasets underneath. It depends on how things currently live.

(Substitute "nas" for your actual pool's name.)


EDIT: Oh shoot. I might have replied too late? Did you destroy the pool yet?

 

[mangof60]

What do you mean by this? You can't change a non-encrypted dataset to an encrypted dataset. Encryption is decided at the time of the dataset's creation. (Yes, even the "root dataset", which shares the same name as the pool.)

I created nas pool with a couple datasets in it, I encrypted only the backups dataset. Then, I destroyed that pool and recreated it with encryption at the pool level so that nas/backups and nas/storage (and all other datasets) inherited the nas encryption. These are all the changes I remember making.
(nas is the name of the pool, I have a dataset under that called backups and another called storage)

zfs list -r -t filesystem -o name,encryption,encryptionroot,keyformat nas

Code:

root@truenas[~]# zfs list -r -t filesystem -o name,encryption,encryptionroot,keyformat nas
NAME                                                  ENCRYPTION   ENCROOT  KEYFORMAT
nas                                                   aes-256-gcm  nas      hex
nas/.system                                           aes-256-gcm  nas      hex
nas/.system/configs-ffda40d37689475cb5402d12a5cf7794  aes-256-gcm  nas      hex
nas/.system/cores                                     aes-256-gcm  nas      hex
nas/.system/rrd-ffda40d37689475cb5402d12a5cf7794      aes-256-gcm  nas      hex
nas/.system/samba4                                    aes-256-gcm  nas      hex
nas/.system/services                                  aes-256-gcm  nas      hex
nas/.system/syslog-ffda40d37689475cb5402d12a5cf7794   aes-256-gcm  nas      hex
nas/.system/webui                                     aes-256-gcm  nas      hex
nas/backups                                           aes-256-gcm  nas      hex
nas/media                                             aes-256-gcm  nas      hex
nas/storage                                           aes-256-gcm  nas      hex

I have not destroyed anything yet (on purpose) and because I'm crazy I plan to keep all of these drives in tact just in case quantum computing breaks encryption in my lifetime



Please excuse my ignorance and poor explanations, this is my first truenas server and I find myself learning new things about it every day and I'm constantly playing with new features I discover. Please do ask clarifying questions if I say anything that doesn't make sense, and I really appreciate the help as well

 

[winnielinnie]

Based on the output, you’re out of luck.

Then, I destroyed that pool and recreated it with encryption at the pool level so that nas/backups and nas/storage (and all other datasets) inherited the nas encryption. These are all the changes I remember making.

The only last-ditch idea is that somehow one of your old keystrings is actually “new” and will work if pasted manually.

Or if your config file from December 14, 2022 happens to contain the correct keystring.

If you did the above (quoted) steps sometime after December 14, then there’s nowhere else the keyfile could exist (unless you explicitly backed up the config DB + secret seed and/or the dataset keys at a later date beyond December 14.)

 

[mangof60]

Based on the output, you’re out of luck.



The only last-ditch idea is that somehow one of your old keystrings is actually “new” and will work if pasted manually.

Or if your config file from December 14, 2022 happens to contain the correct keystring.

If you did the above (quoted) steps sometime after December 14, then there’s nowhere else the keyfile could exist (unless you explicitly backed up the config DB + secret seed and/or the dataset keys at a later date beyond December 14.)

I wish I could celebrate with good news, but unfortunately I tried both of those already and didn't have any luck

I appreciate the help and advice from you all though and hopefully this thread can help someone else in the same situation in the future