geli-encrypted pool failed to decrypt after adding vdev on TrueNAS 13

[fuzzydunlop]

Hello, I have a server running TrueNAS 13 which has been running with two usb drives for the boot-pool and a single storage pool of two 4tb drives in a mirror vdev for many years. This pool is geli encrypted as it was originally created when the server was built, running FreeNAS 11, but the server has since been upgraded all the way to TrueNAS (13.0 U6). This is my drive setup, the bottom entry is not relevant, it's just a flash drive plugged into the host:

I then used the menu under Storage > Pools > Gear Icon > Add vdevs. This appeared to work as expected (and I downloaded the new encryption key when prompted), but eventually the server was rebooted and now the entire pool fails to decrypt. When I go to Storage > Pools I see this:

When I look for geli in console.log I see several entries which seem to indicate either the geli provider or the key file are incorrect

The key still exists in the proper folder

The geli list still shows two providers as expected, but they appear to have different geom IDs than TrueNAS is expecting

I have not yet done anything more on this host, as I wanted to check here first, but I was able to recreate the issue in a VM so I could test potential solutions by doing the following:

1. Install FreeNAS 11
2. Create encrypted pool of one mirror vdev (uses geli)
3. Upgrade to TrueNAS 12 and then to TrueNAS 13.0 u6
4. Add new mirror vdev to pool as described above above
5. Reboot, now I see the pool failed to decrypt

From the actual server with the issue I have the recovery key originally created on FreeNAS 11 when the pool was first created (I assume this is no longer useful after adding a vdev though), and the encryption key I was prompted to download when creating the new vdev in TrueNAS 13.

In my VM lab environment I've tried the following with no success:

• Run
  Code:

  geli attach

  with this key but the proper geom names
  • 
• Run the same but with the encryption key I downloaded when adding the new vdev, same results as above
• Export/disconnect pool in UI using the gear icon menu so I can try to reimport it, fails to export
  • 
• Considered using
  Code:

  zpool export

  but there's only the boot pool available when running
  Code:

  zpool list

  • 
• Try using
  Code:

  geli detach

  , which works (they no longer show when listing geoms) but the pool still shows up in the UI even after rebooting and the drives are not available in the dropdown when trying to add a new pool.
  • Also tried to
    Code:

    geli attach

    after this, both with the key originally on the server filesystem as well as the key downloaded when adding a vdev, but in both cases I see the same "error with at least one provider" shown above

Does anyone have suggestions on other options to try, further debugging steps, or good reference materials to read through? I've read the other forum posts where users report similar errors decrypting geli pools, but most seem to be resolved by placing the key in the right location which didn't work here. I do have a cloud backup of the important data on this drive, but restoring that data would be costly in egress fees and require me to do a decent amount of reconfiguring on this server so I'd prefer to be able to decrypt this existing pool if possible.

 

[fuzzydunlop]

So after posting this, I tried downgrading the VM to 12 using the boot environment menu. After that I was able to disconnect and reconnect the pool. Unfortunately in my real environment all the boot items are 13. Is there another way for me to downgrade safely?

 

[fuzzydunlop]

I'm back into the pool! Although there was no option to downgrade to 12 on the impacted server, I tried the default boot option which took it back to truenas 13 release. From there I was still able to disconnect/export the pool then reimport it with the key from when I added a vdev. Now I need to figure out how to safely upgrade the encryption on this pool as I can't update TrueNAS any more until that's done.