Cannot unlock volume: 'utf-8' codec can't decode byte error

[ibrt]

I am attempting to migrate from OpenZFS on a Debian 11 server to TrueNAS Core 13. I have been able to import my pool which has encrypted datasets.

When I attempt to unlock my data set by uploading the 32 byte (256 bit AES) key file, TrueNAS fails to import the key because for some reason it is trying to treat it as JSON?

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
    await self.future
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 393, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1154, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1151, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 2574, in encryption_summary
    keys_supplied = self._retrieve_keys_from_file(job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 2791, in _retrieve_keys_from_file
    data = json.loads(job.pipes.input.r.read(10240))
  File "/usr/local/lib/python3.9/json/__init__.py", line 341, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x<REDACTED> in position 3: invalid continuation byte

What is going on here? Why is it trying to treat this key file as a JSON document?

 

[ibrt]

I figured out the format by creating a new encrypted data set and exporting the key. I learned the key must be specified for each child data set as well, even if the key is the same for all so I wrote a quick Bash script to create the expected JSON format from a key file and list of data set names.

Example Usage:

bash truenas_key_format.sh path_to_keyfile tank/dataset1,tank/dataset1/sub1,tank/dataset1,sub2

Code:

#!/usr/bin/env bash

set -eou pipefail

PATH_TO_KEY_FILE="${1}"
DATA_SET_NAMES="${2}"

error () {
    echo "${1}"
    exit 1
}

if [[ -z "${PATH_TO_KEY_FILE}" ]]; then
    error "Path to the AES key file must be supplied."
fi
if [[ -z "${DATA_SET_NAMES}" ]]; then
    error "At least one data set must be specified."
fi

KEY=$(xxd -p -c 64 "${PATH_TO_KEY_FILE}")
JSON="{"

IFS="," read -ra DS_NAMES <<< "${DATA_SET_NAMES}"
for DS in "${DS_NAMES[@]}"; do
    JSON="${JSON}\"${DS}\":\"${KEY}\","
done

JSON="${JSON%?}}"
echo "${JSON}"

 

[winnielinnie]

When I attempt to unlock my data set by uploading the 32 byte (256 bit AES) key file

You can alternatively just copy+paste the keystring instead of uploading a file.

Then you can export the .json From TrueNAS, which will automatically populate any relevant child datasets that are their own "encryptionroot".

 

[ibrt]

You can alternatively just copy+paste the keystring instead of uploading a file.

Then you can export the .json From TrueNAS, which will automatically populate any relevant child datasets that are their own "encryptionroot".

It's not a string, it's random bytes. The UI did not present me with any option other than uploading a key file.

 

[winnielinnie]

The UI did not present me with any option other than uploading a key file.

Really? The last time I checked on TrueNAS Core, there's a checkbox to let you type/paste the HEX string(s), rather than uploading a file.

For instance, rather than upload my file named "dataset_mypool_keys.json", I can open up the .json file in any text viewer:

Code:

{"mypool": "f4e228d17e09775dfdf3d4afca13fdade6a89b41aba37138e8ec3ae0ea2f44f0"}

I can then copy+paste the string f4e228d17e09775dfdf3d4afca13fdade6a89b41aba37138e8ec3ae0ea2f44f0 directly into the input field when importing and unlocking the dataset(s). No need to upload anything.

Unless they removed this option recently?

(Before anyone asks. Yes. That is indeed a fake string I made up for this example.)