Pool not found error after successful migration

[flawlessgr]

After successfully importing an existing encrypted pool to new hardware, I cannot remove the passphrase or even use it for unlocking the pool.
I had a freenas system for 2 years now, recently updated to 11.2
The only pool was encrypted without passphrase. Recently I decided to move to new hardware and started reading the documentation for how to migrate the pool.
I had to set a passphrase to the encrypted pool or else I wouldn't be able to import it again. The detailed steps I took follow:
1. I set a passphrase, downloaded the encryption key, downloaded the recovery key
2. I saved the config including the Password Secret Seed
3. I disconnected the pool selecting only Confirm Export/Disconnect
4. I installed freenas 11.3 on a new server
5. I connected the 6 disks of the pool to the new server
6. Through GUI I imported the encrypted pool. When prompted I uploaded the encryption key and entered the passphrase. The import was successful. I immediately restored the config of the old system and the freenas rebooted to complete the process.
7. After rebooting I had to unlock the pool. When I entered the passphrase the unlock failed. I didn't keep the message. I tried to unlock the pool using the recovery key and was able to unlock it.
8. I tried to remove the passphrase but the operation failed and the detail message was:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 97, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 53, in _run
return await self._call(name, serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
return methodobj(*params)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 390, in import_pool
'Failed to mount datasets after importing "%s" pool: %s', name_or_guid, str(e), exc_info=True
File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 380, in import_pool
raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 14733605188247631968 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1667, in unlock
'cachefile': ZPOOL_CACHE_FILE,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1141, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1081, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1101, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1036, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1010, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 14733605188247631968 not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
await self.future
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 385, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1679, in unlock
raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 6 devices failed to decrypt.

To avoid any wrong action that could destroy everything I left it as is and ask for your help.
Please explain what caused the error and how it can be resolved. At least it is not a disaster yet as I have a working system with an available pool.

Thank you in advance for any help.

New system info can be shared if needed, but I don't think is relevant to my problem.

 

[flawlessgr]

Hello,
Not even the slightest idea what to do?

 

[sretalla]

Through GUI I imported the encrypted pool. When prompted I uploaded the encryption key and entered the passphrase. The import was successful. I immediately restored the config of the old system and the freenas rebooted to complete the process.

This might be where the problem happened.

Restoring the config will overwrite the list of pools which automount. Normally in a restore, you don't import the pool before having done the config upload and reboots.

I disconnected the pool selecting only Confirm Export/Disconnect

Also, this will have left the pool in a state where it thinks it's exported, but the config backup doesn't agree about that. Subsequently, your import before the restore would have potentially put the pool back to imported, but with the "fresh" system, not the restored one.

 

[flawlessgr]

Thank you for your answer. Reading it now it makes perfect sense.
So, I should make a fresh install of freenas, restore the config backup and then import the pool?

If I do this now, would I be able to import the pool. or it will not have been disconnected properly due to the issue I am facing?
I wouldn't like to follow the "trial and error" process myself...

 

[sretalla]

So, I should make a fresh install of freenas, restore the config backup and then import the pool?

Normally that wouldn't be necessary as the config includes the pools already imported... if you exported your pool before the config backup, you will need to import it.

Because you created a non-normal situation by exporting your pool after the last config backup, the pool thinks it's exported, but the config will not... I guess you will need to do some messing around to get it to match up at the end.

Importing and exporting the pool at the command line should correct anything and make it ready for import again in the GUI. (zpool import <poolname>, zpool export <poolname>)