Unable to import pool after upgrading from Bluefin to Cobia

[terrysimons]

Hi folks,

I'm seeing something similar to https://www.truenas.com/community/threads/unable-to-import-pool-after-version-upgrade.115158/ but my tank size is 25.5T and only 53% used space, so the above thread discussion, while the same error, seems like it must be some other issue than what is described in the above thread.

My system is an ASRock Rack X570D4I-2T with a Ryzen 5950X with 128GB of DDR4 ECC with an ASUS Hyper M.2 x16 Gen 4 Card hosting 4xSamsung SSD 983 DCT M.2 960GB drives for 2x mirrored zlog and 2x cache VDEVs.

My topology is:

Code:

Data VDEVs 1 x RAIDZ1 | 7 wide | Mixed Capacity

Metadata VDEVsVDEVs not assigned

Log VDEVs 1 x MIRROR | 2 wide | 894.25 GiB

Cache VDEVs 2 x 894.25 GiB

Spare VDEVs VDEVs not assigned

Dedup VDEVs VDEVs not assigned

The specific drive makeup of my RAIDZ1 pool is:

Code:

/mnt/tank:

5x3.64TB ST4000NM0035-1V4107
1x12.73TB WDC_WD140EFFX-68VBXN0
1x16.37TB WDC_WD181KRYZ-01AGBB0

Log VDEVs 2xSamsung SSD 983 DCT M.2 960GB (mirrored)

Cache VDEV:

2xSamsung SSD 983 DCT M.2 960GB

A couple of potentially pertinent points:

IIRC I had done several upgrades over time of my TrueNAS MiniX+ system, which eventually I upgraded from TrueNAS Core to TrueNAS Scale, then I built the above system and did a replication to this new system from the MiniX+, and then applied the MiniX+ config backup to restore my share configurations and accounts.

TL;DR I had done a lot of CORE updates over time and then a migration to scale before migrating my data from my MiniX+ to the new system listed above.

I've also upgraded BlueFin on this ASRock system several times (22.12.3.1 -> 22.12.3.2 -> 22.12.4.2 -> 23.10.1). The Cobia 23.10.1 upgrade broke things, however.

Just in case my issue is different from the above (seems very similar from the error messages) here's my specific error.

When I boot up Cobia my pool is disconnected. If I import it, I get an error:

Code:

Error importing pool​
(sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name [SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)] [parameters: ('tank', '5840402838289317082')] (Background on this error at: https://sqlalche.me/e/14/gkpj)

 Error: Traceback (most recent call last):  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context    self.dialect.do_execute(  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute    cursor.execute(statement, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: storage_volume.vol_name The above exception was the direct cause of the following exception: Traceback (most recent call last):  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run    await self.future  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body    rv = await self.method(*([self] + args))         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf    return await func(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf    res = await f(*args, **kwargs)          ^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 186, in import_pool    pool_id = await self.middleware.call('datastore.insert', 'storage.volume', {              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call    return await self._call(           ^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in _call    return await methodobj(*prepared_call.args)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf    return await func(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/write.py", line 62, in insert    result = await self.middleware.call(             ^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call    return await self._call(           ^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in _call    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run    result = self.fn(*self.args, **self.kwargs)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/connection.py", line 106, in execute_write    result = self.connection.execute(sql, binds)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1365, in execute    return self._exec_driver_sql(           ^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1669, in _exec_driver_sql    ret = self._execute_context(          ^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context    self._handle_dbapi_exception(  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception    util.raise_(  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 211, in raise_    raise exception  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context    self.dialect.do_execute(  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute    cursor.execute(statement, parameters) sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name [SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)] [parameters: ('tank', '5840402838289317082')] (Background on this error at: https://sqlalche.me/e/14/gkpj)  

After import, my pool is online and I can access it as expected after manually restarting all services.

Since my system is well below the 80% capacity that the above thread discussion was about, but I'm still seeing import errors similar to the above post, I'm curious what other information I can gather to debug this issue.

 

[terrysimons]

Here is some additional information about my pool after I manually import:

Code:

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:17 with 0 errors on Sat Dec 23 03:45:18 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb3    ONLINE       0     0     0
            sda3    ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 03:29:38 with 0 errors on Sun Nov 19 03:29:40 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            ae821803-c186-46fa-abbc-2d42216ede86  ONLINE       0     0     0
            a5d13517-941d-4b02-8f4e-7a9d5ad932e3  ONLINE       0     0     0
            148e75be-c14d-44fd-9a80-bf48a530e2d6  ONLINE       0     0     0
            4c228148-69aa-412a-8913-f933a08ca628  ONLINE       0     0     0
            94cb4165-8af7-43e7-9fb6-c357f22c7c45  ONLINE       0     0     0
            200d58b8-077d-403e-a680-dfabad61ca96  ONLINE       0     0     0
            b25ec283-4147-42e9-9714-46fbd796a067  ONLINE       0     0     0
        logs
          mirror-1                                ONLINE       0     0     0
            fa7a8cb3-78db-472a-9e8f-d3dd9a1701b5  ONLINE       0     0     0
            ab1b8786-f4fd-4eb7-8804-6b3e9904d006  ONLINE       0     0     0
        cache
          af6ca5bf-0a15-4bc7-8b91-8d0c3635a873    ONLINE       0     0     0
          1f4a5192-ddf0-4aac-b2fd-f8008fa52325    ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   912G  10.4G   902G        -         -     0%     1%  1.00x    ONLINE  -
tank       25.5T  11.2T  14.3T        -         -     8%    43%  1.00x    ONLINE  /mnt
root@truenas[~]#