Unable to import pool after version upgrade

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
I am having a problem where my TrueNAS Scale server is having an issue importing the storage pool after an upgrade to version 23. When it first starts up, there is no pool attached and if go to Storage > Import Pool, I get the following error:

1703473206992.png

1703473226402.png

1703473242667.png


1703473820197.png


Code:
(sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name [SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)] [parameters: ('RAIDZ1Pool', '13164212532915322410')] (Background on this error at: https://sqlalche.me/e/14/gkpj)


Code:
Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: storage_volume.vol_name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body
    rv = await self.method(*([self] + args))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 186, in import_pool
    pool_id = await self.middleware.call('datastore.insert', 'storage.volume', {
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/write.py", line 62, in insert
    result = await self.middleware.call(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/connection.py", line 106, in execute_write
    result = self.connection.execute(sql, binds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1365, in execute
    return self._exec_driver_sql(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1669, in _exec_driver_sql
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name
[SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)]
[parameters: ('RAIDZ1Pool', '13164212532915322410')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)


Code:
Dec 23 03:24:41 epsilonshrike kernel: block device autoloading is deprecated and will be removed.
Dec 23 03:24:41 epsilonshrike kernel: md/raid1:md127: not clean -- starting background reconstruction
Dec 23 03:24:41 epsilonshrike kernel: md/raid1:md127: active with 2 out of 2 mirrors
Dec 23 03:24:41 epsilonshrike kernel: md127: detected capacity change from 0 to 4188160
Dec 23 03:24:41 epsilonshrike kernel: md: resync of RAID array md127
Dec 23 03:24:41 epsilonshrike kernel: block device autoloading is deprecated and will be removed.
Dec 23 03:24:41 epsilonshrike kernel: md/raid1:md126: not clean -- starting background reconstruction
Dec 23 03:24:41 epsilonshrike kernel: md/raid1:md126: active with 2 out of 2 mirrors
Dec 23 03:24:41 epsilonshrike kernel: md126: detected capacity change from 0 to 4188160
Dec 23 03:24:41 epsilonshrike kernel: md: resync of RAID array md126
Dec 23 03:24:42 epsilonshrike kernel: block device autoloading is deprecated and will be removed.
Dec 23 03:24:42 epsilonshrike kernel: md/raid1:md125: not clean -- starting background reconstruction
Dec 23 03:24:42 epsilonshrike kernel: md/raid1:md125: active with 2 out of 2 mirrors
Dec 23 03:24:42 epsilonshrike kernel: md125: detected capacity change from 0 to 4188160
Dec 23 03:24:42 epsilonshrike kernel: md: resync of RAID array md125
Dec 23 03:24:42 epsilonshrike kernel: block device autoloading is deprecated and will be removed.
Dec 23 03:24:42 epsilonshrike kernel: md/raid1:md124: not clean -- starting background reconstruction
Dec 23 03:24:42 epsilonshrike kernel: md/raid1:md124: active with 2 out of 2 mirrors
Dec 23 03:24:42 epsilonshrike kernel: md124: detected capacity change from 0 to 4188160
Dec 23 03:24:42 epsilonshrike kernel: md: resync of RAID array md124
Dec 23 03:24:43 epsilonshrike kernel: Adding 2094076k swap on /dev/mapper/md127.  Priority:-2 extents:1 across:2094076k FS
Dec 23 03:24:44 epsilonshrike kernel: Adding 2094076k swap on /dev/mapper/md126.  Priority:-3 extents:1 across:2094076k FS
Dec 23 03:24:44 epsilonshrike kernel: Adding 2094076k swap on /dev/mapper/md125.  Priority:-4 extents:1 across:2094076k FS
Dec 23 03:24:44 epsilonshrike kernel: Adding 2094076k swap on /dev/mapper/md124.  Priority:-5 extents:1 across:2094076k FS
Dec 23 03:24:56 epsilonshrike kernel: md: md127: resync done.
Dec 23 03:24:56 epsilonshrike kernel: md: md126: resync done.
Dec 23 03:24:56 epsilonshrike kernel: md: md125: resync done.
Dec 23 03:24:57 epsilonshrike kernel: md: md124: resync done.


Afterwards, I see that the pool has in fact imported, however, I am unable to access it via SMB. I can see the server but when I navigate to it in Windows, it just shows an empty folder. This may be a separate issue, though. I was able to access the share before upgrading.

1703473851538.png


I am able to browse the share via SSH or the built in shell.

If I restart the server, the pool is again disconnected and I have to import it again to access it.

Here are my boot environments to show my past versions:
1703473897131.png


Any help would be greatly appreciated.
 
Last edited:

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Any help would be greatly appreciated!
 

tprelog

Patron
Joined
Mar 2, 2016
Messages
297
I can't help with your issue, but I think that Zpool is dangerously overfilled.

For more experienced users to help, please provide the details and specs of your NAS. (I see you did this in your signature but you have not said which server is having issues, also those signatures do not show up when viewed from a mobile device so it will be helpful to include them again in your post)
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Server specs:

Dell PowerEdge T320 Chassis
Intel Xeon E5-2430 v2 2.5-3.0 GHz 6C/12T
48GB (3x16GB) DDR3-1066 ECC
IBM M1215 LSI SAS3008 HBA cross flashed to 9300-8i 12Gb/s SAS in IT Mode w/ Noctua NF-A4x20 PWM
Storage Pool: 8 x 10TB WD100EMAZ in RAIDZ1 in Hot Swap Caddies
Boot: 2 x 60GB Silicon Power S60 MLC SSD RAIDZ Mirror in Icy Dock MB994SP-4SB-1
Mellanox ConnectX-3 EN 10Gbe SFP+ NIC w/ Noctua NF-A4x20 PWM
Redundant 495W Platinum Hot Swap PSU's
iDRAC 7 Enterprise w/ Virtual Console
Chassis fan swapped for Noctua NF-F12 PWM 120mm

The pool is near capacity, but it still has 4.5 terabytes of free space. I will try and clean it up, though. I know it should be around 80% but I was under the impression that was not as important if I am using it as a host for video media.
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
This caught my eye.

It seems like some kind of config file corruption. I (think) hopefully your pool is unaffected, but you probably should do something about two things:
  • 92% capacity.
  • 8-wide RAIDZ1 topography.
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
This caught my eye.

It seems like some kind of config file corruption. I (think) hopefully your pool is unaffected, but you probably should do something about two things:
  • 92% capacity.
  • 8-wide RAIDZ1 topography.
I appreciate it. I looked at that when I was doing my initial troubleshooting but I wasn't able to act upon any of the information in there. What is the issue with the 92% capacity and the 8 wide topography?
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
I appreciate it. I looked at that when I was doing my initial troubleshooting but I wasn't able to act upon any of the information in there. What is the issue with the 92% capacity and the 8 wide topography?
To quote Matthew Ahrens (co-founder of ZFS):
“For best performance on random IOPS, use a small number of disks in each RAID-Z group. E.g, 3-wide RAIDZ1, 6-wide RAIDZ2, or 9-wide RAIDZ3 (all of which use ⅓ of total storage for parity, in the ideal case of using large blocks). This is because RAID-Z spreads each logical block across all the devices (similar to RAID-3, in contrast with RAID-4/5/6). For even better performance, consider using mirroring.
Basically, your pool is not only at 92% capacity, but the vdev is so wide that even without being overfilled, performance is already terrible.
That's not even getting into how it's kinda' dangerous to have only 1-drive redundancy in a vdev of 8, but that's another topic for another day.
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
To quote Matthew Ahrens (co-founder of ZFS):

Basically, your pool is not only at 92% capacity, but the vdev is so wide that even without being overfilled, performance is already terrible.
That's not even getting into how it's kinda' dangerous to have only 1-drive redundancy in a vdev of 8, but that's another topic for another day.

Understood. Once I am able to access my array again, I will look at making those changes. As it stands, I can't even go in and do any of this, so I really need some help figuring out what is wrong in the first place :(
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Anybody? I'm pretty desperate here :(
 

PhilD13

Patron
Joined
Sep 18, 2020
Messages
203
Edit Forgot not successful - Try running zpool import POOLNAME (where POOLNAME is name of pool) from the command line
It may provide more info.

If it imports but you ca't see it, what does zpool status show?
 
Last edited:

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Edit Forgot not successful - Try running zpool import POOLNAME (where POOLNAME is name of pool) from the command line
It may provide more info.

If it imports but you ca't see it, what does zpool status show?
So after a reboot, this is what I see:

Code:
root@epsilonshrike[~]# zpool status
  pool: freenas-boot
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:10:01 with 0 errors on Wed Dec 27 03:55:03 2023
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            sdi2      ONLINE       0     0     0
            sde2      ONLINE       0     0     0

errors: No known data errors


When I run the zpool import command, it does not show anything in the shell:

Code:
root@epsilonshrike[~]# zpool import RAIDZ1Pool
root@epsilonshrike[~]#


This is what the status says afterwards:

Code:
root@epsilonshrike[~]# zpool status
  pool: RAIDZ1Pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 19:07:28 with 0 errors on Mon Jan  1 00:07:30 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        RAIDZ1Pool                                ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            051e4f92-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            05999c85-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            0613a6bc-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            069bdc5e-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            0730f19c-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            07dfa9da-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            087f368c-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0
            0923b666-059b-11e9-928b-a0369f1e306a  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:10:01 with 0 errors on Wed Dec 27 03:55:03 2023
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            sdi2      ONLINE       0     0     0
            sde2      ONLINE       0     0     0

errors: No known data errors
 

PhilD13

Patron
Joined
Sep 18, 2020
Messages
203
Thanks for posting the info. There seems to be a number of posts with similar issues. So this may be a bug issue that needs to be addressed.

pool: freenas-boot
state: ONLINE
status: One or more features are enabled on the pool despite not being
requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
adding needed features to the relevant file in
/etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
I don't see anything with the pool except the above
I'm not sure what feature is not enabled that compatibility needs wants or does not want. Maybe someone else has an idea. Is it possible you updated the Flags on the pool before upgrading and now that is interfering?

Some are able to manually start the services so you might try that to get SMB going again.
The common thing seems to be the internal database not being populated correctly during the upgrade. Your pool does look like it is online,

A long shot is trying lsblk might shed some light on if the system sees the structure correctly without errors. I think it will though.
 

terrysimons

Cadet
Joined
Dec 3, 2022
Messages
6
I'm dealing with this too. I have a second post here, but happy to move my discussion over here if it's helpful: https://www.truenas.com/community/t...after-upgrading-from-bluefin-to-cobia.115326/

One thing I realized thinking about the genesis of my system is that I *DID* have a drive die on me, so I replaced it in Bluefin - I may be mis-reading the error message, but it almost seems like it's complaining it can't find a disk - I'm wondering if there's some issue related to having replaced a failing disk and then migrating to Cobia.

@oRAirwolf did you ever have to replace a disk in Bluefin before your Cobia upgrade?

[edit] I can also confirm that restarting my services after importing the pool results in my system functioning. I'm using Samba & iSCSI.
 

terrysimons

Cadet
Joined
Dec 3, 2022
Messages
6
Thanks for posting the info. There seems to be a number of posts with similar issues. So this may be a bug issue that needs to be addressed.


I don't see anything with the pool except the above
I'm not sure what feature is not enabled that compatibility needs wants or does not want. Maybe someone else has an idea. Is it possible you updated the Flags on the pool before upgrading and now that is interfering?

Some are able to manually start the services so you might try that to get SMB going again.
The common thing seems to be the internal database not being populated correctly during the upgrade. Your pool does look like it is online,

A long shot is trying lsblk might shed some light on if the system sees the structure correctly without errors. I think it will though.
I'm seeing a slightly different message, so I don't think the issue is related to this. Mine complains that my zpool version is old:

Code:
root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:21 with 0 errors on Mon Jan  1 03:45:22 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda3    ONLINE       0     0     0
            sdb3    ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 03:29:38 with 0 errors on Sun Nov 19 03:29:40 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            ae821803-c186-46fa-abbc-2d42216ede86  ONLINE       0     0     0
            a5d13517-941d-4b02-8f4e-7a9d5ad932e3  ONLINE       0     0     0
            148e75be-c14d-44fd-9a80-bf48a530e2d6  ONLINE       0     0     0
            4c228148-69aa-412a-8913-f933a08ca628  ONLINE       0     0     0
            94cb4165-8af7-43e7-9fb6-c357f22c7c45  ONLINE       0     0     0
            200d58b8-077d-403e-a680-dfabad61ca96  ONLINE       0     0     0
            b25ec283-4147-42e9-9714-46fbd796a067  ONLINE       0     0     0
        logs
          mirror-1                                ONLINE       0     0     0
            fa7a8cb3-78db-472a-9e8f-d3dd9a1701b5  ONLINE       0     0     0
            ab1b8786-f4fd-4eb7-8804-6b3e9904d006  ONLINE       0     0     0
        cache
          af6ca5bf-0a15-4bc7-8b91-8d0c3635a873    ONLINE       0     0     0
          1f4a5192-ddf0-4aac-b2fd-f8008fa52325    ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# 


This looks more like a TrueNAS migration failure, since it throws an error in the UI about SQL database migration. The zpool import part seems to work fine.
 

PhilD13

Patron
Joined
Sep 18, 2020
Messages
203
@terrysimons
That would be a normal message if the zfs feature flags have not been updated. I don't see that message on your boot pool in the status info posted. I do agree it may be a migration issue where the boot pool flags may be partially updated during the migration leaving the system to expect a feature flag for something that is actually missing then that may be where the issues are resulting from. The new release of Cobia does have new zfs features.

I did a new fresh bare metal install last week on a system using Cobia and the zpool status does not show the feature flag message for any of the pools, so this kind of reinforces the thought it's a issue in the migration.
 

flmmartins

Dabbler
Joined
Sep 19, 2022
Messages
31
I don't see anything with the pool except the above
I'm not sure what feature is not enabled that compatibility needs wants or does not want. Maybe someone else has an idea. Is it possible you updated the Flags on the pool before upgrading and now that is interfering?

Some are able to manually start the services so you might try that to get SMB going again.
The common thing seems to be the internal database not being populated correctly during the upgrade. Your pool does look like it is online,

A long shot is trying lsblk might shed some light on if the system sees the structure correctly without errors. I think it will though.

I seem to have exactly this issue where my database is not populated correctly after upgrade from Core to Scale here:

How can I correct the database?
 

janerjak

Dabbler
Joined
Feb 14, 2024
Messages
11
Any update here? This post was created Dec 25, 2023 and this seems to still be an issue. I have many issues with TrueNAS that I hope are resolved in the new Cobia release - but I can't test any of that without my pool.

(Unrelated, but e.g. I want to finally get Nextcloud running and couldn't so far, because the official Nextcloud app has been broken with custom user storage host paths since over a year and still is to this day. Current TrueCharts apps require Cobia and I failed to setup the community maintained Docker image using anything but SQLite)

What can we do to assist in this? I really want to somehow stay on TrueNAS
 

janerjak

Dabbler
Joined
Feb 14, 2024
Messages
11
Top