Upgrade from Bluefin to Cobia: Pool fails to import

[janerjak]

Update: This seems to have been fixed.


Today I attempted to upgrade my TrueNAS SCALE install from Bluefin 22.12.4.2 to Cobia 23.10.1.3.

After rebooting, my pool is not automatically imported and unavailable. Reimporting fails.

I am experiencing the same issue as two other users described here:

Unable to import pool after version upgrade

I am having a problem where my TrueNAS Scale server is having an issue importing the storage pool after an upgrade to version 23. When it first starts up, there is no pool attached and if go to Storage > Import Pool, I get the following error: (sqlite3.IntegrityError) UNIQUE constraint...

www.truenas.com

Unable to import pool after upgrading from Bluefin to Cobia

Hi folks, I'm seeing something similar to https://www.truenas.com/community/threads/unable-to-import-pool-after-version-upgrade.115158/ but my tank size is 25.5T and only 53% used space, so the above thread discussion, while the same error, seems like it must be some other issue than what is...

www.truenas.com

Other to these folks, I neither have a "dangerously full" pool, nor have I previously upgraded from TrueNAS Core or other older SCALE versions before.
I clean-installed Bluefin on the 18th of December 2023 and this is my first TrueNAS update ever.

This is how the storage dashboard looks after a reboot:

Clicking "Import Pool" and then selecting my pool named "Rapid" results in this error:

(Note that this is an SQLite UNIQUE constraint failing on the volume names)
After refreshing the page, the pool seems to have imported fine, but it is clearly in a limbo state: Rebooting results in the same issue as above again. This is how it looks after refreshing the Web UI:

I can now browse the pool using SSH. Samba does not work (eventhough the service is supposedly running, I don't get a valid response using macOS). The application panel states that "apps are not running":

As you can see, this was a upgrade from a fresh Bluefin install, using 9% disk usage on one pool with a 4 disk RAIDZ1 vdev. This seems to be an issue with TrueNAS and not my config.

Here are my system specs:

AMD Ryzen 5 PRO 4650G with Radeon Graphics
64 GB Unregistered ECC DDR4-3000
4x 4TB Crucial P3 Plus SSD
1x 500 GB Samsung 970 EVO SSD

If there is anything else that could help to debug this, please tell me. I will also file a bug ticket (?). For the time I will try to rollback to Bluefin and hopefully get back up and running.


Thanks, Jan

 

[janerjak]

Today I attempted to upgrade my TrueNAS SCALE install from Bluefin 22.12.4.2 to Cobia 23.10.1.3.

After rebooting, my pool is not automatically imported and unavailable. Reimporting fails.

I am experiencing the same issue as two other users described here:

Unable to import pool after version upgrade

I am having a problem where my TrueNAS Scale server is having an issue importing the storage pool after an upgrade to version 23. When it first starts up, there is no pool attached and if go to Storage > Import Pool, I get the following error: (sqlite3.IntegrityError) UNIQUE constraint...

www.truenas.com

Unable to import pool after upgrading from Bluefin to Cobia

Hi folks, I'm seeing something similar to https://www.truenas.com/community/threads/unable-to-import-pool-after-version-upgrade.115158/ but my tank size is 25.5T and only 53% used space, so the above thread discussion, while the same error, seems like it must be some other issue than what is...

www.truenas.com

Other to these folks, I neither have a "dangerously full" pool, nor have I previously upgraded from TrueNAS Core or other older SCALE versions before.
I clean-installed Bluefin on the 18th of December 2023 and this is my first TrueNAS update ever.

This is how the storage dashboard looks after a reboot:
View attachment 75678

Clicking "Import Pool" and then selecting my pool named "Rapid" results in this error:

View attachment 75679

(Note that this is an SQLite UNIQUE constraint failing on the volume names)
After refreshing the page, the pool seems to have imported fine, but it is clearly in a limbo state: Rebooting results in the same issue as above again. This is how it looks after refreshing the Web UI:

View attachment 75681

I can now browse the pool using SSH. Samba does not work (eventhough the service is supposedly running, I don't get a valid response using macOS). The application panel states that "apps are not running":
View attachment 75682

As you can see, this was a upgrade from a fresh Bluefin install, using 9% disk usage on one pool with a 4 disk RAIDZ1 vdev. This seems to be an issue with TrueNAS and not my config.

The error seems to point towards duplicate volume names. I have multiple datasets that are sharing the same base name. For instance I have Rapid/app-data/nextcloud/data, but also Rapid/app-data/nginx-proxy-manager/data. Just a wild guess and I don't expect this to be the issue, but something is clashing here. This is my pool dataset tree for reference (first images, then the output of zfs list):

View attachment 75683
View attachment 75684

Code:

root@gigakiste[/mnt/Rapid/home/darkn]# zfs list
NAME                                                                           USED  AVAIL  REFER  MOUNTPOINT
Rapid                                                                          715G  9.66T   186K  /mnt/Rapid
Rapid/.system                                                                 1.32G  9.66T  1.20G  /mnt/Rapid/.system
Rapid/.system/configs-f1f5036a6e4448d09a9ddb3c45165866                        14.4M  9.66T  14.4M  /mnt/Rapid/.system/configs-f1f5036a6e4448d09a9ddb3c45165866
Rapid/.system/cores                                                            140K  1024M   140K  /mnt/Rapid/.system/cores
Rapid/.system/ctdb_shared_vol                                                  140K  9.66T   140K  /mnt/Rapid/.system/ctdb_shared_vol
Rapid/.system/glusterd                                                         151K  9.66T   151K  /mnt/Rapid/.system/glusterd
Rapid/.system/rrd-f1f5036a6e4448d09a9ddb3c45165866                            66.0M  9.66T  66.0M  /mnt/Rapid/.system/rrd-f1f5036a6e4448d09a9ddb3c45165866
Rapid/.system/samba4                                                          1.86M  9.66T   953K  /mnt/Rapid/.system/samba4
Rapid/.system/services                                                         140K  9.66T   140K  /mnt/Rapid/.system/services
Rapid/.system/syslog-f1f5036a6e4448d09a9ddb3c45165866                         33.5M  9.66T  33.5M  /mnt/Rapid/.system/syslog-f1f5036a6e4448d09a9ddb3c45165866
Rapid/.system/webui                                                            140K  9.66T   140K  /mnt/Rapid/.system/webui
Rapid/app-data                                                                1.31G  9.66T   163K  /mnt/Rapid/app-data
Rapid/app-data/game-servers                                                   4.81M  9.66T   151K  /mnt/Rapid/app-data/game-servers
Rapid/app-data/game-servers/super-mario-odyssey-online                        4.43M  9.66T   163K  /mnt/Rapid/app-data/game-servers/super-mario-odyssey-online
Rapid/app-data/game-servers/ttt2                                               140K  9.66T   140K  /mnt/Rapid/app-data/game-servers/ttt2
Rapid/app-data/jellyfin                                                       48.0M  9.66T   140K  /mnt/Rapid/app-data/jellyfin
Rapid/app-data/jellyfin/cache                                                 7.54M  9.66T   140K  /mnt/Rapid/app-data/jellyfin/cache
Rapid/app-data/jellyfin/config                                                40.1M  9.66T  23.1M  /mnt/Rapid/app-data/jellyfin/config
Rapid/app-data/jellyfin/transcode                                              140K  9.66T   140K  /mnt/Rapid/app-data/jellyfin/transcode
Rapid/app-data/nextcloud                                                       425M  9.66T   151K  /mnt/Rapid/app-data/nextcloud
Rapid/app-data/nextcloud-aio                                                   140K  9.66T   140K  /mnt/Rapid/app-data/nextcloud-aio
Rapid/app-data/nextcloud/data                                                  412M  9.66T   412M  /mnt/Rapid/app-data/nextcloud/data
Rapid/app-data/nextcloud/postgress                                            12.7M  9.66T   140K  /mnt/Rapid/app-data/nextcloud/postgress
Rapid/app-data/nextcloud/postgress/backup                                      140K  9.66T   140K  /mnt/Rapid/app-data/nextcloud/postgress/backup
Rapid/app-data/nextcloud/postgress/data                                       12.4M  9.66T  12.3M  /mnt/Rapid/app-data/nextcloud/postgress/data
Rapid/app-data/nginx                                                           140K  9.66T   140K  /mnt/Rapid/app-data/nginx
Rapid/app-data/nginx-proxy-manager                                             150M  9.66T   140K  /mnt/Rapid/app-data/nginx-proxy-manager
Rapid/app-data/nginx-proxy-manager/certificates                               2.83M  9.66T  1.84M  /mnt/Rapid/app-data/nginx-proxy-manager/certificates
Rapid/app-data/nginx-proxy-manager/data                                        147M  9.66T   135M  /mnt/Rapid/app-data/nginx-proxy-manager/data
Rapid/app-data/pihole                                                          220M  9.66T   151K  /mnt/Rapid/app-data/pihole
Rapid/app-data/pihole/config                                                   220M  9.66T   112M  /mnt/Rapid/app-data/pihole/config
Rapid/app-data/pihole/dnsmasq                                                  546K  9.66T   174K  /mnt/Rapid/app-data/pihole/dnsmasq
Rapid/app-data/plex                                                            367M  9.66T   151K  /mnt/Rapid/app-data/plex
Rapid/app-data/plex/config                                                     367M  9.66T   284M  /mnt/Rapid/app-data/plex/config
Rapid/app-data/plex/data                                                       140K  9.66T   140K  /mnt/Rapid/app-data/plex/data
Rapid/app-data/plex/transcode                                                  174K  9.66T   174K  /mnt/Rapid/app-data/plex/transcode
Rapid/app-data/syncthing                                                      12.1M  9.66T   285K  /mnt/Rapid/app-data/syncthing
Rapid/app-data/tautulli                                                       19.2M  9.66T  9.38M  /mnt/Rapid/app-data/tautulli
Rapid/app-data/tdarr                                                          88.2M  9.66T   140K  /mnt/Rapid/app-data/tdarr
Rapid/app-data/tdarr/configs                                                   418K  9.66T   174K  /mnt/Rapid/app-data/tdarr/configs
Rapid/app-data/tdarr/handbrake-presets                                         244K  9.66T   221K  /mnt/Rapid/app-data/tdarr/handbrake-presets
Rapid/app-data/tdarr/logs                                                     1.98M  9.66T   314K  /mnt/Rapid/app-data/tdarr/logs
Rapid/app-data/tdarr/server                                                   85.4M  9.66T  51.2M  /mnt/Rapid/app-data/tdarr/server
Rapid/app-data/vaultwarden                                                    2.89M  9.66T  1.14M  /mnt/Rapid/app-data/vaultwarden
Rapid/home                                                                    2.72M  9.66T  2.72M  /mnt/Rapid/home
Rapid/ix-applications                                                         11.7G  9.66T  2.60M  /mnt/Rapid/ix-applications
Rapid/ix-applications/catalogs                                                 894M  9.66T   894M  /mnt/Rapid/ix-applications/catalogs
Rapid/ix-applications/default_volumes                                          140K  9.66T   140K  /mnt/Rapid/ix-applications/default_volumes
Rapid/ix-applications/docker                                                  10.5G  9.66T  10.5G  /mnt/Rapid/ix-applications/docker
Rapid/ix-applications/k3s                                                      290M  9.66T   271M  /mnt/Rapid/ix-applications/k3s
Rapid/ix-applications/k3s/kubelet                                             12.3M  9.66T  10.8M  legacy
Rapid/ix-applications/releases                                                14.0M  9.66T   163K  /mnt/Rapid/ix-applications/releases
Rapid/ix-applications/releases/jellyfin                                       2.38M  9.66T   151K  /mnt/Rapid/ix-applications/releases/jellyfin
Rapid/ix-applications/releases/jellyfin/charts                                1.45M  9.66T  1.33M  /mnt/Rapid/ix-applications/releases/jellyfin/charts
Rapid/ix-applications/releases/jellyfin/volumes                                802K  9.66T   140K  /mnt/Rapid/ix-applications/releases/jellyfin/volumes
Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes                     663K  9.66T   151K  /mnt/Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes
Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/cache               140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/cache
Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/config              140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/config
Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/transcodes          140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/jellyfin/volumes/ix_volumes/transcodes
Rapid/ix-applications/releases/nextcloud                                      1.44M  9.66T   151K  /mnt/Rapid/ix-applications/releases/nextcloud
Rapid/ix-applications/releases/nextcloud/charts                               1.02M  9.66T   953K  /mnt/Rapid/ix-applications/releases/nextcloud/charts
Rapid/ix-applications/releases/nextcloud/volumes                               279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nextcloud/volumes
Rapid/ix-applications/releases/nextcloud/volumes/ix_volumes                    140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nextcloud/volumes/ix_volumes
Rapid/ix-applications/releases/nginx                                          1.09M  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx
Rapid/ix-applications/releases/nginx-proxy-manager                             814K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx-proxy-manager
Rapid/ix-applications/releases/nginx-proxy-manager/charts                      395K  9.66T   395K  /mnt/Rapid/ix-applications/releases/nginx-proxy-manager/charts
Rapid/ix-applications/releases/nginx-proxy-manager/volumes                     279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx-proxy-manager/volumes
Rapid/ix-applications/releases/nginx-proxy-manager/volumes/ix_volumes          140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx-proxy-manager/volumes/ix_volumes
Rapid/ix-applications/releases/nginx/charts                                    407K  9.66T   407K  /mnt/Rapid/ix-applications/releases/nginx/charts
Rapid/ix-applications/releases/nginx/volumes                                   570K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx/volumes
Rapid/ix-applications/releases/nginx/volumes/ix_volumes                        430K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx/volumes/ix_volumes
Rapid/ix-applications/releases/nginx/volumes/ix_volumes/cache                  151K  9.66T   151K  /mnt/Rapid/ix-applications/releases/nginx/volumes/ix_volumes/cache
Rapid/ix-applications/releases/nginx/volumes/ix_volumes/run                    140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/nginx/volumes/ix_volumes/run
Rapid/ix-applications/releases/pihole                                         1.32M  9.66T   140K  /mnt/Rapid/ix-applications/releases/pihole
Rapid/ix-applications/releases/pihole/charts                                   581K  9.66T   465K  /mnt/Rapid/ix-applications/releases/pihole/charts
Rapid/ix-applications/releases/pihole/volumes                                  628K  9.66T   140K  /mnt/Rapid/ix-applications/releases/pihole/volumes
Rapid/ix-applications/releases/pihole/volumes/ix_volumes                       488K  9.66T   140K  /mnt/Rapid/ix-applications/releases/pihole/volumes/ix_volumes
Rapid/ix-applications/releases/pihole/volumes/ix_volumes/dnsmasq               256K  9.66T   163K  /mnt/Rapid/ix-applications/releases/pihole/volumes/ix_volumes/dnsmasq
Rapid/ix-applications/releases/plex                                           1.30M  9.66T   151K  /mnt/Rapid/ix-applications/releases/plex
Rapid/ix-applications/releases/plex/charts                                     901K  9.66T   808K  /mnt/Rapid/ix-applications/releases/plex/charts
Rapid/ix-applications/releases/plex/volumes                                    279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/plex/volumes
Rapid/ix-applications/releases/plex/volumes/ix_volumes                         140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/plex/volumes/ix_volumes
Rapid/ix-applications/releases/super-mario-odyssey-online                      709K  9.66T   151K  /mnt/Rapid/ix-applications/releases/super-mario-odyssey-online
Rapid/ix-applications/releases/super-mario-odyssey-online/charts               279K  9.66T   279K  /mnt/Rapid/ix-applications/releases/super-mario-odyssey-online/charts
Rapid/ix-applications/releases/super-mario-odyssey-online/volumes              279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/super-mario-odyssey-online/volumes
Rapid/ix-applications/releases/super-mario-odyssey-online/volumes/ix_volumes   140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/super-mario-odyssey-online/volumes/ix_volumes
Rapid/ix-applications/releases/syncthing                                      1.26M  9.66T   140K  /mnt/Rapid/ix-applications/releases/syncthing
Rapid/ix-applications/releases/syncthing/charts                                872K  9.66T   756K  /mnt/Rapid/ix-applications/releases/syncthing/charts
Rapid/ix-applications/releases/syncthing/volumes                               279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/syncthing/volumes
Rapid/ix-applications/releases/syncthing/volumes/ix_volumes                    140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/syncthing/volumes/ix_volumes
Rapid/ix-applications/releases/tautulli                                       1.92M  9.66T   151K  /mnt/Rapid/ix-applications/releases/tautulli
Rapid/ix-applications/releases/tautulli/charts                                1.27M  9.66T  1.18M  /mnt/Rapid/ix-applications/releases/tautulli/charts
Rapid/ix-applications/releases/tautulli/volumes                                512K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tautulli/volumes
Rapid/ix-applications/releases/tautulli/volumes/ix_volumes                     372K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tautulli/volumes/ix_volumes
Rapid/ix-applications/releases/tautulli/volumes/ix_volumes/config              140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tautulli/volumes/ix_volumes/config
Rapid/ix-applications/releases/tdarr                                           698K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tdarr
Rapid/ix-applications/releases/tdarr/charts                                    279K  9.66T   279K  /mnt/Rapid/ix-applications/releases/tdarr/charts
Rapid/ix-applications/releases/tdarr/volumes                                   279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tdarr/volumes
Rapid/ix-applications/releases/tdarr/volumes/ix_volumes                        140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/tdarr/volumes/ix_volumes
Rapid/ix-applications/releases/vaultwarden                                     924K  9.66T   151K  /mnt/Rapid/ix-applications/releases/vaultwarden
Rapid/ix-applications/releases/vaultwarden/charts                              494K  9.66T   494K  /mnt/Rapid/ix-applications/releases/vaultwarden/charts
Rapid/ix-applications/releases/vaultwarden/volumes                             279K  9.66T   140K  /mnt/Rapid/ix-applications/releases/vaultwarden/volumes
Rapid/ix-applications/releases/vaultwarden/volumes/ix_volumes                  140K  9.66T   140K  /mnt/Rapid/ix-applications/releases/vaultwarden/volumes/ix_volumes
Rapid/nextcloudtemp                                                            140K  9.66T   140K  /mnt/Rapid/nextcloudtemp
Rapid/os-images                                                               4.66G  9.66T  4.66G  /mnt/Rapid/os-images
Rapid/remote-transfer                                                         51.9G  9.66T   151K  /mnt/Rapid/remote-transfer
Rapid/remote-transfer/inbox                                                    523K  9.66T   140K  /mnt/Rapid/remote-transfer/inbox
Rapid/remote-transfer/inbox/gigakisten-seed                                    151K  9.66T   140K  /mnt/Rapid/remote-transfer/inbox/gigakisten-seed
Rapid/remote-transfer/inbox/home-rudnev                                        140K  9.66T   140K  /mnt/Rapid/remote-transfer/inbox/home-rudnev
Rapid/remote-transfer/outbox                                                   546K  9.66T   151K  /mnt/Rapid/remote-transfer/outbox
Rapid/remote-transfer/outbox/gigakisten-seed                                   151K  9.66T   140K  /mnt/Rapid/remote-transfer/outbox/gigakisten-seed
Rapid/remote-transfer/outbox/home-rudnev                                       151K  9.66T   140K  /mnt/Rapid/remote-transfer/outbox/home-rudnev
Rapid/remote-transfer/shared                                                  51.9G  9.66T  51.9G  /mnt/Rapid/remote-transfer/shared
Rapid/remote-transfer/synced                                                   151K  9.66T   140K  /mnt/Rapid/remote-transfer/synced
Rapid/scripts                                                                 12.1M  9.66T  12.1M  /mnt/Rapid/scripts
Rapid/storage                                                                  641G  9.66T   186K  /mnt/Rapid/storage
Rapid/storage/file-sharing                                                     140K  9.66T   140K  /mnt/Rapid/storage/file-sharing
Rapid/storage/local                                                            180G  9.66T   151K  /mnt/Rapid/storage/local
Rapid/storage/local/encrypted                                                 1.39M  9.66T   395K  /mnt/Rapid/storage/local/encrypted
Rapid/storage/local/encrypted/darkn                                            308K  9.66T   308K  /mnt/Rapid/storage/local/encrypted/darkn
Rapid/storage/local/encrypted/jennifer                                         256K  9.66T   256K  /mnt/Rapid/storage/local/encrypted/jennifer
Rapid/storage/local/encrypted/shared                                           459K  9.66T   459K  /mnt/Rapid/storage/local/encrypted/shared
Rapid/storage/local/longterm                                                   152G  9.66T   140K  /mnt/Rapid/storage/local/longterm
Rapid/storage/local/longterm/darkn                                             152G  9.66T   152G  /mnt/Rapid/storage/local/longterm/darkn
Rapid/storage/local/longterm/jennifer                                          140K  9.66T   140K  /mnt/Rapid/storage/local/longterm/jennifer
Rapid/storage/local/longterm/shared                                            162M  9.66T   133M  /mnt/Rapid/storage/local/longterm/shared
Rapid/storage/local/multipurpose                                              17.9G  9.66T  17.8G  /mnt/Rapid/storage/local/multipurpose
Rapid/storage/local/multipurpose/darkn                                         144M  9.66T   144M  /mnt/Rapid/storage/local/multipurpose/darkn
Rapid/storage/local/volatile                                                  9.36G  9.66T  9.36G  /mnt/Rapid/storage/local/volatile
Rapid/storage/media                                                            461G  9.66T   163K  /mnt/Rapid/storage/media
Rapid/storage/media-transcoding                                                133M  9.66T   140K  /mnt/Rapid/storage/media-transcoding
Rapid/storage/media-transcoding/done                                           133M  9.66T   133M  /mnt/Rapid/storage/media-transcoding/done
Rapid/storage/media-transcoding/open                                           186K  9.66T   174K  /mnt/Rapid/storage/media-transcoding/open
Rapid/storage/media-transcoding/tdarr                                          186K  9.66T   174K  /mnt/Rapid/storage/media-transcoding/tdarr
Rapid/storage/media/gigakisten-seed                                            663K  9.66T   151K  /mnt/Rapid/storage/media/gigakisten-seed
Rapid/storage/media/gigakisten-seed/content                                    209K  9.66T   174K  /mnt/Rapid/storage/media/gigakisten-seed/content
Rapid/storage/media/gigakisten-seed/libraries                                  174K  9.66T   174K  /mnt/Rapid/storage/media/gigakisten-seed/libraries
Rapid/storage/media/not-synced                                                 452G  9.66T   140K  /mnt/Rapid/storage/media/not-synced
Rapid/storage/media/not-synced/content                                         452G  9.66T   452G  /mnt/Rapid/storage/media/not-synced/content
Rapid/storage/media/not-synced/libraries                                       244K  9.66T   244K  /mnt/Rapid/storage/media/not-synced/libraries
Rapid/storage/media/uni-videos                                                8.60G  9.66T  3.58G  /mnt/Rapid/storage/media/uni-videos
Rapid/storage/nextcloud-data                                                   140K  9.66T   140K  /mnt/Rapid/storage/nextcloud-data
Rapid/vm-storage                                                               814K  9.66T   151K  /mnt/Rapid/vm-storage
Rapid/vm-storage/data                                                          523K  9.66T   151K  /mnt/Rapid/vm-storage/data
Rapid/vm-storage/data/dockervm                                                 140K  9.66T   140K  /mnt/Rapid/vm-storage/data/dockervm
Rapid/vm-storage/data/windowsvm                                                140K  9.66T   140K  /mnt/Rapid/vm-storage/data/windowsvm
Rapid/vm-storage/volumes                                                       140K  9.66T   140K  /mnt/Rapid/vm-storage/volumes
Rapid/webroot                                                                 2.45G  9.66T   140K  /mnt/Rapid/webroot
Rapid/webroot/default                                                         2.44G  9.66T  2.44G  /mnt/Rapid/webroot/default
boot-pool                                                                     5.00G  87.0G    96K  none
boot-pool/.system                                                             3.43M  87.0G   120K  legacy
boot-pool/.system/configs-f1f5036a6e4448d09a9ddb3c45165866                      96K  87.0G    96K  legacy
boot-pool/.system/cores                                                         96K  1024M    96K  legacy
boot-pool/.system/ctdb_shared_vol                                               96K  87.0G    96K  legacy
boot-pool/.system/glusterd                                                      96K  87.0G    96K  legacy
boot-pool/.system/netdata-f1f5036a6e4448d09a9ddb3c45165866                    2.32M  87.0G  2.32M  legacy
boot-pool/.system/rrd-f1f5036a6e4448d09a9ddb3c45165866                          96K  87.0G    96K  legacy
boot-pool/.system/samba4                                                       340K  87.0G   224K  legacy
boot-pool/.system/services                                                      96K  87.0G    96K  legacy
boot-pool/.system/webui                                                         96K  87.0G    96K  legacy
boot-pool/ROOT                                                                4.98G  87.0G    96K  none
boot-pool/ROOT/22.12.4.2                                                      2.65G  87.0G  2.65G  legacy
boot-pool/ROOT/23.10.1.3                                                      2.33G  87.0G  2.33G  legacy
boot-pool/grub                                                                8.20M  87.0G  8.20M  legacy

Here are my system specs:

AMD Ryzen 5 PRO 4650G with Radeon Graphics
64 GB Unregistered ECC DDR4-3000
4x 4TB Crucial P3 Plus SSD
1x 500 GB Samsung 970 EVO SSD

If there is anything else that could help to debug this, please tell me. I will also file a bug ticket (?). For the time I will try to rollback to Bluefin and hopefully get back up and running.


Thanks, Jan

Just as I posted this, I noticed that I forgot to mention that there is a ZFS upgrade available on the pool:

I will not attempt this until some more info on this issue is available, since I assume this to break compatibility with Bluefin.


Thanks again, Jan

 

[janerjak]

After checking the details of the error again, I don't think this has anything to do with my datasets. My bad. Here is the full error in text form for reference:

Error importing pool​

Code:

(sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name [SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)] [parameters: ('Rapid', '2271247162968676926')] (Background on this error at: https://sqlalche.me/e/14/gkpj)

Code:

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: storage_volume.vol_name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body
    rv = await self.method(*([self] + args))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 186, in import_pool
    pool_id = await self.middleware.call('datastore.insert', 'storage.volume', {
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/write.py", line 62, in insert
    result = await self.middleware.call(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/connection.py", line 106, in execute_write
    result = self.connection.execute(sql, binds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1365, in execute
    return self._exec_driver_sql(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1669, in _exec_driver_sql
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: storage_volume.vol_name
[SQL: INSERT INTO storage_volume (vol_name, vol_guid) VALUES (?, ?)]
[parameters: ('Rapid', '2271247162968676926')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

Best regards, Jan

 

[probain]

Just as I posted this, I noticed that I forgot to mention that there is a ZFS upgrade available on the pool:
I will not attempt this until some more info on this issue is available, since I assume this to break compatibility with Bluefin.

Thanks again, Jan

This is correct reasoning. Do *not* upgrade your pools to a newer ZFS version until you are 100(0000)% sure that things run stable and without issues. You won't be able to go back to previous version of TN if you do. And there is no danger running "old" ZFS versions.

 

[janerjak]

This is correct reasoning. Do *not* upgrade your pools to a newer ZFS version until you are 100(0000)% sure that things run stable and without issues. You won't be able to go back to previous version of TN if you do. And there is no danger running "old" ZFS versions.

Thanks for the reassurance about not upgrading the ZFS version!

I have multiple issues with TrueNAS Bluefin as it is and was kind of excited for the new app changes and upcoming features in Dragonfish. So upgrading would be pretty cool.

What I am currently struggling with for instance: I can not get the official Nextcloud app to run using a custom host path as the user storage. The TrueCharts app requires me to upgrade TrueNAS to Cobia because of missing dependencies.

There is other stuff that I hoped would maybe be resolved by upgrading as well.

Just to be sure: You don't expect upgrading the ZFS version to resolve the upgrade issues to Cobia, right?

 

[probain]

Just to be sure: You don't expect upgrading the ZFS version to resolve the upgrade issues to Cobia, right?

I'm not proficient enough to give a definitive answer , unlike some really knowledgeable people on this forums. And as such, that's why I'm not giving tips on what you should do. However. From everything I've read. Upgrading to a newer ZFS-version has rarely (if ever) solved the issues that people tend to need help with. So my recomendation is to wait with upgrading your ZFS-version. Since you can always upgrade later. But you cannot rollback to a previous version.

I do hope that you get some good tips from those that really know the ins and outs of ZFS.

 

[janerjak]

I found two issues on Jira with the same underlying problem:

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

The latter has an engineering representive suggest exporting the pool through the WebUI, not deleting any configuration, then reimporting through the WebUI. That fixed it for me as well. I still don't know what the actual problem is and I sincerely hope this will still be investigated by the dev team.

Both issues were closed without the underlying problem fixed, so I've created a new one:

[NAS-127416] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Glad I can finally use the new app and jail system.
In hindsight - I really should've come up with that "fix" myself :D But I was a bit scared to break things further.

 

[70tas]

I hope it is okay to jump in here. I just performed a GUI update and had similar issues.
I am running on an HP EliteDesk 800 G5, i7-9700t, 64GB RAM, 512GB boot SSD.
My storage is a USB 3.1 Gen2 Terramaster D6-320, with six SEAGATE 16TB drives in a Z2 with a spare.

After the upgrade I was missing one drive and was running in reduced mode. I tried various things which I knew would make no difference, and they did not. I then updated again, an update was available, and when they system came up all VDEV's were gone.
I restored the configuration and rebooted, but that did not help. I rebooted, and noticed I had three OS entries... I chose the oldest, 23.10.0.1 and when it came up I re-restored my config. On reboot it came up without any VDEV's. I rebooted and selected 23.10.0.1, and it came up and started re-silvering.

 

[janerjak]

I'm happy to report that the devs prosumably fixed the problem and closed the issue I opened.
The root cause in my case seems to have been exporting and reimporting the pool using the zfs CLI only. This did not update UUIDs in the database causing two pools to be present with the same name on upgrade.

 

[70tas]

Thank you. I may have been caught with an older version, prior to the fix. Now I just need to fix my boot screen, and I should be good to go.

 

[janerjak]

Thank you. I may have been caught with an older version, prior to the fix. Now I just need to fix my boot screen, and I should be good to go.

Nice, good luck! :)