SOLVED Cannot import pool - Permission Denied

A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Hello all,

I am at a loss for what has happened to a pool that i've had running for several years without issue. The pool is a 4TB mirror with 1 of the 2 drives presently failed and physically removed (and has been for quite some months).

Anyway, I recently upgraded to TrueNAS from FreeNAS mostly successfully. After the upgrade, both of my pools came up. One of the pools (not the one I am having issues with now) had a disk IO problem that resolved after I replaced a SATA cable.

After a couple of other reboots, I noticed that my 4TB pool was showing offline in the GUI. The only option on the GUI was to Export/Disconnect. Before doing this, I tried importing the pool on the CLI. and can see that it is showing ONLINE (and still is):

Code:
root@sol:~ # zpool import
   pool: vol_Data-1
     id: 16557953576057239499
  state: ONLINE
status: One or more devices contains corrupted data.
 action: The pool can be imported using its name or numeric identifier.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
 config:

    vol_Data-1                                      ONLINE
      mirror-0                                      ONLINE
        gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78  ONLINE
        2827245120054948424                         UNAVAIL  corrupted data
root@sol:~ #


However, when I try to import it with options it fails:
root@sol:~ # zpool import vol_Data-1 cannot import 'vol_Data-1': one or more devices is currently unavailable root@sol:~ #

After some trying of many things, I exported the pool from the GUI and tried re-adding via CLI with some success:
Code:
oot@sol:~ # zpool import -F vol_Data-1
root@sol:~ # zpool status
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:03:26 with 0 errors on Thu Apr 29 03:48:26 2021
config:

    NAME          STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2       ONLINE       0     0     0

errors: No known data errors

  pool: vol_Data-1
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
  scan: scrub repaired 0B in 21:38:21 with 0 errors on Sun Mar 28 21:38:41 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    vol_Data-1                                      DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78  ONLINE       0     0     0
        2827245120054948424                         UNAVAIL      0     0     0  was /dev/ada1

errors: No known data errors

  pool: vol_Data-2
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: resilvered 204K in 00:00:01 with 0 errors on Wed Apr 28 22:48:43 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    vol_Data-2                                      ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/fd9cff80-60e4-11ea-a188-00247e0a3e78  ONLINE       0     0     0
        gptid/fe615845-60e4-11ea-a188-00247e0a3e78  ONLINE       0     0     0
        gptid/03261724-60e5-11ea-a188-00247e0a3e78  ONLINE       0     0     0

errors: No known data errors
root@sol:~ #


Then, so I could check if everything came up again, I rebooted and I have not been able to bring back the pool since.

Now, whenever I try to add the pool through the GUI I get the following error:
Code:
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool
libzfs.ZFSException: permission denied
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run
    await self.future
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 1411, in import_pool
    await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('permission denied',)


OR the error "one or more devices is currently unavailable".

And through the CLI:
Code:
oot@sol:~ # zpool import -F vol_Data-1
cannot import 'vol_Data-1': one or more devices is currently unavailable
root@sol:~ # zpool import 
   pool: vol_Data-1
     id: 16557953576057239499
  state: ONLINE
status: One or more devices contains corrupted data.
 action: The pool can be imported using its name or numeric identifier.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
 config:

    vol_Data-1                                      ONLINE
      mirror-0                                      ONLINE
        gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78  ONLINE
        2827245120054948424                         UNAVAIL  corrupted data
root@sol:~ # zpool import vol_Data-1
cannot import 'vol_Data-1': permission denied
    Destroy and re-create the pool from
    a backup source.
root@sol:~ #


I think the problem is that zpool import sees the pool as ONLINE whereas it clearly isn't.

I have performed SMART long test and it completed without errors and there are no SMART errors logged, so I am pretty sure the disk is good.

Any ideas?

FYI, I am on the current version of TrueNAS-12.0-U3

I'm off to bed, but will check in the AM...

Thanks in advance.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Oh, I forgot to mention that my system dataset was on the pool I can't bring online. I've moved it to the working pool.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Ok, maybe a better way to get to the bottom of my problem... Where does the zpool command get or read the operating state of a pool or device. Is there a config database or file? The fact that zpool import sees the pool as ONLINE but zpool status or list does not even list the pool makes me think there is some sort of problem with my zfs config/state files or database.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
albyone said:
Where does the zpool command get or read the operating state of a pool or device
Click to expand...
From the kernel. The problem, whatever it is, is in ZFS itself. You need someone who knows zdb to dig deeper... This may or may not be recoverable.

The Python traceback sounds like noise caused by propagating the weird permission denied error.

albyone said:
and has been for quite some months
Click to expand...
Not to say that it's the cause of this or that it excuses anything, but if this is a semi-permanent thing, you can remove the device from the mirror.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Ericloewe said:
you can remove the device from the mirror.
Click to expand...
When I try to do this (at least I think this is what you mean) it says the pool doesn't exist:
Code:
root@sol:~ # zpool offline vol_Data-1 2827245120054948424
cannot open 'vol_Data-1': no such pool
root@sol:~ #


Also, trying to OFFLINE/ONLINE the pool fails:
Code:
root@sol:~ # zpool offline vol_Data-1 /dev/ada1
cannot open 'vol_Data-1': no such pool
root@sol:~ # zpool online vol_Data-1 /dev/ada1
cannot open 'vol_Data-1': no such pool
root@sol:~ #


So, it seems ONLY zpool import is able to see my pool...
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
albyone said:
When I try to do this (at least I think this is what you mean) it says the pool doesn't exist:
Click to expand...
I meant it more generally, as something that could've been done with a pool that is working normally. Although I meant zpool detach.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Yep, the detach command doesn't see the pool as available (even though zpool import does):
Code:
root@sol:~ # zpool import
   pool: vol_Data-1
     id: 16557953576057239499
  state: ONLINE
status: One or more devices contains corrupted data.
 action: The pool can be imported using its name or numeric identifier.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
 config:

    vol_Data-1                                      ONLINE
      mirror-0                                      ONLINE
        gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78  ONLINE
        2827245120054948424                         UNAVAIL  corrupted data
root@sol:~ # zpool detach vol_Data-1 2827245120054948424
cannot open 'vol_Data-1': no such pool
root@sol:~ #


Will see if I can import the pool on a live Ubuntu session.
 
Arwen

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
You show an import scan and then attempt to work on the pool. But, that did not do the actual import. You HAVE to use something like zpool import vol_Data-1 to actually import the pool.

However, if that does not work, you may be able to use the "-f" option to force an import. Whence the pool is imported, you can then detach the failed device.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Arwen said:
You show an import scan and then attempt to work on the pool. But, that did not do the actual import. You HAVE to use something like zpool import vol_Data-1 to actually import the pool.

However, if that does not work, you may be able to use the "-f" option to force an import. Whence the pool is imported, you can then detach the failed device.
Click to expand...
Thanks Arwen. I have tried those commands - you can see the output from them in my first post (although a -f isn't shown, I have tried that option too).

As an update, yes it does import in an Ubuntu live session (v20.04) so, details aren't kept in the kernel. After some looking around I found these comments stating that the details of the ZFS pool are stored on the disk/s itself/themselves.

Therefore, it seems TrueNAS might not be interrogating my disk correctly (or at all) for info on the pool. Rather it seems to be reading from some weird local cache - possibly /etc/zfs/zpool.cache - will do some poking around on this over the next couple of days...
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
The cache file... Yeah, I didn't think about it, but it would be able to throw zpool import for a loop.
Conceptually, nuking it makes sense, but I'm not 100% sure about the implications for booting.

One thing you can try is to run zpool import and give it the path to the disk. That might bypass the cache. The man page should have the details for how to invoke it that way.
 
A

albyone

Cadet
Joined
Apr 29, 2021
Messages
8
Problem is fixed.

After booting back into a Ubuntu Live session, I could once again see the pool and import it:
Code:
ubuntu@ubuntu:~$ sudo zpool import
   pool: vol_Data-1
     id: 16557953576057239499
  state: DEGRADED
 status: One or more devices contains corrupted data.
 action: The pool can be imported despite missing or damaged devices.  The
    fault tolerance of the pool may be compromised if imported.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
 config:

    vol_Data-1               DEGRADED
      mirror-0               DEGRADED
        sdd                  ONLINE
        2827245120054948424  UNAVAIL
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo zpool import 16557953576057239499
ubuntu@ubuntu:/media$ zpool status
  pool: vol_Data-1
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 0 days 21:38:21 with 0 errors on Sun Mar 28 10:38:41 2021
config:

    NAME                     STATE     READ WRITE CKSUM
    vol_Data-1               DEGRADED     0     0     0
      mirror-0               DEGRADED     0     0     0
        sdd                  ONLINE       0     0     0
        2827245120054948424  UNAVAIL      0     0     0  was /dev/ada1

errors: No known data errors
ubuntu@ubuntu:/media$


This is where I noticed what cold be causing an issue. The previously failed disk that was still associated with the pool was showing that it was connected to /dev/ada1. However, this was where the working disk was connected (see in outputs above). Therefore, the import on TrueNAS was likely trying to import a disk that didn't exist on a device that does exist.

I removed the device from the pool on ubuntu:
Code:
ubuntu@ubuntu:/media$ sudo zpool detach vol_Data-1 2827245120054948424
ubuntu@ubuntu:/media$
ubuntu@ubuntu:/media$ zpool status vol_Data-1
  pool: vol_Data-1
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 0 days 21:38:21 with 0 errors on Sun Mar 28 10:38:41 2021
config:

    NAME        STATE     READ WRITE CKSUM
    vol_Data-1  ONLINE       0     0     0
      sdd       ONLINE       0     0     0

errors: No known data errors
ubuntu@ubuntu:/media$
ubuntu@ubuntu:/media$ sudo zpool export vol_Data-1
ubuntu@ubuntu:/media$


Then imported in TrueNAS without any issues. And it survives a reboot:
Code:
root@sol:~ # zpool status vol_Data-1
  pool: vol_Data-1
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 21:38:21 with 0 errors on Sun Mar 28 21:38:41 2021
config:

    NAME                                          STATE     READ WRITE CKSUM
    vol_Data-1                                    ONLINE       0     0     0
      gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78  ONLINE       0     0     0

errors: No known data errors
root@sol:~ #


Thanks for your help.
 
You must log in or register to reply here.

Similar threads

C
SOLVED Disk Replacement - failed on TrueNAS-12.0-U3.1
Replies
1
Views
930
cherup
C
D
Cannot replace disks in pool
2
Replies
22
Views
7K
melbournemac
melbournemac
S
Possible iSCSI volume expansion
Replies
1
Views
668
sretalla
sretalla
M
Replication error. Bug?
Replies
1
Views
840
Inxsible
I
F
Cannot start VM after loading backup config file
Replies
3
Views
748
Fred974
F
Top