Lost a pool --cannot import

[superfour]

One server (older Quad core CPU + 8GB) with two pools (3 + 5 disks respectively), TrueNAS 12. Runs smoothly for ~2 years now (started from FreeNAS).

I fired up the server this morning, and <Array1> is not there. Dashboard said "Pool status OFFLINE" with "Data not available". <Array2> seemed to be OK (on the dashboard), although I couldn't connect from my windows machine. Notifications was like: "CRITICAL Pool Array1 state is OFFLINE: None"

Disks page showed all the existing physical drives properly. Lost pool is an array of 3 x 1TB WD drives.

Couldn't do anything for Array1, so I disconnected it, and tried to reconnect. The add-existing-pool page did find the missing Array, and I clicked NEXT, but then the system stayed forever on the "Importing array..." screen.

I reloaded the server page --required login again. Menus/pages seem to work, EXCEPT for the dashboard, which never loads. All other menus/pages load properly.

I tried to turn it off with the poweroff command from the shell. However, the machine did NOT power down, nor restarted. So a hard reset followed. (And afterwards started normally). But I am now back to the start: Array1 showing offline, with the option to EXPORT/DISCONNECT. So there I am now; won't try again disconnecting-importing without asking you guys first!

Thank you in advance.

 

[winnielinnie]

From an SSH connection or the Shell itself, what is the output for,
zpool status -v

 

[superfour]

From an SSH connection or the Shell itself, what is the output for,
zpool status -v

root@truenas[~]# zpool status -v
pool: Array2
state: ONLINE
scan: resilvered 1.34M in 00:00:01 with 0 errors on Tue Aug 3 09:35:13 2021
config:

NAME STATE READ WRITE CKSUM
Array2 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/b102bf2d-f1ed-11eb-9bb8-001fd0a402b8 ONLINE 0 0 0
gptid/b13c3bb1-f1ed-11eb-9bb8-001fd0a402b8 ONLINE 0 0 0
gptid/b142d5cb-f1ed-11eb-9bb8-001fd0a402b8 ONLINE 0 0 0
cache
gptid/8c960511-f437-11eb-b69e-001fd0a402b8 ONLINE 0 0 0
spares
gptid/7efb236a-f398-11eb-bb52-001fd0a402b8 AVAIL

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:35 with 0 errors on Thu Oct 28 03:45:35 2021

Array1 seems to be missing from this output...

 

[winnielinnie]

Not a fan of using the command-line in an appliance, but have you tried using the Shell or an SSH session to manually import it?

To list available pools:
zpool import

Followed by:
zpool import Array1

Does it spit out any warnings or errors?

There are other flags you can use, namely -F and -n but I would hold off on them for now.

zpool-import.8 — OpenZFS documentation

openzfs.github.io

 

[jgreco]

One server (older Quad core CPU + 8GB) with two pools (3 + 5 disks respectively), TrueNAS 12. Runs smoothly for ~2 years now (started from FreeNAS).

Welcome to the forums. Sorry you're having trouble.

Please take a few moments to post a summary of your hardware, as outlined in the Forum Rules, conveniently linked at the top of every page in red. Details such as how your drives are connected often turn out to play a major role in these sorts of problems. For example, this is reminiscent of a variety of RAID-controller-gone-wrong types of errors, as discussed in

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

This will help people start solving the problem from the top down. Without this information, there's more random guesswork about what could be a problem, while often missing a problem that would have been obvious from the hardware manifest.

 

[superfour]

I appreciate your help guys.

Hardware is an older Gigabyte mobo EP45-UD3R and Intel Q9560 proc, with an extra PCIx SATA Marvell controller and 8GB RAM. Running TrueNAS-12.0-U4.1. There are 9 disks onboard, a 64GB SSD SanDisk for system, 3x 1TB WD HDDs for Array1, 4x 1TB Seagate HDDs for Array2 (plus an old small WD Velociraptor HDD for Array2 cache). All 9 disks show correctly on Storage/Disks page.

BTW, since yesterday Array2 is back online for windows/samba users, working properly. Was ok on dashboard as I said, but I could not connect via smb.

One more notable thing I just discovered: I cannot connect via SSH from a windows machine (PuTTY). It refuses my root credentials.

zpool import shows the missing array! (But ONLINE?!)

...at the same time:

That sums it up. I'm guessing there's some magical command-line trick to do the job :)

 

[jgreco]

So if you manually imported the array, the middleware may be blissfully unaware of the fact. I don't think it sits there polling for it, so, my suspicion is that you could try the export/disconnect and then import it via the GUI, or maybe just try a reboot if you feel comfortable with that. Do NOT power cycle the machine, just in case it is a failing drive of some sort. Sometimes they don't spin back up ...

My theory is that there was some problem that prevented it from importing the pool that has since cleared. The middleware saw it as "offline". You managed to manually tease it online, but no one told the middleware. That would leave you in this broken state.

I'm torn because I want to go test this theory on a test host, but I do not have time right now. Maybe hold off a few hours and see if I find any of my stock of round tuits.

 

[superfour]

Thank you jgreco. I am not in a hurry; the poll is part part of a triple backup scheme of mine, so no hurry here. I can try again to disconnect and import the array. I did it from the GUI the first time, so I'll just do the same steps. If you want me to wait, that's no problem.

 

[superfour]

So, once again, I disconnected the missing pool, and tried to import it.

Step 3 found the array (with a long number attached to it) and afterwards I got this error

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 94, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
self.logger.error(
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool
zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool
File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool
libzfs.ZFSException: one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 403, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1411, in import_pool
await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1241, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1206, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1212, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1139, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1113, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('one or more devices is currently unavailable',)

 

[superfour]

I tried again, disconnecting the importing the Array. Same results.
Furthermore, when I open another browser window to access the system, I can choose to shut it down, but it will not; power stays on.

 

[superfour]

So, would anyone suggest how can I salvage that pool?
Build perhaps another Truenas system and import it there?

 

[Patrick M. Hausen]

One more notable thing I just discovered: I cannot connect via SSH from a windows machine (PuTTY). It refuses my root credentials.

Root login must be explicitly allowed in the SSH service settings.

Then you could proceed with

Code:

zpool import -o altroot=/mnt <pool>
zpool status -v

 

[superfour]

Then you could proceed with

Code:

zpool import -o altroot=/mnt <pool>
zpool status -v

Thank you Patrick. Commands were executed with no errors. However...

Code:

root@truenas[~]# zpool import
   pool: Array1
     id: 3846291834767033103
  state: ONLINE
status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        Array1                                        ONLINE
          gptid/5ac04b0d-a018-11ea-8e9d-001fd0a402b8  ONLINE
          gptid/5adeee85-a018-11ea-8e9d-001fd0a402b8  ONLINE
          gptid/5b379a0d-a018-11ea-8e9d-001fd0a402b8  ONLINE
root@truenas[~]# zpool import -o altroot=/mnt Array1
root@truenas[~]# zpool status -v
  pool: Array1
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 0B in 00:00:01 with 0 errors on Fri Nov 12 16:50:54 2021
config:

        NAME                                          STATE     READ WRITE CKSUM
        Array1                                        ONLINE       0     0     0
          gptid/5ac04b0d-a018-11ea-8e9d-001fd0a402b8  ONLINE       0     0     0
          gptid/5adeee85-a018-11ea-8e9d-001fd0a402b8  ONLINE       0     0     0
          gptid/5b379a0d-a018-11ea-8e9d-001fd0a402b8  ONLINE       0     0     0

errors: No known data errors

and GUI does not display the pool. Not in the dashboard, nor if try to import an existing pool.
But the disk page displays the HDDs of the array properly (ie belonging to Array1).

 

[superfour]

So, back to my problem again...

I set up another TrueNAS system, which imported the array with no problem. After that, I moved the array back to the first system, which then was imported(!) with no problems. After a few days, the array got lost again (still, on the first system). Trying to import it will give me:

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 94, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool
libzfs.ZFSException: one or more devices is currently unavailable

All disks are working & accounted for in the disks page. I'm still open to ideas...