Cannot import one or more devices is currently unavilable

[AceRoehrborn]

While on vacation my Truenas went offline it had been working fine for years before this. Not sure if we lost power or what happened. I am a home hobbyist so nothing super critical here just trying to goof around and learn. I use the drive primarily to host my media for a plex server.

My System
I have Dell Server with a PERC H710 controller in it. I have two Raid 5 arrays in the server. 4 @500gb drives I am using for vmware and have TrueNas installed here as a virtual machine, the second is 4 2TB drives in Raid 5 at the PERC Controller level. This presents to TrueNas as a single drive.

All drives in the server are functioning and online. The 2b raid array is connected to TrueNas at the VM level.

Version:
TrueNAS-12.0-U5.1

Original Problem
I was unable to access the drive and was getting a device not available error. In trying to sort this out I disconnected the pool. I am now having trouble reconnecting it.


From Storage->Pools->Import pool I am able to select my pool "Data" however when importing the pool I receive an error "one or more devices is currently unavailable".

I have tried to import with zpool command with both the -m and -f modifiers and get the same error.

"zpool import" command shows my pool "Data" as online and the action is listed as this pool can be imported using its name or numeric identifier.

"zpool import Data" command shows the same one or more devices is currently unavailable.

I have also tried zpool import -m Data, and zpool import -f Data with similar results.

The questions:

1. Is this recoverable, can i get my data back out of the pool?
2. When TrueNas refers to Devices unavailable what "devices" is it referring to?
   1. Is there some step I need to do to make the "devices" available before I import the pool?
3. Any quick easy fixes or steps I am missing to import my pool?

More info button on the web UI import gives these detailed error results:

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 94, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool
libzfs.ZFSException: one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run
    await self.future
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 403, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1421, in import_pool
    await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1248, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1219, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1146, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1120, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('one or more devices is currently unavailable',)

 

[NugentS]

"This presents to TrueNas as a single drive"
Welcome to how NOT to run ZFS. The H710 needs to be flashed to IT Mode (assuming it can be) or replaced with an HBA and the pool completely rebuilt.

Whats All The Noise About HBAs and Why Can't I Use a RAID Controller - Read this

 

[sretalla]

Is this recoverable, can i get my data back out of the pool?

Probably not without help from a data recovery professional or using software like Klennet.

When TrueNas refers to Devices unavailable what "devices" is it referring to?

1. Is there some step I need to do to make the "devices" available before I import the pool?

The only "device" in the pool is likely to be your RAID array, which contains some kind of non-correctable corruption (something that ZFS guards against if you don't put a RAID controller between it and the disks).

You're probably unable to do anything other than running Klennet (paid recovery software that runs on Windows) or sending it out to a recovery service.

Any quick easy fixes or steps I am missing to import my pool?

Sorry, but no... unless you have a time machine and can read the RAID controller/HBA advice before building your pool.

 

[Arwen]

To clarify, one of the biggest issues with hardware RAID controllers, is that if bit rot occurs and the controller or LUN was not setup for patrol reads, (aka parity verification), then you loose data on disk failure.

This happened to me once in 2000 or so. Had an external HW RAID disk array loose a disk. It was a RAID-5, and not using that large of disks by today's standards, (9GB?). (I don't remember how many disks were in that RAID-5 set.) But, in the process of replacing the failed disk, we found corruption on another disk, (aka bit rot). Thus, the entire RAID-5 LUN was lost and had to be restored from backups.

Later the vendor of the HW RAID disk array came out with a firmware update that included patrol reads.


But, even with patrol reads, unless the "bit rot" disk block is reported as a problem by the disk's CRC / Checksum, RAID-5 parity is useless to recover the data. This is because you don't know which of the member disks returned garbage. The parity comes out as wrong, but who is wrong? The parity block? One of the data blocks?

ZFS does a better job of this by having a Merkle tree, (checksums all the way up the file system). With the RAID-Zx parity being stored above the data blocks, also parity protected.


Back to the original poster's problem. @AceRoehrborn, if you can initiate a patrol read or what ever the controller calls it, it may be possible that the hardware RAID-5 controller can find the problem and correct it. It is a long shot. Good luck.

 

[Whattteva]

Obligatory Works well till it doesn't post. I even got the post count right!!! And @jgreco even added the bit about VMWare the next post after, which you are also using.

 

[AceRoehrborn]

Thank you all for this good info. I clearly made an oops when setting this up behind the raid controller as I did. I have a Windows VM on my server and was able to connect the datastore over to the Windows side and am running a Klennet scan now. We will see how that turns out and if I am able to recover any of my data.

Flashing the raid controller to IT Mode probably is not ideal for my setup as I have some other stuff going on with other drives in the system I would like to keep in hardware raid. Let's say I wanted to keep my server and raid controller setup as it currently is but still have a network storage function happening. It would seem TrueNas is probably the wrong software since it does not like the Raid controller in the middle.

I could probably just share this out via a Windows VM, but was looking for something more like true nas that was a little more of a dedicated and less prone to Windows updates, random restarts etc...

Is there another "NAS" type software that would run well inside esxi and work with my hardware raid configuration and share my media files out to my network?

 

[Arwen]

...
It would seem TrueNas is probably the wrong software since it does not like the Raid controller in the middle.
...

Yes, TrueNAS, (both Core & SCALE), is the wrong solution when using hardware RAID. But, it is more ZFS, which TrueNAS uses that is the reason hardware RAID is not advisable.


To clarify, ZFS does all 3 of RAID, Volume Manager, & File System. Using hardware RAID removes ZFS' RAID functions and the ability to auto-correct data blocks. Their are 2 copies of metadata blocks by default, thus is may be possible under hardware RAID to repair 1 failing metadata block. Except that hardware RAID may very well store both copies of metadata blocks on the same physical disk :-(.

 

[Whattteva]

Is there another "NAS" type software that would run well inside esxi and work with my hardware raid configuration and share my media files out to my network?

Not exactly a "NAS", but you can just use vanilla FreeBSD + UFS or any Linux Distro and set it up manually.

Obviously the downside with that is no nice little Web UI and it's more manual.

 

[sretalla]

Open Media Vault (OMV) is probably one candidate.

 

[jgreco]

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com