Can't import pool I/O error

[jackelofnar]

Hi,
I'm trying to import pool via 'zpool import Hydra' but I keep getting this error
"cannot import 'Hydra': I/O error
Destroy and re-create the pool from
a backup source."

from what I can see my 6 * 8TB drives look OK (See screenshot)
Is there a way I can force it to mount? even if it readonly mode it will give me a chance to compare against my latest backup.

LBA: Dell H200 (LSI 9211-8i)
TrueNas Version: 12.0-U8.1

I have read other threads on this with no luck.

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
self.logger.error(
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool
zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool
File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool
libzfs.ZFSException: I/O error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 403, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1421, in import_pool
await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1221, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1227, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1154, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('I/O error',)

 

[Nick2253]

Have you tried forcing it? # zpool import -f hydra

Importing ZFS Storage Pools - Managing ZFS File Systems in Oracle® Solaris 11.2

After a pool has been identified for import, you can import it by specifying the name of the pool or its numeric identifier as an argument to the zpool import command.

docs.oracle.com

 

[jackelofnar]

Have you tried forcing it? # zpool import -f hydra

Importing ZFS Storage Pools - Managing ZFS File Systems in Oracle® Solaris 11.2

After a pool has been identified for import, you can import it by specifying the name of the pool or its numeric identifier as an argument to the zpool import command.

docs.oracle.com

Yes but still get the I/O error

 

[Samuel Tai]

If your pool is already imported, you can't import it again. If it appears in zpool status -v Hydra as ONLINE, then it's already imported. If it's not imported, you'll get a different error message.

 

[jackelofnar]

If your pool is already imported, you can't import it again. If it appears in zpool status -v Hydra as ONLINE, then it's already imported. If it's not imported, you'll get a different error message.

These are the messages I get

 

[Samuel Tai]

OK, try zpool import -f -F -R /mnt Hydra.

 

[jackelofnar]

OK, try zpool import -f -F -R /mnt Hydra.

Still get the "cannot import 'Hydra': I/O error" error

 

[Samuel Tai]

How did you export this pool? Did you manually export this pool via the GUI, or was this an automatic export as part of shutting down?

 

[jackelofnar]

How did you export this pool? Did you manually export this pool via the GUI, or was this an automatic export as part of shutting down?

Manually exported via the gui

 

[Samuel Tai]

Manually exported via the gui

I was afraid of that. Did you uncheck the boxes labeled "Destroy data on this pool?" and "Delete configuration of shares that used this pool?" before confirming the export?

If you didn't, then we'll need a stronger import technique: zpool import -f -F -D -R /mnt Hydra.

 

[jackelofnar]

I was afraid of that. Did you uncheck the boxes labeled "Destroy data on this pool?" and "Delete configuration of shares that used this pool?" before confirming the export?

If you didn't, then we'll need a stronger import technique: zpool import -f -F -D -R /mnt Hydra.

I did unselect "Destroy data on this pool" but selected "Delete configuration of shares that used this pool"
Have i lost it all?

 

[Samuel Tai]

I did unselect "Destroy data on this pool" but selected "Delete configuration of shares that used this pool"
Have i lost it all?

Maybe not. I was afraid you'd not unselected the first. The second is only supposed to affect the shares defined on that pool. My theory is the UI may have misinterpreted your selection behind the scenes. Please try that import with the -D option, which will try to undelete pool data.

 

[jackelofnar]

Maybe not. I was afraid you'd not unselected the first. The second is only supposed to affect the shares defined on that pool. My theory is the UI may have misinterpreted your selection behind the scenes. Please try that import with the -D option, which will try to undelete pool data.

With "zpool import -f -F -D -R /mnt Hydra"
cannot impoty 'Hydra': no such pool available

 

[Samuel Tai]

What if you try with the pool ID instead of the pool name? zpool import -f -F -D -R /mnt 17354336885520832522

 

[jackelofnar]

What if you try with the pool ID instead of the pool name? zpool import -f -F -D -R /mnt 17354336885520832522

cannot import '17354336885520832522': no such pool available

 

[Samuel Tai]

OK, so zpool import doesn't see this pool as deleted, so that rules out that failure mode. Do you have SMART data from all the drives in the pool?

 

[jackelofnar]

OK, so zpool import doesn't see this pool as deleted, so that rules out that failure mode. Do you have SMART data from all the drives in the pool?

I am not sure how do I find out

 

[Samuel Tai]

Please provide smartctl -a /dev/dax for all the disks in your pool. (Note the x is a placeholder for 0, 1, 2, etc.) Also, glabel status to match up each drive's GPTID with the physical port.

I'm beginning to think there's possibly a bad drive in the pool. If we can identify which one it is, and disconnect it, your pool may import, but in a degraded state, but you'd be able to access your files.

 

[jackelofnar]

Please provide smartctl -a /dev/dax for all the disks in your pool. (Note the x is a placeholder for 0, 1, 2, etc.) Also, glabel status to match up each drive's GPTID with the physical port.

I'm beginning to think there's possibly a bad drive in the pool. If we can identify which one it is, and disconnect it, your pool may import, but in a degraded state, but you'd be able to access your files.

I'm guessing this is what you are looking for

Code:

smartctl -a /dev/da3
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p14 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar He10/12
Device Model:     WDC WD80EMAZ-00WJTA0
Serial Number:    7HK9PREF
LU WWN Device Id: 5 000cca 257eeaf66
Firmware Version: 83.H0A83
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jun  7 22:26:29 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (   93) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (1024) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   128   128   054    Old_age   Offline      -       116
  3 Spin_Up_Time            0x0007   163   163   024    Pre-fail  Always       -       401 (Average 407)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       2593
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       19006
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       264
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   096   096   000    Old_age   Always       -       4807
193 Load_Cycle_Count        0x0012   096   096   000    Old_age   Always       -       4807
194 Temperature_Celsius     0x0002   141   141   000    Old_age   Always       -       46 (Min/Max 11/56)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       226

SMART Error Log Version: 1
ATA Error Count: 226 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 226 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 48 00 40 28 81 40 00      00:09:54.438  READ FPDMA QUEUED
  60 78 10 c8 29 81 40 00      00:09:54.438  READ FPDMA QUEUED
  60 30 08 90 29 81 40 00      00:09:54.438  READ FPDMA QUEUED
  60 01 38 3f 28 81 40 00      00:09:54.438  READ FPDMA QUEUED
  60 01 30 3e 28 81 40 00      00:09:54.438  READ FPDMA QUEUED

Error 225 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 40 26 81 40 00      00:09:54.419  READ FPDMA QUEUED
  60 01 38 3f 04 00 40 00      00:09:54.389  READ FPDMA QUEUED
  60 01 30 3e 04 00 40 00      00:09:54.389  READ FPDMA QUEUED
  60 01 28 3d 04 00 40 00      00:09:54.389  READ FPDMA QUEUED
  60 01 20 3c 04 00 40 00      00:09:54.389  READ FPDMA QUEUED

Error 224 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 01 30 ce 02 00 40 00      00:09:54.363  READ FPDMA QUEUED
  60 01 38 cf 02 00 40 00      00:09:54.363  READ FPDMA QUEUED
  60 01 28 cd 02 00 40 00      00:09:54.363  READ FPDMA QUEUED
  60 01 20 cc 02 00 40 00      00:09:54.363  READ FPDMA QUEUED
  60 01 18 cb 02 00 40 00      00:09:54.363  READ FPDMA QUEUED

Error 223 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 f8 00 48 02 00 40 00      00:09:54.356  READ FPDMA QUEUED
  60 01 38 3f 02 00 40 00      00:09:54.355  READ FPDMA QUEUED
  60 01 30 3e 02 00 40 00      00:09:54.355  READ FPDMA QUEUED
  60 01 28 3d 02 00 40 00      00:09:54.355  READ FPDMA QUEUED
  60 01 20 3c 02 00 40 00      00:09:54.355  READ FPDMA QUEUED

Error 222 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 10 08 68 00 00 40 00      00:09:54.334  READ FPDMA QUEUED
  60 f8 20 48 01 00 40 00      00:09:54.334  READ FPDMA QUEUED
  60 78 18 c8 00 00 40 00      00:09:54.334  READ FPDMA QUEUED
  60 30 10 88 00 00 40 00      00:09:54.334  READ FPDMA QUEUED
  60 08 00 50 00 00 40 00      00:09:54.334  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay

 

[Samuel Tai]

Yup, it looks like da3 is the culprit for the I/O errors on import. Are there any other drives with smartctl errors? Where does this drive fit within your pool?