Replacing RAID0 DEGRADED disk

[MrRusch]

I have a simple FreeNAS 11.3-STABLE setup, with 2 physical 4 TB drives hosting two separate solutions. Backup of valuable data is managed with Borg, and sent to an offsite NAS which is properly RAID:ed.

The pool I am having issues with looks like this:

This drive has become degraded, and I bought a new drive to replace it.

This is the first time I replace a drive on FreeNAS, so I turned to the documentation, plugged in the new drive and used the GUI:s replacement function.
The replacement process seemed to go OK, and even though there's an error count, the silvering seemed to finish.

ada1p2 is the new drive, and ada2p2 is the old one.

But this is all that happens.. It doesn't automatically detach the old drive, and when I try to do so manually I get an error saying "Can detach disks from mirrors and spares only". Not quite sure where to go from here..

Issuing "zpool status -v CLOUD" gives:

Code:

  pool: CLOUD
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 691G in 0 days 02:25:19 with 53 errors on Thu Oct  8 21:33:14 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        CLOUD                                           DEGRADED     0     0    53
          replacing-0                                   DEGRADED     0     0   106
            gptid/3eff98dd-91c3-11e8-a5ef-7085c27a9a14  DEGRADED     0     0   106  too many errors
            gptid/735693a6-08cf-11eb-822d-7085c27a9a14  ONLINE       0     0   106

errors: Permanent errors have been detected in the following files:

        CLOUD/os:<0x1>

The file it's referring to is the OS volume for the VM I am using to host my cloud solution. The VM itself boots and works fine.

Issuing "smartctl -q noserial -a /dev/ada2" provides:

Code:

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate BarraCuda 3.5
Device Model:     ST4000DM004-2CV104
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Oct  8 22:22:10 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 489) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   077   048   006    Pre-fail  Always       -       52749585
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       53
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   094   060   045    Pre-fail  Always       -       2261310426
  9 Power_On_Hours          0x0032   079   079   000    Old_age   Always       -       18885 (97 164 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       53
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       160
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   057   040    Old_age   Always       -       33 (Min/Max 25/36)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       735
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       976
194 Temperature_Celsius     0x0022   033   043   000    Old_age   Always       -       33 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   077   064   000    Old_age   Always       -       52749585
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       18836h+55m+07.390s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       55110612196
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       33619752481

SMART Error Log Version: 1
ATA Error Count: 151 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 151 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.357  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.283  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.283  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.282  READ LOG EXT
  60 00 18 ff ff ff 4f 00      00:27:37.269  READ FPDMA QUEUED

Error 150 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.269  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.268  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.195  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.194  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.194  READ LOG EXT

Error 149 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.180  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.179  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.106  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.106  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.105  READ LOG EXT

Error 148 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.075  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.074  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.006  READ LOG EXT
  60 00 18 ff ff ff 4f 00      00:27:37.003  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.993  READ FPDMA QUEUED

Error 147 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.003  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.993  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.991  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.991  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.981  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     18885         825532056
# 2  Extended offline    Completed: read failure       90%     18879         825532056

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

"smartctl -q noserial -a /dev/ada1" seems fine:

Code:

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Oct  8 22:33:52 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  581) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 621) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   066   044    Pre-fail  Always       -       148443263
  3 Spin_Up_Time            0x0003   097   097   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       4
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   069   060   045    Pre-fail  Always       -       7422601
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       25 (151 99 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       4
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   068   066   040    Old_age   Always       -       32 (Min/Max 29/34)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       5
194 Temperature_Celsius     0x0022   032   040   000    Old_age   Always       -       32 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       25 (101 219 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       7478359850
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2932029794

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I am planning to run "smartctl -t long /dev/ada2" tonight and see what insights it provides. I started it a few times just now but realized it would take forever and figured I'd hold off a few hours.

Have I done something wrong, or is this type of replacement simply not possible when there are errors? Should I have replicated and not replaced?

Any ideas on how to move forward are greatly appreciated.
Thanks!

 

[HoneyBadger]

Try offlining the bad drive first and then detaching it.

Note that any checksum or data errors are going to be permanent, as your original configuration had no redundancy.

 

[MrRusch]

Try offlining the bad drive first and then detaching it.

Note that any checksum or data errors are going to be permanent, as your original configuration had no redundancy.

When I try to send the drive offline, I get an error saying "[EZFS_NOREPLICAS] no valid replicas"

Code:

Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 247, in __zfs_vdev_operation
    op(target, *args)
  File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 247, in __zfs_vdev_operation
    op(target, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 263, in <lambda>
    self.__zfs_vdev_operation(name, label, lambda target: target.offline())
  File "libzfs.pyx", line 1806, in libzfs.ZFSVdev.offline
libzfs.ZFSException: no valid replicas

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 97, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 53, in _run
    return await self._call(name, serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 263, in offline
    self.__zfs_vdev_operation(name, label, lambda target: target.offline())
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 249, in __zfs_vdev_operation
    raise CallError(str(e), e.code)
middlewared.service_exception.CallError: [EZFS_NOREPLICAS] no valid replicas
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1249, in offline
    await self.middleware.call('zfs.pool.offline', pool['name'], found[1]['guid'])
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1141, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1081, in _call
    return await self._call_worker(name, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1101, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1036, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1010, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EZFS_NOREPLICAS] no valid replicas

 

[sretalla]

The problem is that you have a corrupt file, so the resilver isn't officially complete and hence no matching replica exists to rely on.

You'll need to solve that (by restoring or deleting the corrupt file) first before anything else will work.

 

[MrRusch]

The problem is that you have a corrupt file, so the resilver isn't officially complete and hence no matching replica exists to rely on.

You'll need to solve that (by restoring or deleting the corrupt file) first before anything else will work.

OK. But the reference is for the entire bhyve VM OS zvol.. Is there any way to get more specific about what files are the actual culprits in this volume?
If not, I guess I will have to destroy the zvol completely for the process to complete. But is it possible to abort the replacement process somehow? I would need the new drive in a normal state so that I can start over on it in parallell..

 

[MrRusch]

After long SMART test, "smartctl -q noserial -a /dev/ada2" shows:

Code:

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate BarraCuda 3.5
Device Model:     ST4000DM004-2CV104
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Oct 11 10:20:06 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 489) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   048   006    Pre-fail  Always       -       209226237
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       53
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   094   060   045    Pre-fail  Always       -       2266552847
  9 Power_On_Hours          0x0032   079   079   000    Old_age   Always       -       18945 (164 75 0)
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       53
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       160
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   057   040    Old_age   Always       -       33 (Min/Max 25/36)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       738
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       979
194 Temperature_Celsius     0x0022   033   043   000    Old_age   Always       -       33 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   083   064   000    Old_age   Always       -       209226237
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       18896h+47m+05.734s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       55198661516
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       35078403463

SMART Error Log Version: 1
ATA Error Count: 151 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 151 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.357  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.283  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.283  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.282  READ LOG EXT
  60 00 18 ff ff ff 4f 00      00:27:37.269  READ FPDMA QUEUED

Error 150 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.269  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.268  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.195  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.194  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.194  READ LOG EXT

Error 149 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.180  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.179  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.106  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.106  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.105  READ LOG EXT

Error 148 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.075  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:37.074  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:27:37.006  READ LOG EXT
  60 00 18 ff ff ff 4f 00      00:27:37.003  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.993  READ FPDMA QUEUED

Error 147 occurred at disk power-on lifetime: 18869 hours (786 days + 5 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 18 ff ff ff 4f 00      00:27:37.003  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.993  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.991  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.991  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:27:36.981  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     18888         825532056
# 2  Extended offline    Completed: read failure       90%     18885         825532056
# 3  Extended offline    Completed: read failure       90%     18885         825532056
# 4  Extended offline    Completed: read failure       90%     18879         825532056

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Trying to figure out from this if I can find more specific references to what is actually wrong in the ZVOL.

The test actually passes, which seems positive..

Reading here: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbwl.html#gbcuz
"If the object number to a file path cannot be successfully translated, either due to an error or because the object doesn't have a real file path associated with it, as is the case for a dnode_t, then the dataset name followed by the object's number is displayed. For example:
monkey/dnode:<0x0>"
...which seems to be my case.

But the only guidance provided for how to repair the issue is
"If the corruption is within a directory or a file's metadata, the only choice is to move the file elsewhere. You can safely move any file or directory to a less convenient location, allowing the original object to be restored in its place."

So given that the actual file path cannot be resolved, should I consider MOVING the entire ZVOL - is this a correct interpretation?
All other solutions I have read about is 1) scrub, 2) clear, 3) repeat...

 

[MrRusch]

In case anyone finds this I'll finish the story.. I never found a way to determine what exactly was wrong with the OS ZVOL. So I ended up using the situation as an opportunity to recreate my VM setup in a FreeBSD jail.. Had both running in parallel until I was sure about the move, and finally removed the VM and subsequently the ZVOL. After restarting FreeNAS once more, the resilvering finally completed and the disk replacement was finalized. The old disk is marked as unused and the new one is alone in the pool's status.