One or more devices has experienced an unrecoverable error

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
Hey,

Got this message overnight from my TrueNAS box.

Pool Tank state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
2021-10-01 04:10:19 (Europe/London)

Tank is showing as ONLINE (Unhealthy) (X)

Tank Status:

Code:
Name     Read     Write     Checksum     Status
Tank    0        0        0            ONLINE
MIRROR    0        0        0            ONLINE
da4p2    0        0        0            ONLINE
da3p2    0        0        13            ONLINE
MIRROR    0        0        0            ONLINE
da2p2    0        0        0            ONLINE
da0p2    0        0        0            ONLINE
MIRROR    0        0        0            ONLINE
da5p2    0        0        0            ONLINE
da1p2    0        0        0            ONLINE
MIRROR    0        0        0            ONLINE
da7p2    0        0        0            ONLINE
da6p2    0        0        0            ONLINE


Ran a Short SMART Test on da3 (I do run these reguarly and have had no failures) and got this output (ignore the previous run cancelled by host, I initially did a long test but realised it was going to take hours, cancelled and did a short smart test to get some data quickly)

Code:
# smartctl -a /dev/da3
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD80EFZX-68UW8N0
Serial Number:    VK0HH33Y
LU WWN Device Id: 5 000cca 3b7c70a40
Firmware Version: 83.H0A83
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct 13 14:08:31 2021 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  101) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1083) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   131   131   054    Pre-fail  Offline      -       116
  3 Spin_Up_Time            0x0007   156   156   024    Pre-fail  Always       -       457 (Average 387)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       41
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   095   095   000    Old_age   Always       -       40809
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       41
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   098   098   000    Old_age   Always       -       3160
193 Load_Cycle_Count        0x0012   098   098   000    Old_age   Always       -       3160
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 17/38)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     40809         -
# 2  Extended offline    Aborted by host               90%     40809         -
# 3  Short offline       Completed without error       00%     40774         -
# 4  Extended offline    Completed without error       00%     40715         -
# 5  Short offline       Completed without error       00%     40606         -
# 6  Short offline       Completed without error       00%     40390         -
# 7  Extended offline    Completed without error       00%     40325         -
# 8  Short offline       Completed without error       00%     40223         -
# 9  Short offline       Completed without error       00%     40055         -
#10  Extended offline    Completed without error       00%     39989         -
#11  Short offline       Completed without error       00%     39887         -
#12  Short offline       Completed without error       00%     39647         -
#13  Extended offline    Completed without error       00%     39585         -
#14  Short offline       Completed without error       00%     39479         -
#15  Short offline       Completed without error       00%     39311         -
#16  Extended offline    Completed without error       00%     39243         -
#17  Short offline       Completed without error       00%     39143         -
#18  Short offline       Completed without error       00%     38903         -
#19  Extended offline    Completed without error       00%     38837         -
#20  Short offline       Completed without error       00%     38735         -
#21  Short offline       Completed without error       00%     38567         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


This drive is a WD Red 8TB drive that I bought at the beginning of 2017, so it's approaching 5 years old now.

So I assume the drive is probably starting to show it's age and is beginning to start it's decline, so what are my next steps? Is it worth me running an extended SMART test at this stage? Am I best just replacing it? In regards to replacing it, this drive is part of an early mirror VDEV, so I'm probably best replacing it with a new HDD that I'm now using on other VDEVs, the EXOS 16TB, I'll be able to replace this drive with that won't I, but I won't benefit from the additional capacity of the drive until I replace the other HDD in it's VDEV with the EXOS 16TB?

Thanks!
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
That drive shows checksum errors, so what's on disk didn't match the ZFS checksums for those blocks. Usually these mean sectors failing. You should execute the drive replacement procedure.

As for your second question, your pool won't use the additional space on a larger replacement drive until all members in all VDEVs are upgraded to the larger size.
 

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
Hey,

That's great thanks. I've ordered a replacement drive.

Should I do the offline replace procedure (old disk offline, remove, insert new disk then online new disk), or the online replace (putting in new drive and then replacing old drive). My gut says the online replace as its safer, but the manual says its not recommended if the disk degradation isn't known as it takes longer to resilver.

What would you recommend?

Thanks!
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398

brettyj

Dabbler
Joined
Jul 30, 2020
Messages
23
Okay brilliant. Small hopefully ui issue, but since running the smart test earlier, a new entry has turned up in the disk list for the same disk here's a screenshot (apologies I'm on mobile so this is only showing the two drives)

Is this going to be an issue when I replace the drives? If so how should I resolve it? A reboot?

Thanks, Hopefully this is my last question!
 

Attachments

  • Screenshot_20211014-003805~3.png
    Screenshot_20211014-003805~3.png
    157.7 KB · Views: 184

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
So long as the new entry isn't associated with a pool, it will likely disappear when you pull that drive. This just means the drive isn't reliably reporting its serial number, and is another indication of impending failure.
 
Top