When is a drive good to change?

[coolnodje]

I've had errors reported for years now on a specific drive:

Code:

Sep 29 21:32:53 freenas 1 2020-09-29T21:32:53.602445+02:00 freenas.local smartd 1687 - - Device: /dev/ada3, 255 Currently unreadable (pending) sectors
Sep 29 21:32:53 freenas 1 2020-09-29T21:32:53.602534+02:00 freenas.local smartd 1687 - - Device: /dev/ada3, 83 Offline uncorrectable sectors

Numbers are growing but I don't think that's a problem as long as the sectors are identified and marked as uncorrectable in the end.

But now TrueNAS display this Red cross error and rate it UNHEALTHY.

`zpool -v status` identifies cheksum errors on the disk. I `zpool clean` but this keeps coming up.

Here's the drive in question smartctl information.

Is there anything that should tells me to change it early or can I just wait for it to die slowly?

Code:

freenas% sudo smartctl -a /dev/ada3
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-PRERELEASE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD30EZRX-00D8PB0
Serial Number:    WD-WCC4N1036934
LU WWN Device Id: 5 0014ee 25f69eeb1
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Sep 29 22:23:58 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (40080) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 402) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x7035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   199   199   051    Pre-fail  Always       -       14043
  3 Spin_Up_Time            0x0027   175   172   021    Pre-fail  Always       -       6208
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       415
  5 Reallocated_Sector_Ct   0x0033   197   197   140    Pre-fail  Always       -       94
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   078   078   000    Old_age   Always       -       16122
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       266
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       94
193 Load_Cycle_Count        0x0032   155   155   000    Old_age   Always       -       137507
194 Temperature_Celsius     0x0022   119   094   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   139   139   000    Old_age   Always       -       61
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       255
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       83
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   199   199   000    Old_age   Offline      -       518

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       10%     15945         1041684368
# 2  Short offline       Completed without error       00%      6914         -
# 3  Short offline       Completed without error       00%      6913         -
# 4  Short offline       Completed without error       00%      6912         -
# 5  Short offline       Completed without error       00%      6911         -
# 6  Short offline       Completed without error       00%      6895         -
# 7  Short offline       Completed without error       00%      6893         -
# 8  Short offline       Completed without error       00%      6892         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Spearfoot]

That drive has multiple problems and needs to be replaced as soon as possible.

 

[Redcoat]

@coolnodje, does it help focus to consider the redundancy level of the pool that drive is part of?

If it's Z1, your pool's a goner when that drive croaks.

If it's Z2, your pool's gone if you lose another while replacing ada3.

If it's Z3, you've a little grace on your side...

 

[Spearfoot]

@Redcoat is correct, and I'll repeat my advice to replace this disk immediately.

The drive failed to finish an extended test, failing with a read error at LBA 1041684368. Plus it is returning these worrisome SMART results:

Code:

5 Reallocated_Sector_Ct   0x0033   197   197   140    Pre-fail  Always       -       94
196 Reallocated_Event_Count 0x0032   139   139   000    Old_age   Always       -       61
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       255
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       83

 

[joeschmuck]

@Redcoat is correct, and I'll repeat my advice to replace this disk immediately.

The drive failed to finish an extended test, failing with a read error at LBA 1041684368. Plus it is returning these worrisome SMART results:

Code:

5 Reallocated_Sector_Ct   0x0033   197   197   140    Pre-fail  Always       -       94
196 Reallocated_Event_Count 0x0032   139   139   000    Old_age   Always       -       61
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       255
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       83

Agreed, this drive has failed already, you just have not seen any data impacts YET.

 

[Fredda]

Code:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       10%     15945         1041684368
# 2  Short offline       Completed without error       00%      6914         -
# 3  Short offline       Completed without error       00%      6913         -
# 4  Short offline       Completed without error       00%      6912         -
# 5  Short offline       Completed without error       00%      6911         -
# 6  Short offline       Completed without error       00%      6895         -
# 7  Short offline       Completed without error       00%      6893         -
# 8  Short offline       Completed without error       00%      6892         -

It looks you don't have SMART test enabled as a daily/weekly task. You should change that.

 

[coolnodje]

A lot of info, thanks everyone.

My pool is RAIDZ2, so the drive can fail.

I'll be looking into the regular smart test @Fredda . I feel it's enabled though because I have seen the reported Pending Sector numbers slowly increasing over time!

@Spearfoot
> The drive failed to finish an extended test, failing with a read error at LBA 1041684368
That sounds bad enough to me.
But about
> worrisome SMART results:
at what point these figures become worrisome was the meaning in my original question.

AFAIU, it's safe to have some Offline_Uncorrectable, it doesn't endanger the drive itself.
And Current_Pending_Sector will become Offline_Uncorrectable once the drive tries to write on them. Correct?

 

[coolnodje]

@Fredda you were right, SMART Tests were not scheduled. The default option in Tasks->SMART Test lead me to think it was activated, but it's actually not enabled until you select disks on which to perform it.

I guess the alerts came from SMART Monitoring as opposed to SMART Testing.

Always learning...