Unrecoverable error

G

Ganzir

Explorer
Joined
May 15, 2016
Messages
57
Hello everyone,

my NAS just issued the following alert:

Code:
New alerts:
* Pool bigkegostorage state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected..


zpool status -v gives the following output:

Code:
 
pool: bigkegostorage
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub in progress since Sun Oct 11 00:00:00 2020
        3.83T scanned at 380M/s, 3.25T issued at 322M/s, 5.51T total
        1.56M repaired, 58.89% done, 0 days 02:03:02 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        bigkegostorage                                  ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/a9d5743e-ec7c-11ea-a41b-10c37b4dfcfa  ONLINE       0     0     0
            gptid/ab0913bd-ec7c-11ea-a41b-10c37b4dfcfa  ONLINE       0     0     0
            gptid/ab3257fd-ec7c-11ea-a41b-10c37b4dfcfa  ONLINE       0     0    10
            gptid/ab414e59-ec7c-11ea-a41b-10c37b4dfcfa  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:07 with 0 errors on Tue Oct  6 03:45:07 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          nvd0p2    ONLINE       0     0     0

errors: No known data errors


So I did

glabel status

Code:
                                      Name  Status  Components
gptid/cd609f50-13e7-11e6-8aac-10c37b4dfcfa     N/A  nvd0p1
gptid/ab3257fd-ec7c-11ea-a41b-10c37b4dfcfa     N/A  ada0p2
gptid/ab0913bd-ec7c-11ea-a41b-10c37b4dfcfa     N/A  ada1p2
gptid/a9d5743e-ec7c-11ea-a41b-10c37b4dfcfa     N/A  ada2p2
gptid/ab414e59-ec7c-11ea-a41b-10c37b4dfcfa     N/A  ada3p2


And finally

smartctl -a /dev/ada0

Output as follows:

Code:
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital RE4-GP
Device Model:     WDC WD2003FYPS-27Y2B0
Serial Number:    WD-WCAVY5494816
LU WWN Device Id: 5 0014ee 25a518c8b
Firmware Version: 04.05G11
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Sun Oct 11 03:01:10 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (42900) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 488) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   218   021    Pre-fail  Always       -       8891
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       146
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11393
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       146
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       23
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       122
194 Temperature_Celsius     0x0022   116   100   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       23

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       50%     10225         2356483288

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Is something serious happening here and should I replace ada0?


My configuration:

Version: FreeNAS-11.3-U5
HDDs: 3 x Western Digital WD RE4-GP 2 TB
1 x WD WD80EFAX Red 8TB
RAM: 16 GB
CPU: i5-4690 CPU @ 3.50GH
Mainboard: H97i-Plus from Asus
Case: Fractal Node 304
 
Last edited:
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,996
Your hard drive is failing, replace it as soon as possible. The indicator is the SMART Long/Extended test failed.

If you do not have SMART Short and Long tests running routinely, I would suggest that you do.

Good Luck
 
G

Ganzir

Explorer
Joined
May 15, 2016
Messages
57
Thanks for the answer.

Just to be clear, you are refering to this line?

Code:
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       50%     10225         2356483288


EDIT:

I just ran a SMART Long Test, now the output is:

Code:
root@freenas:~ # smartctl -a /dev/ada0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p14 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital RE4-GP
Device Model:     WDC WD2003FYPS-27Y2B0
Serial Number:    WD-WCAVY5494816
LU WWN Device Id: 5 0014ee 25a518c8b
Firmware Version: 04.05G11
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Sun Oct 11 21:44:21 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (42900) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 488) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   218   021    Pre-fail  Always       -       8891
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       146
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11412
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       146
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       23
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       122
194 Temperature_Celsius     0x0022   119   100   000    Old_age   Always       -       33
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       9

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     11411         -
# 2  Extended offline    Completed: read failure       50%     10225         2356483288
1 of 1 failed self-tests are outdated by newer successful extended offline self-test # 1

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
Last edited:
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,996
Yes that is what I was referring to. That is odd that a long test failed and then passed. You also have errors for multi zone but those alone do not always mean the drive is failing. My advice is to RMA your drive but if you choose not to then perform a long test weekly to ensure you do not have a real problem.

Good luck
 
G

Ganzir

Explorer
Joined
May 15, 2016
Messages
57
Thanks for your answer, unfortunately the drive has no warranty left. So I guess I buy a new one. WD Gold 8TB its going to be, once all drives are replaced, the system is upgraded from 4 x 2TB to 4 x 8TB. All drives than with a 5 year warranty so I am somewhat covered for the nex half decade.

Still can't figure out though, why scubs are performed every Tuesday although none are scheduled at that time.
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Unrecoverable error"

Similar threads

A
  • Locked
Cheksum error in FreeNAS 9.2.0
Replies
1
Views
2K
cyberjock
C
H
  • Locked
Possible Bad Drive?
Replies
8
Views
3K
joeschmuck
joeschmuck
T
Nothing to Lose But I Lost All
Replies
12
Views
4K
themelic
T
D
ZPool degraded, but I can't see which disk is the issue
Replies
3
Views
2K
Redcoat
Redcoat
J
  • Locked
Currently unreadable (pending) sectors - Extended offline Completed without error
Replies
3
Views
3K
jFlatz
J
Top