On Power-On disk error

[pechkin000]

Every time I reboot, I get the following error:


CRITICAL
Device: /dev/da1 [SAT], 1 Offline uncorrectable sector


However my zpool status output looks error free and if I dismiss this error it doesn't come up again. In UI everything looks error free.

When I run smartctl -x /dev/da1 I dont see anything obvious. I am wondering if this is something I need to worry about immediately? This only comes up on reboot, otherwise disk functions perfectly fine.

Here is the full output of the smartctl -x /dev/da1d, I am wondering if someone who is better than me at reading this could help me understand what these errors means and how urgent it is to address it.

Thank you in advance for any help with this.

My system specs just in case:
OS Version:
FreeNAS-11.3-MASTER-201903060936
(Build Date: Mar 6, 2019 5:51)
Processor:
Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (8 cores)
Memory:
16 GiB

Code:

replicator# smartctl -x /dev/da1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD40EZRX-00SPEB0
Serial Number:    WD-WCC4E996889J
LU WWN Device Id: 5 0014ee 2b54cdc9b
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Mar  6 11:23:50 2019 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (53460) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 534) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x7035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   177   177   021    -    8108
  4 Start_Stop_Count        -O--CK   100   100   000    -    35
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   048   048   000    -    38489
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    35
192 Power-Off_Retract_Count -O--CK   200   200   000    -    34
193 Load_Cycle_Count        -O--CK   200   200   000    -    191
194 Temperature_Celsius     -O---K   121   115   000    -    31
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    1
198 Offline_Uncorrectable   ----CK   200   200   000    -    1
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    1
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20701         -
# 2  Extended offline    Completed without error       00%     20197         -
# 3  Extended offline    Completed without error       00%     18014         -
# 4  Extended offline    Completed without error       00%     17511         -
# 5  Extended offline    Completed without error       00%     17007         -
# 6  Extended offline    Completed without error       00%     14826         -
# 7  Extended offline    Completed without error       00%     13153         -
# 8  Extended offline    Completed without error       00%     12649         -
# 9  Extended offline    Completed without error       00%     10468         -
#10  Extended offline    Completed without error       00%      9963         -
#11  Extended offline    Completed without error       00%      7278         -
#12  Extended offline    Completed without error       00%      6775         -
#13  Extended offline    Completed without error       00%      4594         -
#14  Extended offline    Completed without error       00%      3922         -
#15  Extended offline    Completed without error       00%      3419         -
#16  Extended offline    Completed without error       00%      2916         -
#17  Short offline       Completed without error       00%      1262         -
#18  Short offline       Completed without error       00%      1238         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     26/34 Celsius
Lifetime    Min/Max Temperature:     19/37 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (89)

Index    Estimated Time   Temperature Celsius
  90    2019-03-06 03:26    30  ***********
...    ..( 55 skipped).    ..  ***********
146    2019-03-06 04:22    30  ***********
147    2019-03-06 04:23    31  ************
...    ..( 11 skipped).    ..  ************
159    2019-03-06 04:35    31  ************
160    2019-03-06 04:36    32  *************
...    ..( 23 skipped).    ..  *************
184    2019-03-06 05:00    32  *************
185    2019-03-06 05:01    31  ************
...    ..(  2 skipped).    ..  ************
188    2019-03-06 05:04    31  ************
189    2019-03-06 05:05    32  *************
...    ..(  2 skipped).    ..  *************
192    2019-03-06 05:08    32  *************
193    2019-03-06 05:09    31  ************
...    ..( 61 skipped).    ..  ************
255    2019-03-06 06:11    31  ************
256    2019-03-06 06:12    32  *************
...    ..(  2 skipped).    ..  *************
259    2019-03-06 06:15    32  *************
260    2019-03-06 06:16    31  ************
...    ..(194 skipped).    ..  ************
455    2019-03-06 09:31    31  ************
456    2019-03-06 09:32    30  ***********
...    ..(110 skipped).    ..  ***********
  89    2019-03-06 11:23    30  ***********

SCT Error Recovery Control command not supported

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           77  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           78  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4     42202926  Vendor specific

s
Wed, 06 Mar 2019 18:44:04 GMT

 

[Chris Moore]

This is the part of the SMART results that FreeNAS is trying to warn you about. You have a single pending sector, a single Offline Uncorrectable, which is probably the same sector that is pending, and you have a single multi zone error.

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE

197 Current_Pending_Sector  -O--CK   200   200   000    -    1
198 Offline_Uncorrectable   ----CK   200   200   000    -    1

200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    1

This is not insignificant, but it is probably not enough to cause a data error which would be reported in zpool status.
The other thing I notice is that this drive has 38489 power on hours.

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE

  9 Power_On_Hours          -O--CK   048   048   000    -    38489

That works out to 4.3 years and it is a WD40EZRX, which is a Green drive. That means it is well passed the warranty period, but I have had a Green drive that lasted over 7 years, so that isn't a sold decider, however this one has given you a bad sector, so you are probably looking at trouble coming your way. This is probably a warning that it is time to consider getting some new drives on the way.

I had a bunch of drives that were all over 5 years and had to replace them all at once around the beginning of last year. I ended up getting a 25 pack because I had 12 drives in each of my two servers and I wanted to have a spare...

 

[pechkin000]

Thank you so much for your help and explanation. This is my replicator server of the production server that, yes, I did build with greens, which I actually flashed the firmware with didle3. I built the production system with wd reds at a time and this one with all flashed greens. My plan at the time was to replace the failed reds with proper pro versions (I think at the time they were wd black pro) and greens with reds as they fail as money saving measure. It has been 5 years for the production system and just under 5 for the replicator. This is the first time a drive failed. So by the looks of it I will be doing the same thing, when we get closer to running out of space I will be just replacing them all at once. I would need 16 in total. For now I have a spare couple of reds so I will pop it in instead of this one.

I just wanted to make sure that this is not some glitch in the freenas nightly, as unfortunately I don't undertand completely the smart readout. I run stable 11.2 on production and nightlies on replicator, because why not help with development :) Actually managed to report couple of bugs!
Thanks again!

 

[Chris Moore]

I just wanted to make sire that Thi is not some glitch in the freenas nightly, as unfortunately I don't undertand completely the smart readout.
Thanks again!

There is a guide for testing the drives if you want to have a go at it:

Hard Drive Troubleshooting Guide (All Versions of FreeNAS)
https://forums.freenas.org/index.ph...bleshooting-guide-all-versions-of-freenas.17/

You might also find this useful:

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Please ask away if you have any questions. I am sure someone will be happy to help you.

 

[pechkin000]

I will definitely take a look. I am at the point where we need to make some decisions about the drives so now as good of time as ever to look into it.
Cheers!