Seeing unreadable sector errors on one drive

Status
Not open for further replies.
A

adramalech

Dabbler
Joined
Jul 27, 2013
Messages
12
Hey so I had come across an issue recently with one of my hard drives in a RAIDZ1.

The system is (running since August 30th 2013):
G.Skill 8GB dual channel kit (2x4GB not ECC) 1066MHz
Seasonic 360Watt
4x 3TB WD Red
ASRock E350M1 AMD E-350 Dual-Core

Drive #4 /dev/ada3 is showing on the alert, on the web page of the freenas box, "CRITICAL: Device: /dev/ada3, 9 Currently unreadable (pending) sectors."

Below I checked the smart status of each drive.

I am thinking I will have to be replacing at least the #4 hard drive soon, as was wondering if I should just replace it now, or is there a way to repair it to keep it up for awhile?

I was thinking of swapping out motherboards to a C2000 Xeon Atom supermicro board with 8 cores and get 204-pin DDR3 ECC and get 16 GBs. I know I will be, when I can, erasing the Raid Z1 and going to Raid Z2 as I don't use all the storage space the Z1 provides and really don't want to deal with losing my data if that another one drive or more goes out. I also thinking that running with 16GB of ECC ram will help prevent other issues occurring as I don't know if the RAM could go bad at any moment and I could potentially lose my pool.

I currently have a good backup of the data I was wondering what recommendations I could perform, like running a scrub, erasing, destroying and recreating the pool? Any help would be greatly appreciated!


Code:
[root@freenas ~]# smartctl -A /dev/ada0                                                                                           
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                       
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                       
                                                                                                                                  
=== START OF READ SMART DATA SECTION ===                                                                                           
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                           
  3 Spin_Up_Time            0x0027   174   173   021    Pre-fail  Always       -       6300                                       
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       16                                         
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                           
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                           
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14446                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                           
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                           
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       16                                         
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8                                           
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7                                           
194 Temperature_Celsius     0x0022   118   109   000    Old_age   Always       -       32                                         
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                           
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                           
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                           
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                           
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


Code:
[root@freenas ~]# smartctl -A /dev/ada1                                                                                           
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                       
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                       
                                                                                                                                  
=== START OF READ SMART DATA SECTION ===                                                                                           
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                           
  3 Spin_Up_Time            0x0027   180   180   021    Pre-fail  Always       -       5983                                       
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       17                                         
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                           
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                           
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14446                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                           
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                           
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       17                                         
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       9                                           
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7                                           
194 Temperature_Celsius     0x0022   116   108   000    Old_age   Always       -       34                                         
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                           
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                           
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                           
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                           
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


Code:
[root@freenas ~]# smartctl -A /dev/ada2                                                                                           
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                       
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                       
                                                                                                                                  
=== START OF READ SMART DATA SECTION ===                                                                                           
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                           
  3 Spin_Up_Time            0x0027   178   178   021    Pre-fail  Always       -       6083                                       
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       17                                         
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                           
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                           
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14446                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                           
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                           
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       17                                         
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       9                                           
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7                                           
194 Temperature_Celsius     0x0022   117   107   000    Old_age   Always       -       33                                         
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                           
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                           
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                           
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                           
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


Code:
[root@freenas ~]# smartctl -A /dev/ada3                                                                                           
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                       
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                       
                                                                                                                                  
=== START OF READ SMART DATA SECTION ===                                                                                           
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1057                                       
  3 Spin_Up_Time            0x0027   176   176   021    Pre-fail  Always       -       6183                                       
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       16                                         
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       1                                           
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                           
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14446                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                           
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                           
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       16                                         
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8                                           
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7                                           
194 Temperature_Celsius     0x0022   118   111   000    Old_age   Always       -       32                                         
196 Reallocated_Event_Count 0x0032   199   199   000    Old_age   Always       -       1                                           
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       9                                           
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                           
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                           
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


Code:
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                       
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                       
                                                                                                                                  
=== START OF INFORMATION SECTION ===                                                                                               
Model Family:     Western Digital Red                                                                                             
Device Model:     WDC WD30EFRX-68AX9N0                                                                                             
Serial Number:    WD-WCC1T0801826                                                                                                 
LU WWN Device Id: 5 0014ee 25dafcb09                                                                                               
Firmware Version: 80.00A80                                                                                                         
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                                                                               
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                           
Device is:        In smartctl database [for details use: -P show]                                                                 
ATA Version is:   ACS-2 (minor revision not indicated)                                                                             
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                           
Local Time is:    Sat May 16 13:58:32 2015 PDT                                                                                     
SMART support is: Available - device has SMART capability.                                                                         
SMART support is: Enabled                                                                                                         
                                                                                                                                  
=== START OF READ SMART DATA SECTION ===                                                                                           
SMART overall-health self-assessment test result: PASSED                                                                           
                                                                                                                                  
General SMART Values:                                                                                                             
Offline data collection status:  (0x00) Offline data collection activity                                                           
                                        was never started.                                                                         
                                        Auto Offline Data Collection: Disabled.                                                   
Self-test execution status:      (   0) The previous self-test routine completed                                                   
                                        without error or no self-test has ever                                                     
                                        been run.                                                                                 
Total time to complete Offline                                                                                                     
data collection:                (40980) seconds.                                                                                   
Offline data collection                                                                                                           
capabilities:                    (0x7b) SMART execute Offline immediate.                                                           
                                        Auto Offline data collection on/off support.                                               
                                        Suspend Offline collection upon new                                                       
                                        command.                                                                                   
                                        Offline surface scan supported.                                                           
                                        Self-test supported.                                                                       
                                        Conveyance Self-test supported.                                                           
                                        Selective Self-test supported.                                                             
SMART capabilities:            (0x0003) Saves SMART data before entering                                                           
                                        power-saving mode.                                                                         
                                        Supports SMART auto save timer.                                                           
Error logging capability:        (0x01) Error logging supported.                                                                   
                                        General Purpose Logging supported.                                                         
Short self-test routine                                                                                                           
recommended polling time:        (   2) minutes.                                                                                   
Extended self-test routine                                                                                                         
recommended polling time:        ( 411) minutes.                                                                                   
Conveyance self-test routine                               
recommended polling time:        (   5) minutes.                                                                                   
SCT capabilities:              (0x70bd) SCT Status supported.                                                                     
                                        SCT Error Recovery Control supported.                                                     
                                        SCT Feature Control supported.                                                             
                                        SCT Data Table supported.                                                                 
                                                                                                                                  
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1057                                       
  3 Spin_Up_Time            0x0027   176   176   021    Pre-fail  Always       -       6183                                       
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       16                                         
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       1                                           
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                           
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14447                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                           
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0   
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       16                                         
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8                                           
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7                                           
194 Temperature_Celsius     0x0022   118   111   000    Old_age   Always       -       32                                         
196 Reallocated_Event_Count 0x0032   199   199   000    Old_age   Always       -       1                                           
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       9                                           
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                           
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                           
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0                                           
                                                                                                                                  
SMART Error Log Version: 1                                                                                                         
No Errors Logged                                                                                                                   
                                                                                                                                  
SMART Self-test log structure revision number 1                                                                                   
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                   
# 1  Extended offline    Completed without error       00%     14420         -                                                     
# 2  Extended offline    Completed without error       00%     14085         -                                                     
# 3  Extended offline    Completed without error       00%     13701         -                                                     
# 4  Extended offline    Completed without error       00%     13369         -                                                     
# 5  Extended offline    Completed without error       00%     12961         -                                                     
# 6  Extended offline    Completed without error       00%     12627         -                                                     
# 7  Extended offline    Completed: read failure       20%     12299         3150900344                                           
# 8  Extended offline    Completed without error       00%     11956         -                                                     
# 9  Extended offline    Completed without error       00%     11548         -                                                     
#10  Extended offline    Completed without error       00%     11212         -                                                     
#11  Extended offline    Completed without error       00%     10805         -                                                     
#12  Extended offline    Completed without error       00%     10494         -                                                     
#13  Extended offline    Completed without error       00%     10110         -                                                     
#14  Extended offline    Completed without error       00%      9773         -                                                     
#15  Extended offline    Completed: read failure       90%      9358         29291888                                             
#16  Extended offline    Completed without error       00%      9030         -                                                     
#17  Extended offline    Completed without error       00%      8646         -                                                     
#18  Extended offline    Completed without error       00%      8311         -                                                     
#19  Extended offline    Completed without error       00%      7903         -                                                     
#20  Extended offline    Completed without error       00%      7567         -                                                     
#21  Extended offline    Completed without error       00%      7160         -                                                     
2 of 2 failed self-tests are outdated by newer successful extended offline self-test # 1                                           
                                                                                                                                  
SMART Selective self-test log data structure revision number 1                                                                     
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                       
    1        0        0  Not_testing                                                                                               
    2        0        0  Not_testing                                                                                               
    3        0        0  Not_testing                                                                                               
    4        0        0  Not_testing                                                                                               
    5        0        0  Not_testing                                                                                               
Selective self-test flags (0x0):                                                                                                   
  After scanning selected spans, do NOT read-scan remainder of disk.                                                               
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Since this is RAIDZ1, I'd advise you to replace it now that the others are in good working order. It should be possible to resilver before removing the old one in your case, since you only have four drives.

I'd definitely scrub before the resilver, since you're doing this calmly.
 
M

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
The board only has 4 SATA ports, so ^ that's not possible.

Get an advanced RMA nonetheless and good luck. You may want to make a backup of important files with the single parity disk still in place.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
marbus90 said:
The board only has 4 SATA ports, so ^ that's not possible.

Get an advanced RMA nonetheless and good luck. You may want to make a backup of important files with the single parity disk still in place.
Click to expand...
Hooray for manufacturers cheaping out by not including the last two SATA channels.
 
A

adramalech

Dabbler
Joined
Jul 27, 2013
Messages
12
Yea so I am making a second backup, I have an older one like a few days if not a week or so... Then I most likely destroy the zpool, zero out the drives, pull #4 and RMA as they are under a 5 year limited warranty and RMA it, and purchase:

Crucial 16GB (2x8GB) 1600Mhz 240-pin DDR3 ECC unbuffered
Gigabyte GA-9SISL Mini ITX Board

Total cost $600.00.
-- or --

SuperMicro MBD-A1SRi-2758F-O
Kingston 8GB 204-pin SODIMM

Total cost $472.00

Then just re-purpose my Asrock e350m1 board and non-ecc ram as a pfsense box with a dual intel 1Gbps NIC, another power supply, and another miniITX case.

I might just buy one hard drive too and then rma one and fit 6 in my Fractal NODE 304.

Thank you all for the help so far! Really appreciated it. :D
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Details: C2750 boards are better for NAS than C2758 boards. C2758 is designed for stuff like routers.

Also, Kingston has acquired quite a nasty reputation lately. If possible, get something else (and always try to get the stuff listed in Supermicro's compatibility list.
 
A

adramalech

Dabbler
Joined
Jul 27, 2013
Messages
12
Ericloewe said:
Details: C2750 boards are better for NAS than C2758 boards. C2758 is designed for stuff like routers.

Also, Kingston has acquired quite a nasty reputation lately. If possible, get something else (and always try to get the stuff listed in Supermicro's compatibility list.
Click to expand...

Yea I know but if I get super micro which seems to have guaranteed compatibility with bsd, the issue I see is I am stuck with SO-DIMM from kingston or get this from crucial: 8GB SO-DIMM 1600MHz ECC Unbuffered

Has anybody heard of any issues or problems with the Gigabyte GA-9SISL C2750 board with the full 240-pin DDR3 DIMMs?
 
Last edited:
M

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
If you're already shelling out that kind of money and aren't scared of higher energy usage under full load, you may as well go for the ASRock MT-C224 with a socket 1150 CPU.

Alternatively you can get the ASRock C2750D4I, supports fullsize dimms, just avoid the 6 Marvell SATA ports.
 
A

adramalech

Dabbler
Joined
Jul 27, 2013
Messages
12
marbus90 said:
If you're already shelling out that kind of money and aren't scared of higher energy usage under full load, you may as well go for the ASRock MT-C224 with a socket 1150 CPU.

Alternatively you can get the ASRock C2750D4I, supports fullsize dimms, just avoid the 6 Marvell SATA ports.
Click to expand...

You know you are right!

With an Intel Xeon E3-1226 v3, an ASRock MT-C224 mobo, and crucial 16GB of ECC DDR3 RAM I am looking at around $560.

You said the SATA are all MARVELL? So I would have to get like PCI Express SATA III card?

Could I just go with the ASRock C226 board, http://www.newegg.com/Product/Product.aspx?Item=N82E16813157466 It comes with dual Intel NICs and I think Intel 6x SATA III? That would bring the total back around $600.
 
M

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
Ignore Marvells _on the C2750D4I_, do not ignore the Intels on the MT-C224 and do not ignore the intels on the C2750D4I.. Ignore SATA cards. Use only Intel integrated or LSI HBAs.

The MT-C224 has 6x Intel SATA + 2x Gbe as well.
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Seeing unreadable sector errors on one drive"

Similar threads

Marcet
  • Locked
SMART service reports CRITICAL errors, but disks seems fine.
2
Replies
26
Views
9K
Marcet
Marcet
A
  • Locked
When do you actually replace a failing drive?
2
Replies
24
Views
8K
aadje93
A
E
  • Locked
Possible Disk Failure
Replies
6
Views
2K
joeschmuck
joeschmuck
T
  • Locked
WD Red LCC
Replies
8
Views
4K
Something
Something
S
  • Locked
Help with SMART results
Replies
12
Views
10K
droeders
droeders
Top