Numerous ATA errors and unreadable sector (pending) + other stuff

Status
Not open for further replies.
nemisisak

nemisisak

Explorer
Joined
Jun 19, 2015
Messages
69
Hey,

Having a few problems. One of my disks failed from my 6x3Tb Z1 raid. So I decided I would take the opportunity to expand my pool and implement Z2 redundency.
All of the drives are Seagate ST3000DM001 Barracudas. Out of a mere few quid difference I bought 3 additional Toshiba 3Tb DT01ACA300 drives.
Once they arrived I banged them in, destroyed my old pool, created the new one and restored from my backup. Once that completed I was getting alot of errors due to unreadable sectors from ada1 + ada5. So I ran a long SMART test on each and then scrubed my pool. This is what im left with:

Jan 12 21:32:29 freenas smartd[2831]: Device: /dev/ada1, 320 Currently unreadable (pending) sectors
Jan 12 21:32:29 freenas smartd[2831]: Device: /dev/ada5, 672 Currently unreadable (pending) sectors

  • CRITICAL: Device: /dev/ada5, 32 Currently unreadable (pending) sectors
  • CRITICAL: Device: /dev/ada5, Self-Test Log error count increased from 0 to 1
  • CRITICAL: Device: /dev/ada5, ATA error count increased from 179 to 265
  • CRITICAL: Device: /dev/ada1, 104 Currently unreadable (pending) sectors
  • CRITICAL: Device: /dev/ada1, Self-Test Log error count increased from 0 to 1
  • CRITICAL: Device: /dev/ada1, ATA error count increased from 250 to 305
  • CRITICAL: The volume AllDiskVolume (ZFS) state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Code:
[root@freenas] ~# smartctl -a /dev/ada1
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA300
Serial Number:    55D64D5GS
LU WWN Device Id: 5 000039 fe3c2cab7
Firmware Version: MX6OABB0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jan 12 21:32:20 2016 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (22222) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 371) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   139   139   054    Pre-fail  Offline      -       70
  3 Spin_Up_Time            0x0007   100   100   024    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   018   018   005    Pre-fail  Always       -       1618
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   124   124   020    Pre-fail  Offline      -       33
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       74
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0002   193   193   000    Old_age   Always       -       31 (Min/Max 20/35)
196 Reallocated_Event_Count 0x0032   019   019   000    Old_age   Always       -       1623
197 Current_Pending_Sector  0x0022   093   093   000    Old_age   Always       -       320
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 7817 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 7817 occurred at disk power-on lifetime: 63 hours (2 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 40 cc 51 0c  Error: WP at LBA = 0x0c51cc40 = 206687296

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 08 08 b0 d9 75 40 00   2d+14:48:03.700  WRITE FPDMA QUEUED
  60 28 00 48 26 52 40 00   2d+14:48:03.700  READ FPDMA QUEUED
  60 30 f8 28 cc 51 40 00   2d+14:48:03.700  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+14:48:03.696  READ LOG EXT
  61 08 f8 b0 d9 75 40 00   2d+14:48:00.247  WRITE FPDMA QUEUED

Error 7816 occurred at disk power-on lifetime: 63 hours (2 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 40 cc 51 0c  Error: WP at LBA = 0x0c51cc40 = 206687296

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 08 f8 b0 d9 75 40 00   2d+14:48:00.247  WRITE FPDMA QUEUED
  60 28 f0 48 26 52 40 00   2d+14:48:00.247  READ FPDMA QUEUED
  60 30 e8 28 cc 51 40 00   2d+14:48:00.247  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+14:48:00.243  READ LOG EXT
  60 28 e0 48 26 52 40 00   2d+14:47:56.935  READ FPDMA QUEUED

Error 7815 occurred at disk power-on lifetime: 63 hours (2 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 40 cc 51 0c  Error: UNC at LBA = 0x0c51cc40 = 206687296

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 28 e0 48 26 52 40 00   2d+14:47:56.935  READ FPDMA QUEUED
  60 30 d8 28 cc 51 40 00   2d+14:47:56.935  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+14:47:56.931  READ LOG EXT
  60 28 d8 48 26 52 40 00   2d+14:47:53.424  READ FPDMA QUEUED
  60 30 d0 28 cc 51 40 00   2d+14:47:53.424  READ FPDMA QUEUED

Error 7814 occurred at disk power-on lifetime: 63 hours (2 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 40 cc 51 0c  Error: UNC at LBA = 0x0c51cc40 = 206687296

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 28 d8 48 26 52 40 00   2d+14:47:53.424  READ FPDMA QUEUED
  60 30 d0 28 cc 51 40 00   2d+14:47:53.424  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+14:47:53.421  READ LOG EXT
  60 28 d0 48 26 52 40 00   2d+14:47:49.992  READ FPDMA QUEUED
  60 30 c8 28 cc 51 40 00   2d+14:47:49.959  READ FPDMA QUEUED

Error 7813 occurred at disk power-on lifetime: 63 hours (2 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 40 cc 51 0c  Error: UNC at LBA = 0x0c51cc40 = 206687296

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 28 d0 48 26 52 40 00   2d+14:47:49.992  READ FPDMA QUEUED
  60 30 c8 28 cc 51 40 00   2d+14:47:49.959  READ FPDMA QUEUED
  60 28 c0 10 f6 38 40 00   2d+14:47:49.950  READ FPDMA QUEUED
  60 30 b8 80 da 52 40 00   2d+14:47:49.949  READ FPDMA QUEUED
  60 28 b0 08 bf 39 40 00   2d+14:47:49.949  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%        26         445515648

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@freenas] ~# smartctl -a /dev/ada5
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA300
Serial Number:    55D5VS5GS
LU WWN Device Id: 5 000039 fe3c2aa43
Firmware Version: MX6OABB0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jan 12 21:35:15 2016 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (22222) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 371) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   139   139   054    Pre-fail  Offline      -       70
  3 Spin_Up_Time            0x0007   100   100   024    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   050   050   005    Pre-fail  Always       -       981
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   124   124   020    Pre-fail  Offline      -       33
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       74
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 23/39)
196 Reallocated_Event_Count 0x0032   054   054   000    Old_age   Always       -       1009
197 Current_Pending_Sector  0x0022   073   073   000    Old_age   Always       -       672
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 5325 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 5325 occurred at disk power-on lifetime: 61 hours (2 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 58 18 da 74 0b  Error: UNC at LBA = 0x0b74da18 = 192207384

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 e0 70 da 74 40 00   2d+13:15:08.141  READ FPDMA QUEUED
  60 e0 d8 90 d9 74 40 00   2d+13:15:08.141  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+13:15:08.137  READ LOG EXT
  60 e0 d8 70 da 74 40 00   2d+13:15:04.298  READ FPDMA QUEUED
  60 e0 d0 90 d9 74 40 00   2d+13:15:04.298  READ FPDMA QUEUED

Error 5324 occurred at disk power-on lifetime: 61 hours (2 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 58 18 da 74 0b  Error: UNC at LBA = 0x0b74da18 = 192207384

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 d8 70 da 74 40 00   2d+13:15:04.298  READ FPDMA QUEUED
  60 e0 d0 90 d9 74 40 00   2d+13:15:04.298  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+13:15:04.294  READ LOG EXT
  60 e0 d0 70 da 74 40 00   2d+13:15:00.430  READ FPDMA QUEUED
  60 e0 c8 90 d9 74 40 00   2d+13:15:00.430  READ FPDMA QUEUED

Error 5323 occurred at disk power-on lifetime: 61 hours (2 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 58 18 da 74 0b  Error: UNC at LBA = 0x0b74da18 = 192207384

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 d0 70 da 74 40 00   2d+13:15:00.430  READ FPDMA QUEUED
  60 e0 c8 90 d9 74 40 00   2d+13:15:00.430  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+13:15:00.426  READ LOG EXT
  60 e0 c8 70 da 74 40 00   2d+13:14:56.579  READ FPDMA QUEUED
  60 e0 c0 90 d9 74 40 00   2d+13:14:56.579  READ FPDMA QUEUED

Error 5322 occurred at disk power-on lifetime: 61 hours (2 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 58 18 da 74 0b  Error: UNC at LBA = 0x0b74da18 = 192207384

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 c8 70 da 74 40 00   2d+13:14:56.579  READ FPDMA QUEUED
  60 e0 c0 90 d9 74 40 00   2d+13:14:56.579  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+13:14:56.575  READ LOG EXT
  60 e0 c0 70 da 74 40 00   2d+13:14:52.726  READ FPDMA QUEUED
  60 e0 b8 90 d9 74 40 00   2d+13:14:52.726  READ FPDMA QUEUED

Error 5321 occurred at disk power-on lifetime: 61 hours (2 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 58 18 da 74 0b  Error: UNC at LBA = 0x0b74da18 = 192207384

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 c0 70 da 74 40 00   2d+13:14:52.726  READ FPDMA QUEUED
  60 e0 b8 90 d9 74 40 00   2d+13:14:52.726  READ FPDMA QUEUED
  60 e0 b0 b0 d8 74 40 00   2d+13:14:52.725  READ FPDMA QUEUED
  60 e0 a8 d0 d7 74 40 00   2d+13:14:52.725  READ FPDMA QUEUED
  60 e0 a0 f0 d6 74 40 00   2d+13:14:52.725  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%        26         451632936

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


I have been looking up a lot on stuff related to this mess but im a little lost.
Some people have suggested force writing to the bad sectors to reallocate them using -

Code:
sysctl kern.geom.debugflags=16
dd if=/dev/zero of=/dev/adaX bs=4096 count=1 seek=451632936 conv=noerror,sync


However this seems rather sketchy and I dont understand how the code will work as its writing nonsensical data to my pool.

Also I could just use 'zpool clear' but again I dont think this solves anything.

While I WILL be buying new drives and returning these ones I would really appreciate a little explanation on this issue.

Thank you :)

EDIT//- I am also going to run a long smart test on the 3rd Toshiba drive (as these all came from the same supplier) as I only have a Z2 raid I dont particually want problems with this one aswell in the mean time.

Could you also advise if I should resilver or restore from my backup(therefore destroy + recreate my pool)?
 
Last edited:
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
Oh yea, those two drives are bad. RMA them both. Read the SMART link in my tag line and pay attention to the red lines, IDs 5, 196, 197, 198.
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
You are in a precarious situation because your Seagate drives are a model that's known for early failure, and two of your Toshiba's are no good. Glad to see you have a backup.
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
ada1 and ada5 appear to be brand new drives--only 74 power-on hours. They both failed the only SMART self-test that's been run on them. Clearly you didn't do the recommended burn-in or testing on these drives before deploying them--I think you now see why that's recommended. Infant mortality is far from unheard-of when it comes to mechanical devices.

nemisisak said:
However this seems rather sketchy and I dont understand how the code will work as its writing nonsensical data to my pool.
Click to expand...

The idea is that by forcing the drive to write to the bad block, it will cause the drive to remap that block to another, good, sector. I haven't tried it (considering it, as you said, "rather sketchy"), but even if I trusted the procedure, I don't think I'd advise it in your case, as both of these drives have hundreds of bad sectors.

You could try pulling each drive from the pool (separately), running a complete badblocks test (which will take a long time, perhaps a couple of days), and then another long SMART test. It's possible that this will result in the drive remapping the bad sectors so it will complete the SMART test without errors. But since the drives are brand new, I'd go with @joeschmuck's advice and RMA them--ideally with the vendor you bought them from, so the replacements will be new drives, not refurbs. And then do a thorough burn-in test on the replacements before adding them to your pool.

What's your current pool layout? Can you post the output of zpool status?
 
T

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Save your data, then repurpose those Seagate drives for target practice. 5.56mm punches nice holes through them... 7.62x51 tends to make them come apart. That's all they're worth. Yes, I got burned by those drives :)
 
nemisisak

nemisisak

Explorer
Joined
Jun 19, 2015
Messages
69
oh nice link @joeschmuck . I will have a good read of that and also have an analyse of my 3rd drive.

@danb35 No I didnt burn them in, I will with the replacements. Yeah I understand that piece of code aims to relocate bad sectors but e.g. the seek parameter? how does that work and shouldnt the scrub reallocate the bad sectors anyway? but yeah obviously there are far to many bad sectors and these drives are next to duds. My current pool layout? I have 8 drives in RAIDZ2. 3rd toshiba drive status and zpool status below...
Code:
[root@freenas] ~# zpool status
  pool: AllDiskVolume
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 182M in 18h28m with 0 errors on Tue Jan 12 14:42:13 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        AllDiskVolume                                   ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/28e5ae28-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/29c63b3c-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/2a8a7857-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/2b2fce28-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/2bde8c7c-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/2ca8160e-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     1
            gptid/2d7f3771-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0
            gptid/2e4941c4-b71c-11e5-9ff6-d050995bee6c  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: scrub repaired 0 in 0h2m with 0 errors on Thu Dec 17 03:47:04 2015
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0  block size: 8192B configured, 8388608B native

errors: No known data errors


Code:
[root@freenas] ~# smartctl -a /dev/ada6
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA300
Serial Number:    55D62TUGS
LU WWN Device Id: 5 000039 fe3c2c4bd
Firmware Version: MX6OABB0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jan 13 19:28:15 2016 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x85) Offline data collection activity
                                        was aborted by an interrupting command f                                                                                                                                                             rom host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
Total time to complete Offline
data collection:                (21791) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                                                                                                                                                             ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 364) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_                                                                                                                                                             FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -                                                                                                                                                                    0
  2 Throughput_Performance  0x0005   140   140   054    Pre-fail  Offline      -                                                                                                                                                                    68
  3 Spin_Up_Time            0x0007   100   100   024    Pre-fail  Always       -                                                                                                                                                                    0
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -                                                                                                                                                                    3
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -                                                                                                                                                                    0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -                                                                                                                                                                    0
  8 Seek_Time_Performance   0x0005   124   124   020    Pre-fail  Offline      -                                                                                                                                                                    33
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -                                                                                                                                                                    96
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -                                                                                                                                                                    0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -                                                                                                                                                                    3
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -                                                                                                                                                                    4
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -                                                                                                                                                                    4
194 Temperature_Celsius     0x0002   166   166   000    Old_age   Always       -                                                                                                                                                                    36 (Min/Max 20/39)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -                                                                                                                                                                    0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -                                                                                                                                                                    0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -                                                                                                                                                                    0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -                                                                                                                                                                    0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA                                                                                                                                                             _of_first_error
# 1  Extended offline    Completed without error       00%        82         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


That looks to have completed without any errors so atleast I got one healthy drive :P I will order two more drives and might get four to fill my server so I dont have to destroy my pool again.

As for everyone else harping on about Seagate drives, its the Toshibas that are f'ing me over. The other Seagate died because it was about 5 years old and has changed through 2 computers and 2 NAS. Theres nothing wrong with Seagate drives, all drives fail and thats why we always have redundancy and back-ups :) if not its only yourself your screwing over.
 
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
ada6 looks fine like you said. My opinion is to run a Short SMART test each evening and Long SMART test once a week provided you leave your drives running 24/7.
 
nemisisak

nemisisak

Explorer
Joined
Jun 19, 2015
Messages
69
Yeah, daily short SMART tests are setup as I do have my rig on 24/7. Don't have any long tests tasks though. I will sort that out once I have got the new drives and created my new pool. Thanks for your help :)
 
Bidule0hm

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
If you only want one type of test then it's better to have weekly long tests than daily short tests, but why chose one when you can have both? :D
 
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
@Bidule0hm beat me to the punch, but the data is out there now.
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
nemisisak said:
Theres nothing wrong with Seagate drives, all drives fail
Click to expand...
There's nothing wrong with Seagate drives in general, but there are known issues with the ST3000DM001 model. The problems may not affect all revisions of that model, so it's possible yours will have a long, happy life :)
 
Status
Not open for further replies.

Similar threads

W
  • Locked
Is my HDD dying?
Replies
4
Views
2K
cobrakiller58
cobrakiller58
C
Do I need to change drive?
Replies
6
Views
1K
joeschmuck
joeschmuck
B
Areca 1882 for ZFS
Replies
6
Views
1K
dlavigne
D
F
  • Locked
Defective Hard Disk? Device: /dev/ada0, 1536 Currently unreadable (pending) sectors
2
Replies
26
Views
10K
Robert Trevellyan
R
J
Help debugging critical drive alerts and smartctl data
Replies
2
Views
899
jasn
J
Top