Have run several scrubs, still have errors, cant offline disk

[CodeWarrior]

I have a single zpool on 6 disks. A couple days ago I started seeing SMART error reports coming through my email. I logged in, and tried to offline the disk, and was faced with the no valid replicas error message. I did a scrub which found 40-50 errors. After it finished I did a zpool status -v and it still reports 4 errors:

errors: Permanent errors have been detected in the following files:

<metadata>:<0x62>
<metadata>:<0xb2>
<metadata>:<0xcb>
DataLore:<0x0>

I still can't offline the disk which now has a Reallocated_Sector_Ct hovering near 0.

Any thoughts?

 

[cyberjock]

Can you post the output of your smartctl commands for all disks as well as "zpool status".

 

[CodeWarrior]

OK looks like another is going out too, so I will post two outputs....

OUTPUT 1

smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net


=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.12
Device Model: ST31000528AS
Serial Number: 5VP2VQAG
LU WWN Device Id: 5 000c50 01e5ee88b
Firmware Version: CC38
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sun Jul 21 17:09:24 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x82)Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0)The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection:( 609) seconds.
Offline data collection
capabilities:(0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003)Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01)Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time:( 1) minutes.
Extended self-test routine
recommended polling time:( 175) minutes.
Conveyance self-test routine
recommended polling time:( 2) minutes.
SCT capabilities: (0x103f)SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 103 099 006 Pre-fail Always - 144374178
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 102
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 6
7 Seek_Error_Rate 0x000f 072 060 030 Pre-fail Always - 16160133
9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8620
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 51
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 075 075 000 Old_age Always - 25
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 4295032833
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 062 043 045 Old_age Always In_the_past 38 (0 59 39 24 0)
194 Temperature_Celsius 0x0022 038 057 000 Old_age Always - 38 (0 22 0 0 0)
195 Hardware_ECC_Recovered 0x001a 030 017 000 Old_age Always - 144374178
197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 94
198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 94
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 235175229268531
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 4224382411
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3212057407

SMART Error Log Version: 1
ATA Error Count: 24 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 24 occurred at disk power-on lifetime: 8454 hours (352 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 55 ff ff ff 4f 00 07:16:12.014 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:08.850 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:05.736 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:02.564 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:59.494 READ DMA EXT

Error 23 occurred at disk power-on lifetime: 8454 hours (352 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 55 ff ff ff 4f 00 07:16:08.850 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:05.736 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:02.564 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:59.494 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.490 READ DMA EXT

Error 22 occurred at disk power-on lifetime: 8454 hours (352 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 55 ff ff ff 4f 00 07:16:05.736 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:16:02.564 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:59.494 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.490 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.488 READ DMA EXT

Error 21 occurred at disk power-on lifetime: 8454 hours (352 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 55 ff ff ff 4f 00 07:16:02.564 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:59.494 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.490 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.488 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:58.980 READ DMA EXT

Error 20 occurred at disk power-on lifetime: 8454 hours (352 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 55 ff ff ff 4f 00 07:15:59.494 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.490 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:59.488 READ DMA EXT
25 00 55 ff ff ff 4f 00 07:15:58.980 READ DMA EXT
25 00 56 ff ff ff 4f 00 07:15:58.969 READ DMA EXT

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 8617 -
# 2 Short offline Completed without error 00% 8611 -
# 3 Short offline Completed without error 00% 8605 -
# 4 Short offline Completed without error 00% 8600 -
# 5 Short offline Completed without error 00% 8593 -
# 6 Short offline Completed without error 00% 8585 -
# 7 Short offline Completed without error 00% 8579 -
# 8 Short offline Completed without error 00% 8573 -
# 9 Short offline Completed without error 00% 8567 -
#10 Short offline Completed without error 00% 8561 -
#11 Short offline Completed without error 00% 8555 -
#12 Short offline Completed without error 00% 8549 -
#13 Short offline Completed without error 00% 8543 -
#14 Short offline Completed without error 00% 8537 -
#15 Short offline Completed without error 00% 8533 -
#16 Short offline Completed without error 00% 8524 -
#17 Short offline Completed without error 00% 8518 -
#18 Short offline Completed without error 00% 8512 -
#19 Short offline Completed without error 00% 8506 -
#20 Short offline Completed without error 00% 8500 -
#21 Short offline Completed without error 00% 8494 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

OUTPUT 2

smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.11
Device Model: ST31500341AS
Serial Number: 9VS0V8XL
LU WWN Device Id: 5 000c50 03cb04a4b
Firmware Version: CC1H
User Capacity: 1,500,301,910,016 bytes [1.50 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sun Jul 21 17:09:33 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82)Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249)Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection:( 617) seconds.
Offline data collection
capabilities:(0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003)Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01)Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time:( 1) minutes.
Extended self-test routine
recommended polling time:( 298) minutes.
Conveyance self-test routine
recommended polling time:( 2) minutes.
SCT capabilities: (0x103f)SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 103 070 006 Pre-fail Always - 5655097
3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 26
5 Reallocated_Sector_Ct 0x0033 100 100 016 Pre-fail Always - 20
7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 13017997
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 2109
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 26
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 004 004 000 Old_age Always - 96
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 082 082 000 Old_age Always - 18
190 Airflow_Temperature_Cel 0x0022 050 049 045 Old_age Always - 50 (Min/Max 23/51)
194 Temperature_Celsius 0x0022 050 051 000 Old_age Always - 50 (0 21 0 0 0)
195 Hardware_ECC_Recovered 0x001a 055 026 000 Old_age Always - 5655097
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 3
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 3
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 19073949763644
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2176414396
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 78828530

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Self-test routine in progress 90% 2109 -
# 2 Short offline Interrupted (host reset) 30% 2087 -
# 3 Short offline Completed without error 00% 2076 -
# 4 Short offline Completed without error 00% 2070 -
# 5 Short offline Completed without error 00% 2064 -
# 6 Short offline Completed without error 00% 2058 -
# 7 Short offline Completed without error 00% 2052 -
# 8 Short offline Completed without error 00% 2046 -
# 9 Short offline Completed without error 00% 2040 -
#10 Short offline Completed without error 00% 2034 -
#11 Short offline Completed without error 00% 2028 -
#12 Short offline Completed without error 00% 2022 -
#13 Short offline Completed without error 00% 2011 -
#14 Short offline Completed without error 00% 2005 -
#15 Short offline Completed without error 00% 1999 -
#16 Short offline Completed without error 00% 1993 -
#17 Short offline Completed without error 00% 1987 -
#18 Short offline Completed without error 00% 1981 -
#19 Short offline Completed without error 00% 1975 -
#20 Short offline Completed without error 00% 1969 -
#21 Short offline Completed without error 00% 1963 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[cyberjock]

Yeah, the first disk is bad. The second disk isn't in perfect condition, but doesn't warrant a replacement yet. Id watch it closely. If current_pending_sector and offline_uncorrectable start changing you should consider replacing that disk.

You can also do a badblocks test. If you search the forum there's a post that provides a good badblocks command to run. The test is destructive and can be time consuming, but I consider the badblocks test to be the most thorough test you can perform.

 

[CodeWarrior]

Alright. I will have to try that in a day or two. In the meantime I will just keep the NAS turned off. We have a lot of things going on right now, so I don't have a lot of time to mess with it.