Which disk is bad?

[John M. Długosz]

I'm getting "alert" icon flashing, and it tells me "one or more devices has experienced an unrecoverable error". Which one? I've looked at SMART screens and turned on periodic testing but can't find any log of what the error was. If anything, the console view (green screen at bottom of window) gives me a different unrelated error about SMART: smartd: Device /dev/ada1, 8 Currently unreadable (pending) sectors every time it wakes up.

But this screen:

I think answers the question. Does the non-zero value under "Checksum" indicate that this drive is the one having reported problems?

 

[jgreco]

Well, it definitely bears looking-at. Post the output of "smartctl -a da1" and also "smartctl -a ada1" in CODE tags here and lots of people will be happy to offer an opinion.

 

[John M. Długosz]

for da1

Code:

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Desktop HDD.15
Device Model: ST4000DM000-1F2168
Serial Number: W300JPD9
LU WWN Device Id: 5 000c50 069ca9271
Firmware Version: CC52
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jan 6 10:43:54 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 602) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 522) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x1085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 113 099 006 Pre-fail Always - 52893952
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 092 092 020 Old_age Always - 9011
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 077 060 030 Pre-fail Always - 54879750
9 Power_On_Hours 0x0032 079 079 000 Old_age Always - 19207
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 22
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 098 000 Old_age Always - 0 0 12
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 066 059 045 Old_age Always - 34 (Min/Max 22/35)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 22
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 528156
194 Temperature_Celsius 0x0022 034 041 000 Old_age Always - 34 (0 20 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 8902
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 13867h+01m+24.347s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 9996602286
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 86572777103

SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 18936 hours (789 days + 0 hours)
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 18936 hours (789 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 71 09 a9 00 80 40 Device Fault; Error: ABRT at LBA = 0x008000a9 = 8388777

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 10 ff ff ff 4f 00 32d+09:06:32.205 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 32d+09:06:26.061 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 32d+09:06:26.054 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 32d+09:06:26.053 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 32d+09:06:26.052 WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 19185 -
# 2 Short offline Completed without error 00% 19161 -
# 3 Short offline Completed without error 00% 19137 -
# 4 Short offline Completed without error 00% 19113 -
# 5 Short offline Completed without error 00% 19089 -
# 6 Conveyance offline Completed without error 00% 9 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

and for dev/ada1

Code:

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E0100460
LU WWN Device Id: 5 0014ee 2b3ad0dc8
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Jan 6 10:47:15 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (57360) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 573) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 1041
3 Spin_Up_Time 0x0027 179 179 021 Pre-fail Always - 8033
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1552
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19494
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 30
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 15
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1405092
194 Temperature_Celsius 0x0022 117 110 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 19473 204540806
# 2 Short offline Completed: read failure 90% 19449 204540806
# 3 Short offline Completed without error 00% 19425 -
# 4 Short offline Completed: read failure 90% 19401 204540806
# 5 Short offline Completed without error 00% 19377 -
# 6 Extended offline Completed without error 00% 12 -
# 7 Conveyance offline Completed without error 00% 1 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[danb35]

Looks like you just set up scheduled SMART tests on these drives a few days ago, and no long test has been run yet. Make sure those are scheduled too, particularly with da1; I'd be interested in seeing the results after a long test. But here's what I'm seeing so far:

I'm concerned about the load cycle count on da1--over 500,000 load cycles is a lot, and way more (i.e., several orders of magnitude more) than you should see with these drives in a server. If they were repurposed desktop drives, the number might not be too scary, but even then I expect you're near, if not over, the lifetime spec. I'm also concerned about attribute 199, the UDMA CRC error rate. That may have been what resulted in the checksum errors during your last scrub.

As to ada1, the drive is failing SMART tests, so replace it. Three out of the four short tests have failed with a read error. In addition, it has a load cycle count of 1.4M, which, IIRC, is more than double the lifetime spec for that model drive. You should check that drive, and your other WD drives, with WDIDLE3.EXE to make sure the park timer is either disabled (ideally), or at least set to 5 minutes. A Red drive shouldn't need that (it should come that way from the factory), but I remember hearing that a faulty batch came out a while back. This drive should be under warranty, so you can RMA it.

Make sure you have a regular schedule of short and long SMART self-tests running on all drives. Unless something's changed recently, you'll need to re-select all your drives after you replace ada1 for the tests to run on all of them.

 

[John M. Długosz]

Looks like you just set up scheduled SMART tests on these drives a few days ago, and no long test has been run yet.

I only set up for a short test, and was looking for where to find the result.

Make sure those are scheduled too, particularly with da1; I'd be interested in seeing the results after a long test.

OK, I added a long test once a day, hopefully the first to be scheduled shortly.

I'm concerned about the load cycle count on da1--over 500,000 load cycles is a lot, and way more (i.e., several orders of magnitude more) than you should see with these drives in a server. If they were repurposed desktop drives, the number might not be too scary, but even then I expect you're near, if not over, the lifetime spec. I'm also concerned about attribute 199, the UDMA CRC error rate. That may have been what resulted in the checksum errors during your last scrub.

All the drives are the same age, purchased specifically for this file server and never used in anything else. They should all have identical usage, being part of the same stripe set.

I do see that ada1 has the "HDD Standby" time set to 5 minutes instead of 20 like the rest of them. But from the mains power draw I don't think the drives ever stop, as the power demand stays constant. I gave up trying to get it to "sleep" when I'm not using it (expecting to go from 5W to ½W per drive).

As to ada1, the drive is failing SMART tests, so replace it. Three out of the four short tests have failed with a read error. In addition, it has a load cycle count of 1.4M, which, IIRC, is more than double the lifetime spec for that model drive. You should check that drive, and your other WD drives, with WDIDLE3.EXE to make sure the park timer is either disabled (ideally), or at least set to 5 minutes. A Red drive shouldn't need that (it should come that way from the factory), but I remember hearing that a faulty batch came out a while back. This drive should be under warranty, so you can RMA it.

Make sure you have a regular schedule of short and long SMART self-tests running on all drives. Unless something's changed recently, you'll need to re-select all your drives after you replace ada1 for the tests to run on all of them.

So you're saying that two drives are of concern?

 

[danb35]

OK, I added a long test once a day, hopefully the first to be scheduled shortly.

Setting it daily just to make sure that it runs soon is fine, but in general daily is far too often to run a long test. Somewhere between weekly and monthly is more common.

So you're saying that two drives are of concern?

Yes, though ada1 much more so than da1.

 

[John M. Długosz]

Setting it daily just to make sure that it runs soon is fine, but in general daily is far too often to run a long test. Somewhere between weekly and monthly is more common.

Yes, though ada1 much more so than da1.

da1 is "worse" because the SMART tests are failing?

So what's the issue with ada1 and "2.73k" in the display?

 

[danb35]

No, the SMART tests are failing on ada1, and thus I'm saying it is of greater concern.