SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read

[Chronic]

Started receiving these messages in my emails a few days back:

Code:

(da21:mps1:0:16:0): READ(10). CDB: 28 00 e8 71 f3 d0 00 00 18 00 length 12288 SMID 575 terminated ioc 804b scsi 0 state 0 xfer 0
(da21:mps1:0:16:0): READ(10). CDB: 28 00 e8 71 f3 b0 00 00 20 00
(da21:mps1:0:16:0): CAM status: SCSI Status Error
(da21:mps1:0:16:0): SCSI status: Check Condition
(da21:mps1:0:16:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da21:mps1:0:16:0): Info: 0xe871f3b0
(da21:mps1:0:16:0): Error 5, Unretryable error

Everything's still running smoothly though. Trying to decipher the smartctl info for /dev/da21 is out of my league so I'm hoping someone here can advise:

Code:

smartctl -a /dev/da21

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red (AF)
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0850335
LU WWN Device Id: 5 0014ee 6040301bb
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Oct 28 13:29:49 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (41160) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 413) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       46
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       5849
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0032   071   071   000    Old_age   Always       -       389937
194 Temperature_Celsius     0x0022   114   110   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   199   000    Old_age   Offline      -       31

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      5848         -
# 2  Short offline       Completed without error       00%      5838         -
# 3  Short offline       Completed without error       00%      5814         -
# 4  Extended offline    Completed without error       00%      5804         -
# 5  Extended offline    Aborted by host               90%      5797         -
# 6  Extended offline    Completed without error       00%      5775         -
# 7  Short offline       Completed without error       00%      5765         -
# 8  Extended offline    Completed without error       00%      5754         -
# 9  Short offline       Completed without error       00%      5743         -
#10  Extended offline    Completed without error       00%      5728         -
#11  Short offline       Completed without error       00%      5717         -
#12  Extended offline    Completed without error       00%      5706         -
#13  Short offline       Completed without error       00%      5694         -
#14  Extended offline    Completed without error       00%      5680         -
#15  Short offline       Completed without error       00%      5670         -
#16  Short offline       Completed without error       00%      5646         -
#17  Extended offline    Completed without error       00%      5608         -
#18  Short offline       Completed without error       00%      5598         -
#19  Extended offline    Completed without error       00%      5584         -
#20  Short offline       Completed without error       00%      5574         -
#21  Extended offline    Completed without error       00%      5560         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

As far as I can see everything seems to be ok, nothing failed. Hard to RMA a drive unless I can point to something that's not as it should be.


Thanks in advance.

 

[cyberjock]

Things look fine on the surface, but your values for item 1 and 200 suggest your drive is starting to have problems. Have you recently moved the disk in a way that there might be excessive vibrations?

 

[Chronic]

Things look fine on the surface, but your values for item 1 and 200 suggest your drive is starting to have problems. Have you recently moved the disk in a way that there might be excessive vibrations?

Nope, nothing's been moved since it was installed.

 

[Ericloewe]

Well, in that case all you can do is keep an eye on it and have a new hard drive ready on short notice.

It's hard to RMA a drive that says it's not dead, unless the undertaker does you the favor of hitting it on the head.

 

[mjt5282]

nearly 400,000 Load_Cycle_Count's seems kind of high... with the raw read errors, it could also be a failing disk cable. I just checked my WD Red 3Tb's and Caviar Green 3Tb's with a short script :

for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ; do echo $i; sudo smartctl -a /dev/da$i | grep Load_Cycle_Count ; done

0
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 110
1
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 104
2
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 106
3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 4
4
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1331
5
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1224
6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1042
7
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 952
8
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 11975
9
193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 8000
10
193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 7907
11
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 107
12
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 306
13
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 4
14
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 4
15
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 4


The ones with the high values are recycled Caviar Green's that I didn't patch with the required code as soon I as should have ...

 

[HoneyBadger]

nearly 400,000 Load_Cycle_Count's seems kind of high... with the raw read errors, it could also be a failing disk cable. I just checked my WD Red 3Tb which has 4,650 power on hours, and it has a load cycle count of 110.

This jumped out at me too. It's not as bad as the "every few seconds" of a WD Green but that's a high LCC in my mind.

 

[Ericloewe]

This jumped out at me too. It's not as bad as the "every few seconds" of a WD Green but that's a high LCC in my mind.

Well, they're rated for 600 000 cycles, so a few hundred won't be a problem.

 

[HoneyBadger]

Well, they're rated for 600 000 cycles, so a few hundred won't be a problem.

A few hundred cycles no. A few hundred thousand is a big problem, especially when they're increasing at the rate of ~1600/day. This drive has a little over a year of power-on time, and at the current rate he'll burn through the rest of his rated LCC in four months, at which point if it's dead, WD can deny RMA.

 

[Ericloewe]

A few hundred cycles no. A few hundred thousand is a big problem, especially when they're increasing at the rate of ~1600/day. This drive has a little over a year of power-on time, and at the current rate he'll burn through the rest of his rated LCC in four months, at which point if it's dead, WD can deny RMA.

Reading comprehension fail. My bad. 400 000 is definitely cause for concern.

 

[Chronic]

Damn Western Digital... I have 22 of these 3TB WD Reds and now it seems that a lot of them are still suffering from the parking of heads every friggin' 8 seconds or something. Way to go killing my drives WD!

Code:

0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

1
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

2
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

3
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

4
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

5
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

6
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1540

8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1910

9
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1886

10
193 Load_Cycle_Count        0x0032   068   068   000    Old_age   Always       -       398716

11
193 Load_Cycle_Count        0x0032   072   072   000    Old_age   Always       -       386064

12
193 Load_Cycle_Count        0x0032   093   093   000    Old_age   Always       -       321844

13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

14
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

15
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1885

16
193 Load_Cycle_Count        0x0032   005   005   000    Old_age   Always       -       587082

17
193 Load_Cycle_Count        0x0032   009   009   000    Old_age   Always       -       573034

18
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       1109225

19
193 Load_Cycle_Count        0x0032   029   029   000    Old_age   Always       -       514498

20
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0

21
193 Load_Cycle_Count        0x0032   070   070   000    Old_age   Always       -       390295

:'(

 

[Ericloewe]

Do they carry a manufacturing date of late 2013/early 2014? There was a shortish period where the Reds came with the wrong factory configuration.

 

[Chronic]

I'm not sure but that does sound right. I'll check when I'm back at home.
The question is, does this warrant an RMA according to WD? That's 8 drives that have to be replaced if so.

Definitely not looking forward to changing all those drives. :D

 

[cyberjock]

No, WD provided a firmware update quite a while back that allegedly fixed the problem.

I will put things in context here though. We've seen people with 2 million load cycles and the drive performs just fine. Yes, they may be rated for 250k or 650k (depending on the model and year manufactured the number does change) but we're talking about statistical likelihood here. You might never have a problem with those with unusually high load counts or they could be the first to fail. Most of what people are doing on the forum is mitigating risk. Be it RAIDZ3, 2 or 3 backups, off-site backups, or a load_cycle count. The best you can do at this point is fix the problem and try to do your best to mitigate the risk you have with backups and such. There's no evidence that once you exceed the designed criteria that the failure rates go up and its directly attributable to it. But, anyone that's got an engineering background knows that you always over-engineer things and always try not to exceed your design criteria.

So smile, make sure you've prevented these drives from continuing to increase out of control like they have been, do backups, and smile.

 

[Chronic]

WD is letting me RMA the drive in question at least. As for the rest, I'll spend tomorrow evening pulling out and running wdidle3 on the ones with excessive LCC's. :)

 

[cyberjock]

Wow. That's good on WD. Glad to see they are taking care of you like that. Not that I'm bashing WD (I use and recommend them) but I wouldn't have expected that.