SCSI sense: MEDIUM ERROR, Unrecovered read error

[IanWorthington]

I'm running badblocks on a bunch of 12 drives prior to putting them into service.

Three had already failed SMART self testing so badblocks errors were expected, and duly received, from those. One actually disconnected itself, is that usual for some type of errors?

One other disk, which had pass the self testing, started giving errors after 26 hours (one pass of a badblocks test on these 4TB drives is about 8 hours, iirc), and is producing messages such as these about twice per minute, though the error count is not increasing:

Code:

Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 cf 06 db 00 00 00 01 00 00 00                   
Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): CAM status: SCSI Status Error                                                   
Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): SCSI status: Check Condition                                                     
Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)                       
Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): Info: 0x1cf06db77                                                               
Apr 27 19:15:08 freenas kernel: (da4:mps0:0:4:0): Error 5, Unretryable error                                                       
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 cf 06 e0 00 00 00 01 00 00 00                   
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): CAM status: SCSI Status Error                                                   
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): SCSI status: Check Condition                                                     
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)                       
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): Info: 0x1cf06e0e0                                                               
Apr 27 19:15:41 freenas kernel: (da4:mps0:0:4:0): Error 5, Unretryable error  

Have I really got a FOURTH bad disk? Anything else I should check? Because that would make 4 out of 12 bad and I thought 3 was pretty unlikely...

 

[cyberjock]

It looks that way..

If they all came in the same box and someone dropped it during shipping, it can result in a bunch of failed disks.

 

[IanWorthington]

Thanks. Multiple boxes, but I didn't track what came in which.

Well that's a bugger. Getting disks at a reasonable price down here is not easy. I've got 3 new ones coming down from the US later this week, if any of those is bad I'm going to be SOL.

Bit odd it should start playing up on a pass other than 1.

 

[cyberjock]

Just an FYI.. Fedex dropped by my house and I kicked them all very hard for you. ;)

What I'd do though, is look at the SMART info on the disk(smartctl -a /dev/da4). Post it if you can't interpret it for yourself. Quite often when someone sees lots of failures in short order either they are all failing because of old age, being kicked, etc. or they all share a common problem(aka PSU that isn't putting out clean power, bad SATA cables, bad SATA controller, etc.)

So I'd probably stop and see if you can figure out if they *really* are failing or not.

 

[IanWorthington]

Ta, much appreciated!

Currently rerunning badblocks on ONLY da4, I'll see what it looks like after 24 hours.

Must admit I usually use gsmartcontrol to do the smart report analysis for me but I'll take a look. Any reason that isn't installed on on FreeNas? (lack of a gui at a guess?)

 

[cyberjock]

yeah.. no GUI for FreeNAS. ;)

and if I'm not mistaken gsmartcontrol simply uses smartctl and parses the output into a pretty GUI.

 

[IanWorthington]

Shame, that tool really helps in understanding the SMART data.

Another badblocks run confirms da4 has a smallish number of read errors near the top of its range. The LBAs in this second run don't include those in the (aborted) first run so I assume they may have been remapped, though see below.

smartctl -a /dev/da4 returns the following:

Code:

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p3 amd64] (local build)                                                       
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org                                                     
                                                                                                                                 
=== START OF INFORMATION SECTION ===                                                                                             
Device Model:    WDC WD40EFRX-68WT0N0                                                                                           
Serial Number:    WD-WCC4E0541819                                                                                               
LU WWN Device Id: 5 0014ee 25ecfaf0e                                                                                             
Firmware Version: 80.00A80                                                                                                       
User Capacity:    4,000,787,030,016 bytes [4.00 TB]                                                                             
Sector Sizes:    512 bytes logical, 4096 bytes physical                                                                         
Rotation Rate:    5400 rpm                                                                                                       
Device is:        Not in smartctl database [for details use: -P showall]                                                         
ATA Version is:  ACS-2 (minor revision not indicated)                                                                           
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                         
Local Time is:    Tue Apr 29 19:27:05 2014 PDT                                                                                   
SMART support is: Available - device has SMART capability.                                                                       
SMART support is: Enabled                                                                                                       
                                                                                                                                 
=== START OF READ SMART DATA SECTION ===                                                                                         
SMART overall-health self-assessment test result: PASSED                                                                         
                                                                                                                                 
General SMART Values:                                                                                                           
Offline data collection status:  (0x00) Offline data collection activity                                                         
                                        was never started.                                                                       
                                        Auto Offline Data Collection: Disabled.                                                 
Self-test execution status:      (  0) The previous self-test routine completed                                                 
                                        without error or no self-test has ever                                                   
                                        been run.                                                                               
Total time to complete Offline                                                                                                   
data collection:                (55260) seconds.                                                                                 
Offline data collection                                                                                                         
capabilities:                    (0x7b) SMART execute Offline immediate.                                                         
                                        Auto Offline data collection on/off support.                                             
                                        Suspend Offline collection upon new                                                     
                                        command.                                                                                 
                                        Offline surface scan supported.                                                         
                                        Self-test supported.                                                                     
                                        Conveyance Self-test supported.                                                         
                                        Selective Self-test supported.                                                           
SMART capabilities:            (0x0003) Saves SMART data before entering                                                         
                                        power-saving mode.                                                                       
                                        Supports SMART auto save timer.                                                         
Error logging capability:        (0x01) Error logging supported.                                                                 
                                        General Purpose Logging supported.                                                       
Short self-test routine                                                                                                         
recommended polling time:        (  2) minutes.                                                                                 
Extended self-test routine                                                                                                       
recommended polling time:        ( 552) minutes.                                                                                 
Conveyance self-test routine                                                                                                     
recommended polling time:        (  5) minutes.                                                                                   
SCT capabilities:              (0x703d) SCT Status supported.                                                                     
                                        SCT Error Recovery Control supported.                                                     
                                        SCT Feature Control supported.                                                             
                                        SCT Data Table supported.                                                                 
                                                                                                                                   
SMART Attributes Data Structure revision number: 16                                                                               
Vendor Specific SMART Attributes with Thresholds:                                                                                 
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                   
  1 Raw_Read_Error_Rate    0x002f  166  112  051    Pre-fail  Always      -      41370                                       
  3 Spin_Up_Time            0x0027  178  177  021    Pre-fail  Always      -      8066                                       
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      18                                         
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0                                           
  7 Seek_Error_Rate        0x002e  200  200  000    Old_age  Always      -      0                                           
  9 Power_On_Hours          0x0032  100  100  000    Old_age  Always      -      131                                         
10 Spin_Retry_Count        0x0032  100  253  000    Old_age  Always      -      0                                           
11 Calibration_Retry_Count 0x0032  100  253  000    Old_age  Always      -      0                                           
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      18                                         
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      17                                         
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      164                                         
194 Temperature_Celsius    0x0022  113  105  000    Old_age  Always      -      39                                         
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0                                           
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      10                                         
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0                                           
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0                                           
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      1                                           
                                                                                                                                   
SMART Error Log Version: 1                                                                                                         
No Errors Logged                                                                                                                   
                                                                                                                                   
SMART Self-test log structure revision number 1                                                                                   
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                   
# 1  Extended offline    Completed without error      00%        40        -                                                     
# 2  Short offline      Completed without error      00%        30        -                                                     
# 3  Conveyance offline  Completed without error      00%        29        -                                                     
                                                                                                                                   
SMART Selective self-test log data structure revision number 1                                                                     
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                       
    1        0        0  Not_testing                                                                                               
    2        0        0  Not_testing                                                                                               
    3        0        0  Not_testing                                                                                               
    4        0        0  Not_testing                                                                                               
    5        0        0  Not_testing                                                                                               
Selective self-test flags (0x0):                                                                                                   
  After scanning selected spans, do NOT read-scan remainder of disk.                                                               
If Selective self-test is pending on power-up, resume after 0 minute delay.      

If I understand this correctly, the important counts are:

Code:

  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0     
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      10

I clearly don't understand these fully as I would not have expected to see "Reallocated Sector Count" as 0, when sectors clearly have been remapped. And "Current Pending Sector" as 10 doesn't really tally with what badblocks is telling me right now either (33 read errors).

 

[cyberjock]

Actually, Reallocated_Sector_Ct is the number of sectors that have been reallocated in the past. The Current_Pending_Sector tells you how many the drive has already recognized as failed, but hasn't remapped.

The fact that badblocks finds 33 errors and the drive only recognizes 10 means there 22 sectors that are bad that the drive didn't even recognize as failed. In short, that's a drive I would not trust with data.

To make matters worse it passed the last long smart test, which means it doesn't qualify for an RMA right now. Unless you can run a SMART test and get it to fail you are stuck with the drive. If you bought it from Newegg or something I'd return it for an exchange.

 

[dlavigne]

Any reason that isn't installed on on FreeNas? (lack of a gui at a guess?)

Noone has asked for it? If you create a feature request at bugs.freenas.org, post the issue number here.

 

[IanWorthington]

Noone has asked for it? If you create a feature request at bugs.freenas.org, post the issue number here.

Would you even consider accepting it? Installing a GUI would seem a major change to FreeNAS when the direction seems to be to provide monitoring tools through the management web page.

 

[IanWorthington]

Actually, Reallocated_Sector_Ct is the number of sectors that have been reallocated in the past. The Current_Pending_Sector tells you how many the drive has already recognized as failed, but hasn't remapped.

The fact that badblocks finds 33 errors and the drive only recognizes 10 means there 22 sectors that are bad that the drive didn't even recognize as failed. In short, that's a drive I would not trust with data.

To make matters worse it passed the last long smart test, which means it doesn't qualify for an RMA right now. Unless you can run a SMART test and get it to fail you are stuck with the drive. If you bought it from Newegg or something I'd return it for an exchange.

I think these drives hide the number of reallocated sectors. Current stats are:

Code:

5   Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0                                                                                 
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0 

I'm now running badblocks just on the top 2%, which seems to be where the errors are, which reduces the time for each each pass from 18 hours to 40 minutes.

There seem to be additional bad sectors reported by badblocks on each pass, which confirms in my mind that the errors are indeed growing, though I have a feeling that there are more "(da4:mps0:0:4:0): Error 5, Unretryable error" messages than badblocks reports read errors.

Unfortunately, due to Amazon delivery cock ups, I'm out of time to return this drive. The best I can hope for at this moment, I think, is to fill up the spare area by repeated test cycles and push the drive into selftest failure.

I don't trust it either, but I think I might be able to get away with using it in the raidz3 array until I can get a replacement (probably a couple of months), as the damaged area is consistently in the top 2%. I assume zfs allocates from the bottom up?

 

[bigwavealex]

If they are under warranty you should be able to get exchanges, they will be refurbs though. My experience with multiple drive companies has been they will give you an exchange with evidence from most drive test tools. Seatools works on pretty much any drive and a pic from that showing failed drive has never failed to get a return for me. Start by going on line and filling out an RMA form. We keep a dedicated machine just for testing drives, to knock out any other issues.

 

[cyberjock]

That Raw_Read_Error_Rate should be good enough to quality for a return.. it should be zero for that model/brand of disk.

 

[IanWorthington]

Thanks guys. Ran badblocks for a few more days on the top 2% and selftest has now reached the conclusion that the drive's shite afterall :)

So I now have 4 drives to RMA on my next trip north to the US. But will be running without a cold spare until then as I only ordered 3 replacement drives. They all passed extended self test and now have a week of badblocks to look forward to.

 

[IanWorthington]

Noone has asked for it? If you create a feature request at bugs.freenas.org, post the issue number here.

Done. https://bugs.freenas.org/issues/4940

 

[IanWorthington]

And, as anticipated, rejected..

In other news, WD have responded to my email about creeping badblocks errors, and said that that IS sufficient to satisfy RMA requirements. Academic now, but for anyone who wants to know.

i

 

[survive]

Done. https://bugs.freenas.org/issues/4940

Hi IanWorthington,

I requested this:

https://bugs.freenas.org/issues/1565

a while ago which would accomplish pretty much the same thing.

-Will

 

[IanWorthington]

Thanks Will.