What does critical error "Currently unreadable (pending) sectors" mean?

[Brian Stretch]

So FreeNAS threw this at me this morning:
"CRITICAL: June 9, 2018, 9:57 a.m. - Device: /dev/ada3, 1 Currently unreadable (pending) sectors"
smartctl -t long returned:

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:	 CT1000MX500SSD1
Serial Number:	xxxxx
LU WWN Device Id: xxxxx
Firmware Version: M3CR010
User Capacity:	1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Jun  9 11:22:16 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection:		 (	0) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 (  30) minutes.
Conveyance self-test routine
recommended polling time:	 (   2) minutes.
SCT capabilities:			(0x0031)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   100   100   000	Pre-fail  Always	   -	   0
  5 Reallocated_Sector_Ct   0x0032   100   100   010	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   1750
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   17
171 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0
172 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0
173 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   2
174 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000	Pre-fail  Always	   -	   38
183 Runtime_Bad_Block	   0x0032   100   100   000	Old_age   Always	   -	   0
184 End-to-End_Error		0x0032   100   100   000	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
194 Temperature_Celsius	 0x0022   055   044   000	Old_age   Always	   -	   45 (Min/Max 0/56)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   100   100   000	Old_age   Always	   -	   0
202 Unknown_SSD_Attribute   0x0030   100   100   001	Old_age   Offline	  -	   0
206 Unknown_SSD_Attribute   0x000e   100   100   000	Old_age   Always	   -	   0
210 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0
246 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   967017884
247 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   15882755
248 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   42178842

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 ec 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ec 00 00 00 00 00 00 00	  00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00	  00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00	  00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00	  00:00:00.000  IDENTIFY DEVICE
  c8 00 00 00 00 00 00 00	  00:00:00.000  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	  1750		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Odd, but "Warning: ATA error count 0 inconsistent with error log pointer 1" didn't sound good so I put in an RMA with Crucial which they accepted right away... then it occurred to me that I should check the other 4 drives in the array and smartctl returns the same thing. False alarm? Why did FreeNAS flag the error on only one drive? Should I cancel the RMA? Panicked over nothing?

 

[Johnnie Black]

It's a firmware bug on all CX500 SSDs, you'll get a single pending sector from time to time, it will then return to 0, you can ignore that.

 

[Brian Stretch]

Thanks!

 

[Brian Stretch]

FWIW, the June 19th 2018 MX500 firmware update didn't fix this. Got the error on two of the five drives today.

 

[Baenwort]

So I have the Dec 18th 2018 firmware number M3CR023 on my MX500 and it is still giving this error intermittently.

Code:

smartctl -a /dev/ada3
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     CT1000MX500SSD1
Serial Number:    1850E1DDEA0E
LU WWN Device Id: 5 00a075 1e1ddea0e
Firmware Version: M3CR023
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Mar 20 20:14:56 2019 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  30) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x0031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       65
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
173 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       1
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000    Pre-fail  Always       -       38
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   070   050   000    Old_age   Always       -       30 (Min/Max 0/50)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
202 Unknown_SSD_Attribute   0x0030   100   100   001    Old_age   Offline      -       0
206 Unknown_SSD_Attribute   0x000e   100   100   000    Old_age   Always       -       0
210 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
246 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       920302578
247 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       14816707
248 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       29475861

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 ec 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  c8 00 00 00 00 00 00 00      00:00:00.000  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        64         -

 

[Johnnie Black]

Latest firmware still doesn't fix this.

 

[SamM]

I was directed to this thread after having pretty much the same issue. Do we know if Crucial aware of this problem, or if a fix in development?

 

[MikeyG]

Got the same thing today on 2 of my MX500 drives which are brand new.

Firmware Version: M3CR023

 

[SamM]

I live-chatted with Crucial about this. My objective was to find out if they're aware of the firmware bug and if they're working on a new firmware. I didn't expect much from a chat person, but they didn't have a tech support email listed...

Spoiler alert: The agent doesn't think this is a firmware issue...

Katy P (6/6/2019, 4:36:49 PM): Hello, you are chatting with Katy, how can I help you?
Katy P (6/6/2019, 4:40:01 PM): Hello, are you there?
Me (6/6/2019, 4:40:09 PM): Hello. I purchased 14 MX500 1000Gb SSD's for use in a FreeNAS server. Long story short, these drives randomly throw "smartd[58718]: Device: /dev/da## [SAT], 1 Currently unreadable (pending) sectors", where "da" represents a number of these SSD's
Me (6/6/2019, 4:41:49 PM): After doing some research, I hear that this is a firmware bug in these SSD's. My question is Crucial aware of these issues and working on a new firmware? The last firmware did not resolve the issue.
Katy P (6/6/2019, 4:42:17 PM): I am sorry to hear that. Give me a few moments to look into this.
Me (6/6/2019, 4:42:26 PM): Sure.
Katy P (6/6/2019, 4:44:59 PM): Okay so at the base the error looks to be indicating bad sectors on the drive. It is not uncommon for SSDs to show a number of bad sectors but that doesn't necessarily point to a fault on the drive. You can find our details on why this can happen on our Knowledge Base page here:

https://www.crucial.com/usa/en/my-ssd-has-bad-sectors

I would also recommend running our own dedicated Storage Executive software, which can evaluate the health status of your SSD and accurately read the SMART information. You can find details on this at the following link:

http://www.crucial.com/usa/en/support-storage-executive
Me (6/6/2019, 4:47:21 PM): For what it's worth, FreeNAS runs on FreeBSD (as opposed to Windows, so Storage Executive won't run), so this might be an issue with the OS rather than FreeNAS.

Also worth noting is that the drive passes smart diag tests without sector errors.
Katy P (6/6/2019, 4:49:52 PM): That is quite odd. But if you need to test the health of a drive you can always connect it externally to a Windows system and use our tool to check it.

As a side note, the Crucial MX500 is a consumer grade drive and is not intended to be used in servers. Use in servers can shorten the lifetime of the drive and can even void the warranty depending on the TBW.
Me (6/6/2019, 4:51:33 PM): I understand that, but it being the few drives with power loss protection, among other features, it should make an otherwise excellent NAS drive...
Katy P (6/6/2019, 4:52:03 PM): Did you have any further questions while we're here, or was there anything else I could help out with at the moment?
Me (6/6/2019, 4:52:54 PM): Server use or not, ultimately my question is if the dev team is working on a new firmware for this drive that addresses this issue. If the answer is no, that I'd like to add this into the suggestions box per say.
Me (6/6/2019, 4:53:10 PM): That's all.
Katy P (6/6/2019, 4:53:26 PM): I do not believe this is an issue with the SSDs firmware.
Me (6/6/2019, 4:53:53 PM): I see. Ok thank you.
Katy P (6/6/2019, 4:54:12 PM): If you have any future questions, or concerns feel free to visit us again at www.crucial.com/contact. You can also call us at 1-800-336-8915. Our hours are Monday-Friday, 7am-6pm (MST). Thank you for visiting Crucial, and have a beautiful day!

If you need to contact back your case number is 219299.

 

[Glorious1]

By the way, @Brian Stretch , 45 C seems too hot for a drive, although maybe it doesn't matter for SSDs, I don't know.

 

[MikeyG]

Is it possible to set attribute 197 to ignore for specific drives?

Also, could this be an issue with the smartd database not having complete information for crucial mx500 drives?

 

[Johnnie Black]

could this be an issue with the smartd database not having complete information for crucial mx500 drives?

No, it reports a pending sector, then it goes back to zero, it's the attribute's raw value that's changing.

 

[ssbarnea]

While I am not (yet) using FreeNAA, I am facing the same issue on CentOS, even after spending a ridiculous amount of time upgrading the firmware.

I am wondering how long it will take until Crucial will address this? Maybe we should start spamming them on twitter as nowdays they only care about brand visibility.

 

[joeschmuck]

By the way, @Brian Stretch , 45 C seems too hot for a drive, although maybe it doesn't matter for SSDs, I don't know.

Both posters here have a high temp of 56C and 50C, way too high in my opinion for a properly cooled system, BUT the technical specs for these drives is 0C to 70C operational temps. So this is probably not the issue.

Is it possible to set attribute 197 to ignore for specific drives?

Yes it is, in a round about way. You can run a script to run your SMART tests vice having FreeNAS use it's internal testing. Then you can send out the results and if you think it's a problem, change the Subject line of the email to "PROBLEM" or some other warning that catches your eye. I have a two scripts for my drives (short and long tests) that does something similar and you can use a script in the resources section to do the same type of thing, then modify it to your specifications. It's all very doable if you take the time to learn how to script and use what is already out there. Here is the link to the resources I was talking about.

As for any other fix, don't be waiting on Crucial or you could be waiting a long time. And as for the checking of ID 197, I would set my script up to ignore a value of "1" but not anything else. If you find that you are seeing higher values then evaluate the situation and if the values are up to "2", maybe ignore those too but if it starts creeping up, maybe it's not the drive for you? Or you could just ignore a higher value if you desire. But I would not ignore ID 197 as a rule. You could also compare previous data to current data and see if it ever goes back to "0". Lots you can do.

Good luck!

 

[SamM]

While I am not (yet) using FreeNAA, I am facing the same issue on CentOS, even after spending a ridiculous amount of time upgrading the firmware.

I am wondering how long it will take until Crucial will address this? Maybe we should start spamming them on twitter as nowdays they only care about brand visibility.

On this note, I emailed them on the issue. The good news is that they confirm it's effectively a false positive and that it should be reasonably safe to ignore. The bad news is that they seem to have no interest in fixing this 'The Boy Who Cried Wolf!' firmware.

Hello Sam,


Thank you for contacting Crucial Support.


The pending sector attribute is legacy number from traditional HDDs, where there could be multiple blocks pending because they’re quite slow at processing them. SSDs on the other hand are expected to have a pending sector at any given time, as this is its natural way of evaluating if a sector can be corrected or should be retired.

So the number you are seeing is entirely normal and should be ignored if any utility is flagging it.

Micron has a Linux distro of our Storage executive software you are free to download to verify the integrity of the SMART health if you would like, but I can tell you before even trying that it’s going to flag the drive as healthy, since the pending sector count of 1 is normal as previously discussed. This is assuming all other attributes are healthy on the drive, which they presumably are, otherwise SMARTCTL would have flagged them as well.

https://www.micron.com/products/solid-state-drives/storage-executive-software

Storage Executive Software - micron.com
About Micron Insight. Micron Insight brings you stories about how technology transforms information to enrich lives. Learn, imagine, innovate, solve, and gain insight on the technology trends of today and tomorrow from thought leaders around the world.
www.micron.com


If you have further questions, we welcome your reply. You can also visit our Contact page at http://www.crucial.com/usa/en/support-contact for other ways to reach us.

Regards,

[dude]
Micron Consumer Products Group
The Crucial Team
The memory and storage experts. ™
Contact us: http://www.crucial.com/usa/en/support-contact




------------------------



MX500 SSD's firmware often throwing 'Device: /dev/ada#, 1 Currently unreadable (pending) sectors' errors. Firmware needs fix to make MX500 drives less 'false-positive error' prone in systems like BSD.

So yeah, if you really want a firmware fix, ya might have to start doling out some negative PR...

 

[joeschmuck]

Not the answer anyone wants to see but it is an answer.

 

[dr0]

A few weeks ago I bought a 1TB MX500 for my personal system [i7-3770/32GB RAM/H77/GTX1050Ti/Win 7 x64 SP1] to replace a combo of a smaller SSD and an HDD. My MX500 also has this weird behavior when occasionally the drive reports attribute #197 (Current Pending Sector Count) to go from 0 to 1, but in a few minutes, it changes back to 0. Reallocation Event Count (#196) never changes and always stays at 0.

I have never seen anything like this with any SSD that I had a chance to use. My other SSDs have 10s of thousands of hours of active use and 10s of TB of lifetime writes but not a single one of them have pending sector(s) in the log files, including my trusty Crucial M4 that I've been using since 2011, whereas my MX500 has only 0.68TB lifetime writes and only 91 hours of active time but already had 3 events of pending sector. I wonder what causes this? And if it's indeed just a firmware bug then why Crucial hasn't fixed this to this day, after more than 18 month of MX500 being on the market?

 

[Ericloewe]

And if it's indeed just a firmware bug then why Crucial hasn't fixed this to this day, after more than 18 month of MX500 being on the market?

That's rather optimistic when speaking of firmware of any kind.

It might not strictly be a bug. The parameter might mean something other than what your tool is telling you. Crucial's tool should be right, smartmontools has a decent chance of being right. Or it might be normal operation with just a bit too much noise that doesn't make sense to expose to users.

 

[MikeyG]

I ended up setting -i 197 as a SMART option on the MX500s I have a couple days ago, and that seems to be preventing the log errors and emails from showing up. It's not ideal as I guess if a sector goes bad, I won't know about it. But I was getting so many false positives I'd never know anyway.

 

[MikeyG]

I ended up setting -i 197 as a SMART option on the MX500s I have a couple days ago, and that seems to be preventing the log errors and emails from showing up. It's not ideal as I guess if a sector goes bad, I won't know about it. But I was getting so many false positives I'd never know anyway.

Never mind, this doesn't seem to work for some reason.