Currently unreadable (pending) sectors

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
I know this has come up on several occasions - however I have three drives, all reporting the same problem in a single pool. All the drives are SSD's. The usual response is that the drive is failing, swap it out. BUT I have run an extended test (on all the SSD's) and the smart tests seem to indicate nothing is wrong. I have moved all the VM's off the pool, and then destroyed the pool. BTW the zpool status showed no errors. The only place the errors were appearing are in TrueNAS at the top as critical errors

Results below for one drive - the other two have similar results

drives are split across different controllers, 2 on native SATA ports and one on a SAS port. Only SSD's are giving issues and none of the HDD's are having an issue.

I have noticed that when the NAS boots I get two messages
Nov 24 09:22:12 freenas 1 2020-11-24T09:22:12.491802+00:00 freenas.sendarian.co.uk smartd 3858 - - Device: /dev/da1 [SAT], previous self-test completed with error (read test element)
Nov 24 09:22:12 freenas 1 2020-11-24T09:22:12.494050+00:00 freenas.sendarian.co.uk smartd 3858 - - Device: /dev/da1 [SAT], previous self-test completed with error (read test element)

BUT the smartstats don't seem to agree

TrueNAS is TrueNAS-12.0-RELEASE

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
smartctl -a /dev/ada0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron BX/MX1/2/3/500, M5/600, 1100 SSDs
Device Model: CT1000MX500SSD1
Serial Number: 2011E293E907
LU WWN Device Id: 5 00a075 1e293e907
Firmware Version: M3CR032
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Nov 24 09:54:10 2020 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 468
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 31
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 099 099 000 Old_age Always - 18
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 16
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 29
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 081 049 000 Old_age Always - 19 (Min/Max 0/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
202 Percent_Lifetime_Remain 0x0030 099 099 001 Old_age Offline - 1
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 10512769899
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 86182698
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 120431466

SMART Error Log Version: 1
Invalid Error Log index = 0x15 (T13/1321D rev 1c Section 8.41.6.8.2.2 gives valid range from 1 to 5)

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 468 -
# 2 Short offline Completed without error 00% 467 -
# 3 Short offline Completed without error 00% 459 -
# 4 Short offline Completed without error 00% 446 -
# 5 Short offline Completed without error 00% 432 -
# 6 Extended offline Completed without error 00% 431 -
# 7 Short offline Completed without error 00% 426 -
# 8 Short offline Completed without error 00% 413 -
# 9 Short offline Completed without error 00% 356 -
#10 Short offline Completed without error 00% 314 -
#11 Short offline Completed without error 00% 309 -
#12 Short offline Completed without error 00% 295 -
#13 Short offline Completed without error 00% 258 -
#14 Short offline Completed without error 00% 242 -
#15 Extended offline Completed without error 00% 233 -
#16 Short offline Completed without error 00% 232 -
#17 Short offline Completed without error 00% 220 -
#18 Short offline Completed without error 00% 193 -
#19 Short offline Completed without error 00% 174 -
#20 Short offline Completed without error 00% 163 -
#21 Short offline Completed without error 00% 153 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
-------------------------------------------------------------------------------------------------------------------------------------------
root@freenas[~]# smartctl -a /dev/da0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron BX/MX1/2/3/500, M5/600, 1100 SSDs
Device Model: CT1000MX500SSD1
Serial Number: 2016E29BF768
LU WWN Device Id: 5 00a075 1e29bf768
Firmware Version: M3CR032
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Nov 24 09:58:15 2020 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 493
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 31
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 099 099 000 Old_age Always - 18
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 29
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 27
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 083 053 000 Old_age Always - 17 (Min/Max 0/47)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
202 Percent_Lifetime_Remain 0x0030 099 099 001 Old_age Offline - 1
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 10506090491
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 86125768
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 124294096

SMART Error Log Version: 1
Invalid Error Log index = 0x15 (T13/1321D rev 1c Section 8.41.6.8.2.2 gives valid range from 1 to 5)

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 493 -
# 2 Short offline Completed without error 00% 492 -
# 3 Short offline Completed without error 00% 483 -
# 4 Short offline Completed without error 00% 471 -
# 5 Short offline Completed without error 00% 457 -
# 6 Extended offline Completed without error 00% 456 -
# 7 Short offline Completed without error 00% 451 -
# 8 Short offline Completed without error 00% 438 -
# 9 Short offline Completed without error 00% 380 -
#10 Short offline Completed without error 00% 336 -
#11 Short offline Completed without error 00% 332 -
#12 Short offline Completed without error 00% 317 -
#13 Short offline Completed without error 00% 274 -
#14 Short offline Completed without error 00% 258 -
#15 Extended offline Completed without error 00% 249 -
#16 Short offline Completed without error 00% 248 -
#17 Short offline Completed without error 00% 235 -
#18 Short offline Completed without error 00% 206 -
#19 Short offline Completed without error 00% 184 -
#20 Short offline Completed without error 00% 173 -
#21 Short offline Completed without error 00% 164 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
---------------------------------------------------------------------------------------------------
root@freenas[~]# smartctl -a /dev/da2
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron BX/MX1/2/3/500, M5/600, 1100 SSDs
Device Model: CT1000MX500SSD1
Serial Number: 2011E2940534
LU WWN Device Id: 5 00a075 1e2940534
Firmware Version: M3CR032
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Nov 24 09:59:27 2020 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 497
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 32
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 099 099 000 Old_age Always - 18
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 30
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 23
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 083 052 000 Old_age Always - 17 (Min/Max 0/48)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
202 Percent_Lifetime_Remain 0x0030 099 099 001 Old_age Offline - 1
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 10837580595
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 88806516
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 117814821

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 2

ATA Error Count: 0
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error -1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in an unknown state.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
00 ec 00 00 00 00 00

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
c8 00 00 00 00 00 00 00 00:00:00.000 READ DMA

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 497 -
# 2 Short offline Completed without error 00% 496 -
# 3 Extended offline Completed without error 00% 495 -
# 4 Short offline Completed without error 00% 487 -
# 5 Short offline Completed without error 00% 475 -
# 6 Short offline Completed without error 00% 461 -
# 7 Extended offline Completed without error 00% 460 -
# 8 Short offline Completed without error 00% 455 -
# 9 Short offline Completed without error 00% 442 -
#10 Short offline Completed without error 00% 384 -
#11 Short offline Completed without error 00% 341 -
#12 Short offline Completed without error 00% 336 -
#13 Short offline Completed without error 00% 322 -
#14 Short offline Completed without error 00% 279 -
#15 Short offline Completed without error 00% 264 -
#16 Extended offline Completed without error 00% 254 -
#17 Short offline Completed without error 00% 253 -
#18 Short offline Completed without error 00% 241 -
#19 Short offline Completed without error 00% 212 -
#20 Short offline Completed without error 00% 190 -
#21 Short offline Completed without error 00% 179 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
Search is your friend :
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
Damn - I found a number of threads - but not that one. Thank you
I have added -C 0 to each SSD by editing each disk and adding this to the smart options

Thank you - lets hope that works
 
Top