RegularJoe
Patron
- Joined
- Aug 19, 2013
- Messages
- 330
I am getting indications the drive is junk but no in your face error that says "I am junk - replace me"
The drive finishes a short and long smart test fine and are considered PASSED. FWIW I have the volume 83% full with random files from DD and no compression. I am doing and I have finished in the past scrubs with no errors, I am concerned about these drive(s) and do not understand why the FreeNAS creates an alert when smartctl says they pass. If I have to mark them bad that is fine. The drives are attached to a HP d2600 shelf via a LSI SAS2116e HBA and I have other drives on this shelf/system with no issues and other drives in the same ZPOOL without issues. This is a lab machine with lab drives doing nothing except a scrub at the moment.
what I see in the legacy graphs is the disk in question has unpredictable latency and now the counters for "195 Hardware_ECC_Recovered" are skyrocketing, form 3988 to 28,029,079
Good dive next to it in the ZPOOL
Suspect drive:
Alert System :
CRITICAL: March 23, 2019, 1:07 a.m. - Device: /dev/da13 [SAT], ATA error count increased from 1 to 39
freenas1# smartctl -a /dev/da13
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST3500641NS 39M4557 39M0181 IBM
Serial Number: 3PM1Z2H5
Firmware Version: 3.AES
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA/ATAPI-7 (minor revision not indicated)
Local Time is: Sat Mar 23 11:04:58 2019 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 261) minutes.
SCT capabilities: (0x000d) SCT Status supported.
SCT Error Recovery Control supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 113 084 006 Pre-fail Always - 52138149
3 Spin_Up_Time 0x0003 086 086 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 6
7 Seek_Error_Rate 0x000f 088 060 030 Pre-fail Always - 725414232
9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24485
10 Spin_Retry_Count 0x0013 100 100 000 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 47
187 Reported_Uncorrect 0x0032 093 093 000 Old_age Always - 7
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 067 042 045 Old_age Always In_the_past 33 (0 6 34 30 0)
194 Temperature_Celsius 0x0022 033 058 000 Old_age Always - 33 (0 20 0 0 0)
195 Hardware_ECC_Recovered 0x001a 059 048 000 Old_age Always - 46472436
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 57
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 39 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 39 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 24 00 40 00 00:34:34.566 READ DMA EXT
42 00 01 1b 75 28 40 00 00:34:34.564 READ VERIFY SECTOR(S) EXT
ea 00 01 1b 75 28 40 00 00:34:34.555 FLUSH CACHE EXT
35 00 01 1b 75 28 40 00 00:34:34.555 WRITE DMA EXT
25 00 00 00 20 00 40 00 00:34:39.853 READ DMA EXT
Error 38 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 76 21 00 40 Error: UNC at LBA = 0x00002176 = 8566
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 20 00 40 00 00:34:34.566 READ DMA EXT
60 00 00 00 2c 00 40 00 00:34:34.564 READ FPDMA QUEUED
42 00 01 1a 75 28 40 00 00:34:34.555 READ VERIFY SECTOR(S) EXT
ea 00 01 1a 75 28 40 00 00:34:34.555 FLUSH CACHE EXT
35 00 01 1a 75 28 40 00 00:34:34.534 WRITE DMA EXT
Error 37 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 76 21 00 40 Error: UNC at LBA = 0x00002176 = 8566
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 20 00 40 00 00:34:25.780 READ DMA EXT
42 00 01 18 75 28 40 00 00:34:25.759 READ VERIFY SECTOR(S) EXT
ea 00 01 18 75 28 40 00 00:34:25.717 FLUSH CACHE EXT
35 00 01 18 75 28 40 00 00:34:25.715 WRITE DMA EXT
25 00 01 74 65 28 40 00 00:34:25.705 READ DMA EXT
Error 36 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 24 00 40 00 00:34:25.780 READ DMA EXT
2f 00 01 10 00 00 00 00 00:34:25.759 READ LOG EXT
60 00 00 00 20 00 40 00 00:34:25.717 READ FPDMA QUEUED
25 00 01 7a 68 28 40 00 00:34:25.715 READ DMA EXT
60 00 00 00 00 00 00 00 00:34:25.705 READ FPDMA QUEUED
Error 35 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 4d b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
42 00 00 00 24 00 40 00 00:34:16.050 READ VERIFY SECTOR(S) EXT
ea 00 00 00 24 00 40 00 00:34:13.968 FLUSH CACHE EXT
35 00 00 00 24 00 40 00 00:34:13.968 WRITE DMA EXT
60 00 08 00 2c 00 40 00 00:34:13.882 READ FPDMA QUEUED
ea 00 00 00 2c 00 40 00 00:34:13.872 FLUSH CACHE EXT
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 24450 -
# 2 Extended offline Completed without error 00% 24343 -
# 3 Short offline Completed without error 00% 24340 -
# 4 Short offline Completed without error 00% 24280 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
freenas1#
The drive finishes a short and long smart test fine and are considered PASSED. FWIW I have the volume 83% full with random files from DD and no compression. I am doing and I have finished in the past scrubs with no errors, I am concerned about these drive(s) and do not understand why the FreeNAS creates an alert when smartctl says they pass. If I have to mark them bad that is fine. The drives are attached to a HP d2600 shelf via a LSI SAS2116e HBA and I have other drives on this shelf/system with no issues and other drives in the same ZPOOL without issues. This is a lab machine with lab drives doing nothing except a scrub at the moment.
what I see in the legacy graphs is the disk in question has unpredictable latency and now the counters for "195 Hardware_ECC_Recovered" are skyrocketing, form 3988 to 28,029,079
Good dive next to it in the ZPOOL
Suspect drive:
Alert System :
CRITICAL: March 23, 2019, 1:07 a.m. - Device: /dev/da13 [SAT], ATA error count increased from 1 to 39
freenas1# smartctl -a /dev/da13
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST3500641NS 39M4557 39M0181 IBM
Serial Number: 3PM1Z2H5
Firmware Version: 3.AES
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA/ATAPI-7 (minor revision not indicated)
Local Time is: Sat Mar 23 11:04:58 2019 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 261) minutes.
SCT capabilities: (0x000d) SCT Status supported.
SCT Error Recovery Control supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 113 084 006 Pre-fail Always - 52138149
3 Spin_Up_Time 0x0003 086 086 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 6
7 Seek_Error_Rate 0x000f 088 060 030 Pre-fail Always - 725414232
9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24485
10 Spin_Retry_Count 0x0013 100 100 000 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 47
187 Reported_Uncorrect 0x0032 093 093 000 Old_age Always - 7
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 067 042 045 Old_age Always In_the_past 33 (0 6 34 30 0)
194 Temperature_Celsius 0x0022 033 058 000 Old_age Always - 33 (0 20 0 0 0)
195 Hardware_ECC_Recovered 0x001a 059 048 000 Old_age Always - 46472436
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 57
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 39 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 39 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 24 00 40 00 00:34:34.566 READ DMA EXT
42 00 01 1b 75 28 40 00 00:34:34.564 READ VERIFY SECTOR(S) EXT
ea 00 01 1b 75 28 40 00 00:34:34.555 FLUSH CACHE EXT
35 00 01 1b 75 28 40 00 00:34:34.555 WRITE DMA EXT
25 00 00 00 20 00 40 00 00:34:39.853 READ DMA EXT
Error 38 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 76 21 00 40 Error: UNC at LBA = 0x00002176 = 8566
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 20 00 40 00 00:34:34.566 READ DMA EXT
60 00 00 00 2c 00 40 00 00:34:34.564 READ FPDMA QUEUED
42 00 01 1a 75 28 40 00 00:34:34.555 READ VERIFY SECTOR(S) EXT
ea 00 01 1a 75 28 40 00 00:34:34.555 FLUSH CACHE EXT
35 00 01 1a 75 28 40 00 00:34:34.534 WRITE DMA EXT
Error 37 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 76 21 00 40 Error: UNC at LBA = 0x00002176 = 8566
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 20 00 40 00 00:34:25.780 READ DMA EXT
42 00 01 18 75 28 40 00 00:34:25.759 READ VERIFY SECTOR(S) EXT
ea 00 01 18 75 28 40 00 00:34:25.717 FLUSH CACHE EXT
35 00 01 18 75 28 40 00 00:34:25.715 WRITE DMA EXT
25 00 01 74 65 28 40 00 00:34:25.705 READ DMA EXT
Error 36 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 00 24 00 40 00 00:34:25.780 READ DMA EXT
2f 00 01 10 00 00 00 00 00:34:25.759 READ LOG EXT
60 00 00 00 20 00 40 00 00:34:25.717 READ FPDMA QUEUED
25 00 01 7a 68 28 40 00 00:34:25.715 READ DMA EXT
60 00 00 00 00 00 00 00 00:34:25.705 READ FPDMA QUEUED
Error 35 occurred at disk power-on lifetime: 24276 hours (1011 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 4d b3 24 00 40 Error: UNC at LBA = 0x000024b3 = 9395
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
42 00 00 00 24 00 40 00 00:34:16.050 READ VERIFY SECTOR(S) EXT
ea 00 00 00 24 00 40 00 00:34:13.968 FLUSH CACHE EXT
35 00 00 00 24 00 40 00 00:34:13.968 WRITE DMA EXT
60 00 08 00 2c 00 40 00 00:34:13.882 READ FPDMA QUEUED
ea 00 00 00 2c 00 40 00 00:34:13.872 FLUSH CACHE EXT
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 24450 -
# 2 Extended offline Completed without error 00% 24343 -
# 3 Short offline Completed without error 00% 24340 -
# 4 Short offline Completed without error 00% 24280 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
freenas1#