eirik.lars1
Explorer
- Joined
- Jul 16, 2016
- Messages
- 55
Hi,
Five year old system (disks replaced three years ago) Specs:
MB: Supermicro X11SSM-F
CPU: Intel Xeon E3-1220 V5 Skylake
RAM: 32 Gb ECC: 2x Samsung DDR4-2133 CL15 ECC SC - 16GB
Boot: Supermicro SATA DOM
Disks: 6x WDC WD60EFRX-68L0BN1 i raidz2
OS: TrueNAS-12.0-RELEASE (f862218137)
Received this error today after scrub:
Smartctl shows:
Could the ada5 disk be failing? What should be my next move? Do a smartctl long test on all the disks?
Five year old system (disks replaced three years ago) Specs:
MB: Supermicro X11SSM-F
CPU: Intel Xeon E3-1220 V5 Skylake
RAM: 32 Gb ECC: 2x Samsung DDR4-2133 CL15 ECC SC - 16GB
Boot: Supermicro SATA DOM
Disks: 6x WDC WD60EFRX-68L0BN1 i raidz2
OS: TrueNAS-12.0-RELEASE (f862218137)
Received this error today after scrub:
Code:
root@freenas:~ # zpool status
pool: freenas-boot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:01:23 with 0 errors on Fri Jun 11 03:46:23 2021
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 8K in 10:37:45 with 0 errors on Tue Jun 15 12:37:48 2021
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/788fc1ad-5f48-11e8-bd8c-0cc47ab4d97a ONLINE 0 0 0
gptid/13a2c7ac-5f91-11e8-be7a-0cc47ab4d97a ONLINE 0 0 0
gptid/28cd3c61-5fd8-11e8-b6db-0cc47ab4d97a ONLINE 0 0 0
gptid/e0c32132-601c-11e8-8ec6-0cc47ab4d97a ONLINE 0 0 0
gptid/9a4156da-605d-11e8-b0f3-0cc47ab4d97a ONLINE 3 0 0
gptid/60556717-60a4-11e8-bb0f-0cc47ab4d97a ONLINE 0 0 0
errors: No known data errors
root@freenas:~ #Smartctl shows:
Code:
root@freenas:~ # smartctl -x /dev/ada5
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD60EFRX-68L0BN1
Serial Number: WD-WX31D87DYY38
LU WWN Device Id: 5 0014ee 20f4b16d2
Firmware Version: 82.00A82
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5700 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jun 15 14:29:34 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 6344) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 717) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 197 197 021 - 9133
4 Start_Stop_Count -O--CK 100 100 000 - 32
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 100 253 000 - 0
9 Power_On_Hours -O--CK 064 064 000 - 26369
10 Spin_Retry_Count -O--CK 100 253 000 - 0
11 Calibration_Retry_Count -O--CK 100 253 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 31
192 Power-Off_Retract_Count -O--CK 200 200 000 - 6
193 Load_Cycle_Count -O--CK 200 200 000 - 565
194 Temperature_Celsius -O---K 117 106 000 - 35
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 54 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 5
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 5 [4] occurred at disk power-on lifetime: 26360 hours (1098 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 dd 8c 4f 70 40 00 Error: UNC at LBA = 0x1dd8c4f70 = 8011927408
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 40 00 a0 00 02 8e d0 96 20 40 08 26d+00:18:12.496 READ FPDMA QUEUED
60 00 08 00 98 00 02 4c 9d c3 c8 40 08 26d+00:18:12.496 READ FPDMA QUEUED
60 00 50 00 90 00 01 dd 8c 4f 28 40 08 26d+00:18:12.496 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 26d+00:18:12.495 READ LOG EXT
60 00 40 00 80 00 02 8e d0 96 20 40 08 26d+00:18:08.000 READ FPDMA QUEUED
Error 4 [3] occurred at disk power-on lifetime: 26360 hours (1098 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 dd 8c 4f 70 40 00 Error: UNC at LBA = 0x1dd8c4f70 = 8011927408
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 40 00 80 00 02 8e d0 96 20 40 08 26d+00:18:08.000 READ FPDMA QUEUED
60 00 08 00 78 00 02 4c 9d c3 c8 40 08 26d+00:18:08.000 READ FPDMA QUEUED
60 00 50 00 70 00 01 dd 8c 4f 28 40 08 26d+00:18:08.000 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 26d+00:18:07.999 READ LOG EXT
60 00 40 00 60 00 02 8e d0 96 20 40 08 26d+00:18:03.505 READ FPDMA QUEUED
Error 3 [2] occurred at disk power-on lifetime: 26360 hours (1098 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 dd 8c 4f 70 40 00 Error: UNC at LBA = 0x1dd8c4f70 = 8011927408
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 40 00 60 00 02 8e d0 96 20 40 08 26d+00:18:03.505 READ FPDMA QUEUED
60 00 08 00 58 00 02 4c 9d c3 c8 40 08 26d+00:18:03.505 READ FPDMA QUEUED
60 00 50 00 50 00 01 dd 8c 4f 28 40 08 26d+00:18:03.505 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 26d+00:18:03.503 READ LOG EXT
61 00 10 00 40 00 01 f3 8a ae 48 40 08 26d+00:17:59.020 WRITE FPDMA QUEUED
Error 2 [1] occurred at disk power-on lifetime: 26360 hours (1098 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 dd 8c 4f 70 40 00 Error: WP at LBA = 0x1dd8c4f70 = 8011927408
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 10 00 40 00 01 f3 8a ae 48 40 08 26d+00:17:59.020 WRITE FPDMA QUEUED
61 00 40 00 38 00 01 f3 8a 1b 18 40 08 26d+00:17:59.020 WRITE FPDMA QUEUED
61 00 40 00 30 00 01 f3 8a 1a c0 40 08 26d+00:17:59.020 WRITE FPDMA QUEUED
60 00 40 00 28 00 02 8e d0 96 20 40 08 26d+00:17:59.020 READ FPDMA QUEUED
60 00 08 00 20 00 02 4c 9d c3 c8 40 08 26d+00:17:59.019 READ FPDMA QUEUED
Error 1 [0] occurred at disk power-on lifetime: 26360 hours (1098 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 dd 8c 4f 70 40 00 Error: UNC at LBA = 0x1dd8c4f70 = 8011927408
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 08 00 d8 00 01 dd 8c 60 f0 40 08 26d+00:17:54.568 READ FPDMA QUEUED
60 00 50 00 d0 00 01 dd 8c 4f 28 40 08 26d+00:17:54.562 READ FPDMA QUEUED
60 00 38 00 c8 00 01 dd 8c 4d 20 40 08 26d+00:17:54.558 READ FPDMA QUEUED
60 00 08 00 c0 00 01 dd 8b ff 20 40 08 26d+00:17:54.552 READ FPDMA QUEUED
60 00 08 00 b8 00 01 dd 8b fb 38 40 08 26d+00:17:54.546 READ FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 127 -
# 2 Extended offline Completed without error 00% 13 -
# 3 Extended offline Aborted by host 90% 0 -
# 4 Conveyance offline Completed without error 00% 0 -
# 5 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 35 Celsius
Power Cycle Min/Max Temperature: 20/38 Celsius
Lifetime Min/Max Temperature: 20/46 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (280)
Index Estimated Time Temperature Celsius
281 2021-06-15 06:32 37 ******************
... ..(136 skipped). .. ******************
418 2021-06-15 08:49 37 ******************
419 2021-06-15 08:50 36 *****************
... ..( 22 skipped). .. *****************
442 2021-06-15 09:13 36 *****************
443 2021-06-15 09:14 35 ****************
... ..( 80 skipped). .. ****************
46 2021-06-15 10:35 35 ****************
47 2021-06-15 10:36 38 *******************
... ..(106 skipped). .. *******************
154 2021-06-15 12:23 38 *******************
155 2021-06-15 12:24 37 ******************
... ..(124 skipped). .. ******************
280 2021-06-15 14:29 37 ******************
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 2) ==
0x01 0x008 4 31 --- Lifetime Power-On Resets
0x01 0x010 4 26369 --- Power-on Hours
0x01 0x018 6 71888865320 --- Logical Sectors Written
0x01 0x020 6 1818551784 --- Number of Write Commands
0x01 0x028 6 535154745801 --- Logical Sectors Read
0x01 0x030 6 2580298900 --- Number of Read Commands
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 26188 --- Spindle Motor Power-on Hours
0x03 0x010 4 26175 --- Head Flying Hours
0x03 0x018 4 572 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 476 --- Read Recovery Attempts
0x03 0x030 4 0 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 5 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 0 --- Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 35 --- Current Temperature
0x05 0x010 1 36 --- Average Short Term Temperature
0x05 0x018 1 32 --- Average Long Term Temperature
0x05 0x020 1 46 --- Highest Temperature
0x05 0x028 1 20 --- Lowest Temperature
0x05 0x030 1 43 --- Highest Average Short Term Temperature
0x05 0x038 1 21 --- Lowest Average Short Term Temperature
0x05 0x040 1 40 --- Highest Average Long Term Temperature
0x05 0x048 1 26 --- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 168 --- Number of Hardware Resets
0x06 0x010 4 62 --- Number of ASR Events
0x06 0x018 4 0 --- Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c)
Index LBA Hours
0 4303484304 -
1 4303484305 -
2 4303484306 -
3 4303484307 -
4 4303484308 -
5 4303484309 -
6 4303484310 -
7 4303484311 -
8 4303484312 -
9 4303484313 -
10 4303484314 -
11 4303484315 -
12 4303484316 -
13 4303484317 -
14 4303484318 -
15 4303484319 -
16 4321384408 -
17 4321384409 -
18 4321384410 -
19 4321384411 -
20 4321384412 -
21 4321384413 -
22 4321384414 -
23 4321384415 -
24 4321384424 -
25 4321384425 -
26 4321384426 -
27 4321384427 -
28 4321384428 -
29 4321384429 -
30 4321384430 -
... (937 entries not shown)
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 4 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 15166624 Vendor specific
Could the ada5 disk be failing? What should be my next move? Do a smartctl long test on all the disks?
