Been running freenas -> truenas for a long time, but still a bit of a novice (it's been very good to me and I haven't had to mess-around too much with it)...
Just received an alert of "Pool pool1 state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected." Sure enough, the pool (creatively named "pool1" ;) ) shows unhealthy...
Just about 1hr later I get the next alert: "Device: /dev/ada2, ATA error count increased from 0 to 5."
So that's the details I have...can anyone help me understand whether I should replace ata2 ASAP, or try some other troubleshooting?
Thanks very much,
AJ
Just received an alert of "Pool pool1 state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected." Sure enough, the pool (creatively named "pool1" ;) ) shows unhealthy...
Code:
zpool status -v pool1 pool: pool1 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 0B in 08:30:27 with 0 errors on Thu Sep 1 08:30:28 2022 config: NAME STATE READ WRITE CKSUM pool1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/a577c0fe-ab6f-11e6-a104-10bf48b6ddea ONLINE 0 0 0 gptid/a67b47c3-ab6f-11e6-a104-10bf48b6ddea ONLINE 0 0 0 gptid/a77d0f33-ab6f-11e6-a104-10bf48b6ddea ONLINE 1 0 0 errors: No known data errors
Just about 1hr later I get the next alert: "Device: /dev/ada2, ATA error count increased from 0 to 5."
Code:
smartctl -a /dev/ada2 smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68EUZN0 Serial Number: WD-WCC4N4AJUXKZ LU WWN Device Id: 5 0014ee 2630f6d04 Firmware Version: 82.00A82 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Sep 1 14:41:37 2022 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (39540) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 397) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 6 3 Spin_Up_Time 0x0027 183 180 021 Pre-fail Always - 5850 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 61 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 038 038 000 Old_age Always - 45932 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 61 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 43 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 589 194 Temperature_Celsius 0x0022 118 112 000 Old_age Always - 32 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 SMART Error Log Version: 1 ATA Error Count: 5 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 5 occurred at disk power-on lifetime: 45931 hours (1913 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 80 58 f5 18 e6 Error: UNC 128 sectors at LBA = 0x0618f558 = 102298968 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 80 50 f5 18 e6 00 4d+05:34:28.543 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:25.147 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:21.750 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:18.343 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:14.973 READ DMA Error 4 occurred at disk power-on lifetime: 45931 hours (1913 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 80 58 f5 18 e6 Error: UNC 128 sectors at LBA = 0x0618f558 = 102298968 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 80 50 f5 18 e6 00 4d+05:34:25.147 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:21.750 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:18.343 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:14.973 READ DMA Error 3 occurred at disk power-on lifetime: 45931 hours (1913 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 80 58 f5 18 e6 Error: UNC 128 sectors at LBA = 0x0618f558 = 102298968 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 80 50 f5 18 e6 00 4d+05:34:21.750 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:18.343 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:14.973 READ DMA c8 00 80 d0 f2 18 e6 00 4d+05:34:14.961 READ DMA Error 2 occurred at disk power-on lifetime: 45931 hours (1913 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 80 58 f5 18 e6 Error: UNC 128 sectors at LBA = 0x0618f558 = 102298968 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 80 50 f5 18 e6 00 4d+05:34:18.343 READ DMA c8 00 80 50 f5 18 e6 00 4d+05:34:14.973 READ DMA c8 00 80 d0 f2 18 e6 00 4d+05:34:14.961 READ DMA c8 00 80 50 f2 18 e6 00 4d+05:34:14.961 READ DMA Error 1 occurred at disk power-on lifetime: 45931 hours (1913 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 80 58 f5 18 e6 Error: UNC 128 sectors at LBA = 0x0618f558 = 102298968 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 80 50 f5 18 e6 00 4d+05:34:14.973 READ DMA c8 00 80 d0 f2 18 e6 00 4d+05:34:14.961 READ DMA c8 00 80 50 f2 18 e6 00 4d+05:34:14.961 READ DMA c8 00 80 d0 f1 18 e6 00 4d+05:34:14.961 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 45823 - # 2 Short offline Completed without error 00% 45655 - # 3 Short offline Completed without error 00% 45487 - # 4 Short offline Completed without error 00% 45320 - # 5 Short offline Completed without error 00% 45152 - # 6 Short offline Completed without error 00% 44984 - # 7 Short offline Completed without error 00% 44816 - # 8 Short offline Completed without error 00% 44648 - # 9 Short offline Completed without error 00% 44480 - #10 Short offline Completed without error 00% 44313 - #11 Short offline Completed without error 00% 44145 - #12 Short offline Completed without error 00% 43977 - #13 Short offline Completed without error 00% 43809 - #14 Short offline Completed without error 00% 43641 - #15 Short offline Completed without error 00% 43474 - #16 Short offline Completed without error 00% 43314 - #17 Short offline Completed without error 00% 43138 - #18 Short offline Completed without error 00% 42974 - #19 Short offline Completed without error 00% 42802 - #20 Short offline Completed without error 00% 42634 - #21 Short offline Completed without error 00% 42467 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
So that's the details I have...can anyone help me understand whether I should replace ata2 ASAP, or try some other troubleshooting?
Thanks very much,
AJ