Hi all,
So, my TrueNAS build emailed me that it's getting worried as of a few minutes ago. I received 4 alert mails within 10 minutes from the system, so I was getting very curious...
ZFS tells me one of my drives is misbehaving:
Notice the 12 READ errors. That number has been slowly increasing every few minutes (it's at14 17 by the time I posted this).
Smartctl tells me a lot of new ATA errors occured rappidly, yet tells me the drive PASSED the test:
What could cause this, and how scared should I be? Should I maybe replace something like the SATA cable, or should I immediately order a new disk? It would be quite a shame, since I only got the disk a few months...
So, my TrueNAS build emailed me that it's getting worried as of a few minutes ago. I received 4 alert mails within 10 minutes from the system, so I was getting very curious...
ZFS tells me one of my drives is misbehaving:
Code:
# zpool status -x pool: fellowship state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 0B in 1 days 06:42:51 with 0 errors on Tue Feb 8 06:42:56 2022 config: NAME STATE READ WRITE CKSUM fellowship ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/16ec9779-3bd5-11ec-8148-2cf05dac8976 ONLINE 0 0 0 gptid/db21a5a8-3804-11ec-8a20-2cf05dac8976 ONLINE 0 0 0 gptid/dd89848a-4657-11ec-a00e-2cf05dac8976 ONLINE 0 0 0 gptid/c2869176-3806-11ec-8a20-2cf05dac8976 ONLINE 12 0 0 errors: No known data errors
Notice the 12 READ errors. That number has been slowly increasing every few minutes (it's at
Smartctl tells me a lot of new ATA errors occured rappidly, yet tells me the drive PASSED the test:
Code:
# smartctl -a /dev/ada0 smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Exos X16 Device Model: ST12000NM001G-2MV103 Serial Number: ZTN0AL3N LU WWN Device Id: 5 000c50 0db9ef421 Firmware Version: SN03 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Feb 10 20:52:46 2022 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1106) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 058 057 044 Pre-fail Always - 57732216 3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 133 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 176 7 Seek_Error_Rate 0x000f 085 060 045 Pre-fail Always - 294276055 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 2507 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 136 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 045 045 000 Old_age Always - 55 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 067 051 040 Old_age Always - 33 (Min/Max 30/37) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 114 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 262 194 Temperature_Celsius 0x0022 033 048 000 Old_age Always - 33 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 264 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 264 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 2479h+21m+04.717s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 39204426108 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 131763847442 SMART Error Log Version: 1 ATA Error Count: 55 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 55 occurred at disk power-on lifetime: 2507 hours (104 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 10 ff ff ff 4f 00 41d+08:43:08.201 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:08.201 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:08.200 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 41d+08:43:08.200 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 41d+08:43:08.164 READ LOG EXT Error 54 occurred at disk power-on lifetime: 2507 hours (104 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 10 ff ff ff 4f 00 41d+08:43:05.325 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:05.325 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:05.325 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 41d+08:43:05.325 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 41d+08:43:05.231 READ LOG EXT Error 53 occurred at disk power-on lifetime: 2507 hours (104 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 10 ff ff ff 4f 00 41d+08:43:02.386 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:02.385 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:43:02.385 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 41d+08:43:02.385 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 41d+08:43:02.349 READ LOG EXT Error 52 occurred at disk power-on lifetime: 2507 hours (104 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 08 ff ff ff 4f 00 41d+08:42:59.499 WRITE FPDMA QUEUED 60 00 10 ff ff ff 4f 00 41d+08:42:59.498 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:42:59.498 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:42:59.498 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 41d+08:42:59.498 READ FPDMA QUEUED Error 51 occurred at disk power-on lifetime: 2507 hours (104 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 08 ff ff ff 4f 00 41d+08:42:59.326 WRITE FPDMA QUEUED 60 00 10 ff ff ff 4f 00 41d+08:42:59.262 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:42:57.327 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 41d+08:42:56.363 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 41d+08:42:56.294 READ FPDMA QUEUED SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 2507 - # 2 Short offline Completed without error 00% 2507 - # 3 Short offline Completed without error 00% 325 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
What could cause this, and how scared should I be? Should I maybe replace something like the SATA cable, or should I immediately order a new disk? It would be quite a shame, since I only got the disk a few months...