My main pool is currently in a degraded state due to an "unrecoverable error" in one of the drives.
Data1 is 4 disks of 4TB drives in Raidz1. WD Red Pro's. I am running AMD-8350x with 16 GB non-ECC ram. Running TrueNAS-SCALE-22.02.2.1.
`zpool status -v` shows 2 checksum errors in the drive
Truenas tells me that the device is /dev/sdc so `smartctl -a /dev/sdc` shows:
Reading the suggested openzfs docs post (https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P/) it seems like this is a minor issue, but I just wanted to check that I'm ok to `zpool clear` the drive and not replace the drive? I should obviously check the cables and whatnot as well.
I have already:
- Scrubbed the pool
- Performed a long SMART test on the drive
- Restarted the machine
The only weird thing is that before I restarted the number of checksum errors was different. Before it was reading I/O errors on the device but now it is reading checksum errors on the device. This error did occur after I had left the machine off for 3 days and whilst I was writing data to an SMB share.
Thanks for any help.
CRITICAL
Pool Data1 state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Data1 is 4 disks of 4TB drives in Raidz1. WD Red Pro's. I am running AMD-8350x with 16 GB non-ECC ram. Running TrueNAS-SCALE-22.02.2.1.
`zpool status -v` shows 2 checksum errors in the drive
Code:
root@truenas[~]# zpool status -v pool: Data1 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: resilvered 25.4G in 00:07:17 with 0 errors on Mon Aug 8 21:16:07 2022 config: NAME STATE READ WRITE CKSUM Data1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 f514485f-c6e0-4317-984c-25a1ecaf4645 ONLINE 0 0 0 21e09f5f-a64f-479c-8fe5-19c5be56c8b3 ONLINE 0 0 0 0a5a59a2-a8a0-4339-a00b-d53644064a13 ONLINE 0 0 0 8849c89a-a52f-4b2e-852b-37e5139a1744 ONLINE 0 0 2 errors: No known data errors
Truenas tells me that the device is /dev/sdc so `smartctl -a /dev/sdc` shows:
Code:
root@truenas[~]# smartctl -a /dev/sdc smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.120+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD40EFRX-68N32N0 Serial Number: WD-WCC7K6RJ1ANH LU WWN Device Id: 5 0014ee 2bbfc825a Firmware Version: 82.00A82 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Aug 9 08:08:58 2022 EEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (43080) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 457) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 180 161 021 Pre-fail Always - 6000 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 766 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 16122 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 713 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 766 194 Temperature_Celsius 0x0022 112 107 000 Old_age Always - 38 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 16121 - # 2 Extended offline Completed without error 00% 16097 - # 3 Short offline Completed without error 00% 16081 - # 4 Short offline Completed without error 00% 16057 - # 5 Short offline Completed without error 00% 16033 - # 6 Short offline Completed without error 00% 16009 - # 7 Short offline Completed without error 00% 15985 - # 8 Short offline Completed without error 00% 15962 - # 9 Short offline Completed without error 00% 15938 - #10 Short offline Completed without error 00% 15555 - #11 Extended offline Completed without error 00% 15465 - #12 Short offline Completed without error 00% 15347 - #13 Short offline Completed without error 00% 14627 - #14 Short offline Completed without error 00% 13906 - #15 Short offline Completed without error 00% 13162 - #16 Short offline Completed without error 00% 12443 - #17 Short offline Completed without error 00% 11701 - #18 Short offline Completed without error 00% 10981 - #19 Short offline Completed without error 00% 10238 - #20 Short offline Completed without error 00% 9567 - #21 Short offline Completed without error 00% 8824 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Reading the suggested openzfs docs post (https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P/) it seems like this is a minor issue, but I just wanted to check that I'm ok to `zpool clear` the drive and not replace the drive? I should obviously check the cables and whatnot as well.
I have already:
- Scrubbed the pool
- Performed a long SMART test on the drive
- Restarted the machine
The only weird thing is that before I restarted the number of checksum errors was different. Before it was reading I/O errors on the device but now it is reading checksum errors on the device. This error did occur after I had left the machine off for 3 days and whilst I was writing data to an SMB share.
Thanks for any help.