Kevin Horton
Guru
- Joined
- Dec 2, 2015
- Messages
- 730
I've got one drive that has thrown checksum errors twice in the last week, as reported by zpool status. The first time was a single checksum error. I did a SMART long test, and no issues were obvious to me. I did a zpool clear, and now several days later, during a routine scrub, I get another report of a degraded pool, with the same disk as the cause.
I'll run another long SMART test once the scrub has finished. I had started one, then saw that a scrub was in progress, so I aborted it, not wanting to slow down the scrub.
SMART status:
My records show that this same disk has fallen offline twice in the past, once during a reboot and once out of the blue.
Given that there are no UDMA_CRC_Error_Count reported in the SMART results, is there any way this could be a cabling issue?
I have multiple, up to date backups and a burned in disk ready to use as a replacement if needed.
HW config:
Motherboard: Supermicro X10SRH-cF
CPU: E5-1650v4
RAM: 96GB
Storage: 8x WD Red WD40EFRX 4TB in RAIDZ2 (main pool)
8x WD Red WD40EFRX 4TB in RAIDZ2 (backup pool, replicated from main pool)
Boot SSD: SanDisk Plus 120GB
Chassis: Norco RPC-4224
Code:
pool: main state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: scrub in progress since Mon Jun 29 21:00:43 2020 14.9T scanned at 451M/s, 12.7T issued at 381M/s, 15.4T total 2.76M repaired, 82.26% done, 0 days 02:04:58 to go config: NAME STATE READ WRITE CKSUM main DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/73e9c0c5-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/749fe03a-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/75508dd8-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/76057e53-d607-11e6-8ce7-0cc47ad9b1ca DEGRADED 0 0 707 too many errors gptid/76b682c2-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/7760db32-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/781a0c8e-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 gptid/78ce7800-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0 errors: No known data errors
I'll run another long SMART test once the scrub has finished. I had started one, then saw that a scrub was in progress, so I aborted it, not wanting to slow down the scrub.
SMART status:
Code:
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p9 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD40EFRX-68WT0N0 Serial Number: WD-WCC4E7JD09SA LU WWN Device Id: 5 0014ee 2629e18a6 Firmware Version: 82.00A82 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Jun 30 06:44:46 2020 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Unavailable Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 25) The self-test routine was aborted by the host. Total time to complete Offline data collection: (51840) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 518) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0 3 Spin_Up_Time POS--K 188 175 021 - 7600 4 Start_Stop_Count -O--CK 100 100 000 - 102 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 7 Seek_Error_Rate -OSR-K 100 253 000 - 0 9 Power_On_Hours -O--CK 060 060 000 - 29578 10 Spin_Retry_Count -O--CK 100 100 000 - 0 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 102 192 Power-Off_Retract_Count -O--CK 200 200 000 - 57 193 Load_Cycle_Count -O--CK 200 200 000 - 983 194 Temperature_Celsius -O---K 114 112 000 - 38 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 197 Current_Pending_Sector -O--CK 200 200 000 - 0 198 Offline_Uncorrectable ----CK 100 253 000 - 0 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 5 Comprehensive SMART error log 0x03 GPL R/O 6 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log 0xa8-0xb6 GPL,SL VS 1 Device vendor specific log 0xb7 GPL,SL VS 39 Device vendor specific log 0xbd GPL,SL VS 1 Device vendor specific log 0xc0 GPL,SL VS 1 Device vendor specific log 0xc1 GPL VS 93 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (6 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 90% 29578 - # 2 Short offline Completed without error 00% 29568 - # 3 Short offline Completed without error 00% 29544 - # 4 Extended offline Completed without error 00% 29522 - # 5 Short offline Completed without error 00% 29495 - # 6 Short offline Completed without error 00% 29472 - # 7 Short offline Completed without error 00% 29447 - # 8 Short offline Completed without error 00% 29421 - # 9 Short offline Completed without error 00% 29397 - #10 Extended offline Completed without error 00% 29386 - #11 Short offline Completed without error 00% 29373 - #12 Short offline Completed without error 00% 29350 - #13 Short offline Completed without error 00% 29326 - #14 Short offline Completed without error 00% 29301 - #15 Short offline Completed without error 00% 29278 - #16 Short offline Completed without error 00% 29254 - #17 Short offline Completed without error 00% 29230 - #18 Short offline Completed without error 00% 29207 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 258 (0x0102) Device State: Active (0) Current Temperature: 38 Celsius Power Cycle Min/Max Temperature: 29/40 Celsius Lifetime Min/Max Temperature: 3/40 Celsius Under/Over Temperature Limit Count: 0/0 Vendor specific: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (56) Index Estimated Time Temperature Celsius 57 2020-06-29 22:47 38 ******************* ... ..( 89 skipped). .. ******************* 147 2020-06-30 00:17 38 ******************* 148 2020-06-30 00:18 39 ******************** ... ..( 5 skipped). .. ******************** 154 2020-06-30 00:24 39 ******************** 155 2020-06-30 00:25 38 ******************* ... ..( 46 skipped). .. ******************* 202 2020-06-30 01:12 38 ******************* 203 2020-06-30 01:13 39 ******************** ... ..( 6 skipped). .. ******************** 210 2020-06-30 01:20 39 ******************** 211 2020-06-30 01:21 38 ******************* ... ..(158 skipped). .. ******************* 370 2020-06-30 04:00 38 ******************* 371 2020-06-30 04:01 39 ******************** ... ..( 6 skipped). .. ******************** 378 2020-06-30 04:08 39 ******************** 379 2020-06-30 04:09 38 ******************* ... ..( 78 skipped). .. ******************* 458 2020-06-30 05:28 38 ******************* 459 2020-06-30 05:29 39 ******************** 460 2020-06-30 05:30 39 ******************** 461 2020-06-30 05:31 38 ******************* ... ..( 72 skipped). .. ******************* 56 2020-06-30 06:44 38 ******************* SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) Device Statistics (GP/SMART Log 0x04) not supported Pending Defects log (GP Log 0x0c) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 1 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 1 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC 0x8000 4 507501 Vendor specific
My records show that this same disk has fallen offline twice in the past, once during a reboot and once out of the blue.
Given that there are no UDMA_CRC_Error_Count reported in the SMART results, is there any way this could be a cabling issue?
I have multiple, up to date backups and a burned in disk ready to use as a replacement if needed.
HW config:
Motherboard: Supermicro X10SRH-cF
CPU: E5-1650v4
RAM: 96GB
Storage: 8x WD Red WD40EFRX 4TB in RAIDZ2 (main pool)
8x WD Red WD40EFRX 4TB in RAIDZ2 (backup pool, replicated from main pool)
Boot SSD: SanDisk Plus 120GB
Chassis: Norco RPC-4224