Kevin Horton
Guru
- Joined
- Dec 2, 2015
- Messages
- 730
I've got one drive that has thrown checksum errors twice in the last week, as reported by zpool status. The first time was a single checksum error. I did a SMART long test, and no issues were obvious to me. I did a zpool clear, and now several days later, during a routine scrub, I get another report of a degraded pool, with the same disk as the cause.
I'll run another long SMART test once the scrub has finished. I had started one, then saw that a scrub was in progress, so I aborted it, not wanting to slow down the scrub.
SMART status:
My records show that this same disk has fallen offline twice in the past, once during a reboot and once out of the blue.
Given that there are no UDMA_CRC_Error_Count reported in the SMART results, is there any way this could be a cabling issue?
I have multiple, up to date backups and a burned in disk ready to use as a replacement if needed.
HW config:
Motherboard: Supermicro X10SRH-cF
CPU: E5-1650v4
RAM: 96GB
Storage: 8x WD Red WD40EFRX 4TB in RAIDZ2 (main pool)
8x WD Red WD40EFRX 4TB in RAIDZ2 (backup pool, replicated from main pool)
Boot SSD: SanDisk Plus 120GB
Chassis: Norco RPC-4224
Code:
pool: main
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub in progress since Mon Jun 29 21:00:43 2020
14.9T scanned at 451M/s, 12.7T issued at 381M/s, 15.4T total
2.76M repaired, 82.26% done, 0 days 02:04:58 to go
config:
NAME STATE READ WRITE CKSUM
main DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/73e9c0c5-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/749fe03a-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/75508dd8-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/76057e53-d607-11e6-8ce7-0cc47ad9b1ca DEGRADED 0 0 707 too many errors
gptid/76b682c2-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/7760db32-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/781a0c8e-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
gptid/78ce7800-d607-11e6-8ce7-0cc47ad9b1ca ONLINE 0 0 0
errors: No known data errors
I'll run another long SMART test once the scrub has finished. I had started one, then saw that a scrub was in progress, so I aborted it, not wanting to slow down the scrub.
SMART status:
Code:
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E7JD09SA
LU WWN Device Id: 5 0014ee 2629e18a6
Firmware Version: 82.00A82
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jun 30 06:44:46 2020 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 25) The self-test routine was aborted by
the host.
Total time to complete Offline
data collection: (51840) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 518) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 188 175 021 - 7600
4 Start_Stop_Count -O--CK 100 100 000 - 102
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 100 253 000 - 0
9 Power_On_Hours -O--CK 060 060 000 - 29578
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 102
192 Power-Off_Retract_Count -O--CK 200 200 000 - 57
193 Load_Cycle_Count -O--CK 200 200 000 - 983
194 Temperature_Celsius -O---K 114 112 000 - 38
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 39 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Aborted by host 90% 29578 -
# 2 Short offline Completed without error 00% 29568 -
# 3 Short offline Completed without error 00% 29544 -
# 4 Extended offline Completed without error 00% 29522 -
# 5 Short offline Completed without error 00% 29495 -
# 6 Short offline Completed without error 00% 29472 -
# 7 Short offline Completed without error 00% 29447 -
# 8 Short offline Completed without error 00% 29421 -
# 9 Short offline Completed without error 00% 29397 -
#10 Extended offline Completed without error 00% 29386 -
#11 Short offline Completed without error 00% 29373 -
#12 Short offline Completed without error 00% 29350 -
#13 Short offline Completed without error 00% 29326 -
#14 Short offline Completed without error 00% 29301 -
#15 Short offline Completed without error 00% 29278 -
#16 Short offline Completed without error 00% 29254 -
#17 Short offline Completed without error 00% 29230 -
#18 Short offline Completed without error 00% 29207 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 38 Celsius
Power Cycle Min/Max Temperature: 29/40 Celsius
Lifetime Min/Max Temperature: 3/40 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (56)
Index Estimated Time Temperature Celsius
57 2020-06-29 22:47 38 *******************
... ..( 89 skipped). .. *******************
147 2020-06-30 00:17 38 *******************
148 2020-06-30 00:18 39 ********************
... ..( 5 skipped). .. ********************
154 2020-06-30 00:24 39 ********************
155 2020-06-30 00:25 38 *******************
... ..( 46 skipped). .. *******************
202 2020-06-30 01:12 38 *******************
203 2020-06-30 01:13 39 ********************
... ..( 6 skipped). .. ********************
210 2020-06-30 01:20 39 ********************
211 2020-06-30 01:21 38 *******************
... ..(158 skipped). .. *******************
370 2020-06-30 04:00 38 *******************
371 2020-06-30 04:01 39 ********************
... ..( 6 skipped). .. ********************
378 2020-06-30 04:08 39 ********************
379 2020-06-30 04:09 38 *******************
... ..( 78 skipped). .. *******************
458 2020-06-30 05:28 38 *******************
459 2020-06-30 05:29 39 ********************
460 2020-06-30 05:30 39 ********************
461 2020-06-30 05:31 38 *******************
... ..( 72 skipped). .. *******************
56 2020-06-30 06:44 38 *******************
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 1 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 1 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 507501 Vendor specific
My records show that this same disk has fallen offline twice in the past, once during a reboot and once out of the blue.
Given that there are no UDMA_CRC_Error_Count reported in the SMART results, is there any way this could be a cabling issue?
I have multiple, up to date backups and a burned in disk ready to use as a replacement if needed.
HW config:
Motherboard: Supermicro X10SRH-cF
CPU: E5-1650v4
RAM: 96GB
Storage: 8x WD Red WD40EFRX 4TB in RAIDZ2 (main pool)
8x WD Red WD40EFRX 4TB in RAIDZ2 (backup pool, replicated from main pool)
Boot SSD: SanDisk Plus 120GB
Chassis: Norco RPC-4224
