Hello, I understand what I'm asking is similar to what others have reported, I need some advice to understand what is failing in my setup.
I have a raidz2 pool with 6 SATA disks, probably after a scrub the error "CAM status: ATA Status Error" started showing up in logs, making the pool degraded. I searched for advice online and then read the throubleshooting guide. I have not strictly followed the standard procedure written there, this is what I've done so far after seeing the error logged:
- run a SMART long test on the drive
- checked SMART attributes 5, 197, and 198, they are all 0, but it had Device Error Count: 46
- swapped the SATA data cable with another drive from the array (I've discovered that I have no spare sata3 cables left at home...)
- the drive has now changed name, from ada2 to ada4, meaning that now I'm testing both a different cable and sata port at the same time
- run a scrub
- the error showed up and the pool is degraded again
- SMART attributes 5, 197, and 198 are still 0, but now it shows Device Error Count: 59
SMART then is logging something, for example:
Others on this forum already reported a something similar to what I'm seeing, but the LBA in my case seems to be a real value, not something like 0000 or 0xffff (suggesting instead a drive communication failure). Details follow from here.
System hardware:
- Supermicro X10SLH-F
- Intel i3-4160, 16GB (DDR3)
- 6 WDC WD30EFRX-68EUZN0 (3TB SATA WD RED), connected to the onboard sata controller (intel pch?)
OS is TrueNAS-13.0-U6.1 (CORE)
This is a sample of /var/log/messages:
This is what ZFS shows:
This is the SMART information of the suspect drive:
This is another drive from the same array:
At this point, I think I have another test to do, swap the SATA cable again and see what happens, maybe when I swapped it first I managed to somehow connect the same cable to the same drive to a different port on the motherboard (I took the opportunity to improve a bit the cable management of the system while I had the system open). What do you think? Am I just wasting time or worse stressing the array for no reason? Why are the relevant SMART attributes stuck at 0?
I have a raidz2 pool with 6 SATA disks, probably after a scrub the error "CAM status: ATA Status Error" started showing up in logs, making the pool degraded. I searched for advice online and then read the throubleshooting guide. I have not strictly followed the standard procedure written there, this is what I've done so far after seeing the error logged:
- run a SMART long test on the drive
- checked SMART attributes 5, 197, and 198, they are all 0, but it had Device Error Count: 46
- swapped the SATA data cable with another drive from the array (I've discovered that I have no spare sata3 cables left at home...)
- the drive has now changed name, from ada2 to ada4, meaning that now I'm testing both a different cable and sata port at the same time
- run a scrub
- the error showed up and the pool is degraded again
- SMART attributes 5, 197, and 198 are still 0, but now it shows Device Error Count: 59
SMART then is logging something, for example:
Code:
Error 59 [10] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 01 55 c9 4b cf 40 00 Error: UNC at LBA = 0x155c94bcf = 5734222799
Others on this forum already reported a something similar to what I'm seeing, but the LBA in my case seems to be a real value, not something like 0000 or 0xffff (suggesting instead a drive communication failure). Details follow from here.
System hardware:
- Supermicro X10SLH-F
- Intel i3-4160, 16GB (DDR3)
- 6 WDC WD30EFRX-68EUZN0 (3TB SATA WD RED), connected to the onboard sata controller (intel pch?)
OS is TrueNAS-13.0-U6.1 (CORE)
This is a sample of /var/log/messages:
Code:
Jan 26 06:33:46 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 41 9e 40 55 01 00 08 00 00 Jan 26 06:33:46 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:33:46 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:33:46 freenas (ada4:ahcich4:0:0:0): RES: 41 40 c8 49 9e 40 55 01 00 00 00 Jan 26 06:33:46 freenas (ada4:ahcich4:0:0:0): Retrying command, 3 more tries remain Jan 26 06:33:53 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 41 9e 40 55 01 00 08 00 00 Jan 26 06:33:53 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:33:53 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:33:53 freenas (ada4:ahcich4:0:0:0): RES: 41 40 c8 49 9e 40 55 01 00 00 00 Jan 26 06:33:53 freenas (ada4:ahcich4:0:0:0): Retrying command, 2 more tries remain Jan 26 06:34:08 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 18 61 9e 40 55 01 00 07 00 00 Jan 26 06:34:08 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:08 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:08 freenas (ada4:ahcich4:0:0:0): RES: 41 40 88 61 9e 40 55 01 00 00 00 Jan 26 06:34:08 freenas (ada4:ahcich4:0:0:0): Retrying command, 3 more tries remain Jan 26 06:34:15 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 18 61 9e 40 55 01 00 07 00 00 Jan 26 06:34:15 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:15 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:15 freenas (ada4:ahcich4:0:0:0): RES: 41 40 8f 61 9e 40 55 01 00 00 00 Jan 26 06:34:15 freenas (ada4:ahcich4:0:0:0): Retrying command, 2 more tries remain Jan 26 06:34:22 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 18 61 9e 40 55 01 00 07 00 00 Jan 26 06:34:22 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:22 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:22 freenas (ada4:ahcich4:0:0:0): RES: 41 40 88 61 9e 40 55 01 00 00 00 Jan 26 06:34:22 freenas (ada4:ahcich4:0:0:0): Retrying command, 1 more tries remain Jan 26 06:34:29 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 18 61 9e 40 55 01 00 07 00 00 Jan 26 06:34:29 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:29 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:29 freenas (ada4:ahcich4:0:0:0): RES: 41 40 8f 61 9e 40 55 01 00 00 00 Jan 26 06:34:29 freenas (ada4:ahcich4:0:0:0): Retrying command, 0 more tries remain Jan 26 06:34:36 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 18 61 9e 40 55 01 00 07 00 00 Jan 26 06:34:36 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:36 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:36 freenas (ada4:ahcich4:0:0:0): RES: 41 40 88 61 9e 40 55 01 00 00 00 Jan 26 06:34:36 freenas (ada4:ahcich4:0:0:0): Error 5, Retries exhausted Jan 26 06:34:48 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c8 a9 9e 40 55 01 00 01 00 00 Jan 26 06:34:48 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 06:34:48 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 06:34:48 freenas (ada4:ahcich4:0:0:0): RES: 41 40 1f aa 9e 40 55 01 00 00 00 Jan 26 06:34:48 freenas (ada4:ahcich4:0:0:0): Retrying command, 3 more tries remain Jan 26 07:22:58 freenas 1 2024-01-26T07:22:58.092445+01:00 freenas.local collectd 3263 - - nut plugin: nut_connect: upscli_connect (localhost, 3493) failed: Connection failure: Connection refused Jan 26 09:15:51 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f8 49 c9 40 55 01 00 08 00 00 Jan 26 09:15:51 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 09:15:51 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 09:15:51 freenas (ada4:ahcich4:0:0:0): RES: 41 40 cf 4b c9 40 55 01 00 00 00 Jan 26 09:15:51 freenas (ada4:ahcich4:0:0:0): Retrying command, 3 more tries remain Jan 26 09:15:58 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f8 49 c9 40 55 01 00 08 00 00 Jan 26 09:15:58 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 09:15:58 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 09:15:58 freenas (ada4:ahcich4:0:0:0): RES: 41 40 d8 4b c9 40 55 01 00 00 00 Jan 26 09:15:58 freenas (ada4:ahcich4:0:0:0): Retrying command, 2 more tries remain Jan 26 09:16:05 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f8 49 c9 40 55 01 00 08 00 00 Jan 26 09:16:05 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 09:16:05 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 09:16:05 freenas (ada4:ahcich4:0:0:0): RES: 41 40 d8 4b c9 40 55 01 00 00 00 Jan 26 09:16:05 freenas (ada4:ahcich4:0:0:0): Retrying command, 1 more tries remain Jan 26 09:16:12 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f8 49 c9 40 55 01 00 08 00 00 Jan 26 09:16:12 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 09:16:12 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 09:16:12 freenas (ada4:ahcich4:0:0:0): RES: 41 40 cf 4b c9 40 55 01 00 00 00 Jan 26 09:16:12 freenas (ada4:ahcich4:0:0:0): Retrying command, 0 more tries remain Jan 26 09:16:19 freenas (ada4:ahcich4:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f8 49 c9 40 55 01 00 08 00 00 Jan 26 09:16:19 freenas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error Jan 26 09:16:19 freenas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jan 26 09:16:19 freenas (ada4:ahcich4:0:0:0): RES: 41 40 cf 4b c9 40 55 01 00 00 00 Jan 26 09:16:19 freenas (ada4:ahcich4:0:0:0): Error 5, Retries exhausted
This is what ZFS shows:
Code:
root@freenas[/]# zpool status heaven
pool: heaven
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 1.94M in 05:38:45 with 0 errors on Fri Jan 26 09:36:14 2024
config:
NAME STATE READ WRITE CKSUM
heaven DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/73149100-c06f-11e4-9fa4-0cc47a09c018 ONLINE 0 0 0
gptid/73ec5508-c06f-11e4-9fa4-0cc47a09c018 ONLINE 0 0 0
gptid/74b74322-c06f-11e4-9fa4-0cc47a09c018 ONLINE 0 0 0
gptid/75808f9b-c06f-11e4-9fa4-0cc47a09c018 ONLINE 0 0 0
gptid/7648fa64-c06f-11e4-9fa4-0cc47a09c018 ONLINE 0 0 0
gptid/771697bb-c06f-11e4-9fa4-0cc47a09c018 FAULTED 64 0 2 too many errors
errors: No known data errors
This is the SMART information of the suspect drive:
Code:
root@freenas[/]# smartctl -x /dev/ada4
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WCC4N7RE3DC6
LU WWN Device Id: 5 0014ee 2b54c166e
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Jan 28 01:16:35 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (39840) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 399) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 590
3 Spin_Up_Time POS--K 174 171 021 - 6291
4 Start_Stop_Count -O--CK 100 100 000 - 103
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 100 253 000 - 0
9 Power_On_Hours -O--CK 076 076 000 - 17724
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 103
192 Power-Off_Retract_Count -O--CK 200 200 000 - 21
193 Load_Cycle_Count -O--CK 200 200 000 - 203
194 Temperature_Celsius -O---K 122 100 000 - 28
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 172 165 000 - 11466
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 59 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 59 [10] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 c9 4b cf 40 00 Error: UNC at LBA = 0x155c94bcf = 5734222799
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 07 e0 00 20 00 01 55 c9 59 f8 40 08 08:33:44.688 READ FPDMA QUEUED
60 08 00 00 18 00 01 55 c9 51 f8 40 08 08:33:44.688 READ FPDMA QUEUED
60 08 00 00 10 00 01 55 c9 49 f8 40 08 08:33:44.688 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 08:33:44.656 READ LOG EXT
60 07 e0 00 00 00 01 55 c9 59 f8 40 08 08:33:37.662 READ FPDMA QUEUED
Error 58 [9] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 c9 4b cf 40 00 Error: UNC at LBA = 0x155c94bcf = 5734222799
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 07 e0 00 00 00 01 55 c9 59 f8 40 08 08:33:37.662 READ FPDMA QUEUED
60 08 00 00 f8 00 01 55 c9 51 f8 40 08 08:33:37.662 READ FPDMA QUEUED
60 08 00 00 f0 00 01 55 c9 49 f8 40 08 08:33:37.662 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 08:33:37.630 READ LOG EXT
60 07 e0 00 e0 00 01 55 c9 59 f8 40 08 08:33:30.635 READ FPDMA QUEUED
Error 57 [8] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 c9 4b d8 40 00 Error: UNC at LBA = 0x155c94bd8 = 5734222808
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 07 e0 00 e0 00 01 55 c9 59 f8 40 08 08:33:30.635 READ FPDMA QUEUED
60 08 00 00 d8 00 01 55 c9 51 f8 40 08 08:33:30.635 READ FPDMA QUEUED
60 08 00 00 d0 00 01 55 c9 49 f8 40 08 08:33:30.635 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 08:33:30.604 READ LOG EXT
60 07 e0 00 c0 00 01 55 c9 59 f8 40 08 08:33:23.609 READ FPDMA QUEUED
Error 56 [7] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 c9 4b d8 40 00 Error: UNC at LBA = 0x155c94bd8 = 5734222808
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 07 e0 00 c0 00 01 55 c9 59 f8 40 08 08:33:23.609 READ FPDMA QUEUED
60 08 00 00 b8 00 01 55 c9 51 f8 40 08 08:33:23.609 READ FPDMA QUEUED
60 08 00 00 b0 00 01 55 c9 49 f8 40 08 08:33:23.609 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 08:33:23.578 READ LOG EXT
60 07 e0 00 a0 00 01 55 c9 59 f8 40 08 08:33:16.583 READ FPDMA QUEUED
Error 55 [6] occurred at disk power-on lifetime: 17685 hours (736 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 c9 4b cf 40 00 Error: UNC at LBA = 0x155c94bcf = 5734222799
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 07 e0 00 a0 00 01 55 c9 59 f8 40 08 08:33:16.583 READ FPDMA QUEUED
60 08 00 00 98 00 01 55 c9 51 f8 40 08 08:33:16.577 READ FPDMA QUEUED
60 08 00 00 90 00 01 55 c9 49 f8 40 08 08:33:16.577 READ FPDMA QUEUED
60 00 d0 00 88 00 01 55 c9 48 38 40 08 08:33:16.564 READ FPDMA QUEUED
60 07 a0 00 80 00 01 55 c9 37 70 40 08 08:33:16.551 READ FPDMA QUEUED
Error 54 [5] occurred at disk power-on lifetime: 17682 hours (736 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 9e aa 1f 40 00 Error: UNC at LBA = 0x1559eaa1f = 5731428895
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 01 30 00 d0 00 01 55 9e ab 18 40 08 05:52:19.598 READ FPDMA QUEUED
60 01 00 00 c8 00 01 55 9e a9 c8 40 08 05:52:19.598 READ FPDMA QUEUED
60 00 40 00 c0 00 01 55 9e a9 40 40 08 05:52:18.152 READ FPDMA QUEUED
60 00 40 00 b8 00 01 55 9e a8 58 40 08 05:52:18.148 READ FPDMA QUEUED
60 00 c0 00 b0 00 01 55 9e a7 30 40 08 05:52:18.138 READ FPDMA QUEUED
Error 53 [4] occurred at disk power-on lifetime: 17682 hours (736 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 9e 61 88 40 00 Error: UNC at LBA = 0x1559e6188 = 5731410312
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 03 80 00 e8 00 01 55 9e 70 b8 40 08 05:52:10.807 READ FPDMA QUEUED
60 07 c8 00 e0 00 01 55 9e 68 f0 40 08 05:52:10.807 READ FPDMA QUEUED
60 07 d8 00 d8 00 01 55 9e 61 18 40 08 05:52:10.807 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 05:52:10.775 READ LOG EXT
60 03 80 00 c8 00 01 55 9e 70 b8 40 08 05:52:03.780 READ FPDMA QUEUED
Error 52 [3] occurred at disk power-on lifetime: 17682 hours (736 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 01 55 9e 61 8f 40 00 Error: UNC at LBA = 0x1559e618f = 5731410319
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 03 80 00 c8 00 01 55 9e 70 b8 40 08 05:52:03.780 READ FPDMA QUEUED
60 07 c8 00 c0 00 01 55 9e 68 f0 40 08 05:52:03.780 READ FPDMA QUEUED
60 07 d8 00 b8 00 01 55 9e 61 18 40 08 05:52:03.780 READ FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 40 08 05:52:03.748 READ LOG EXT
60 03 80 00 a8 00 01 55 9e 70 b8 40 08 05:51:56.754 READ FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 17723 -
# 2 Extended offline Completed without error 00% 17664 -
# 3 Short offline Completed without error 00% 17626 -
# 4 Extended offline Completed without error 00% 17502 -
# 5 Short offline Completed without error 00% 17480 -
# 6 Short offline Completed without error 00% 17325 -
# 7 Short offline Completed without error 00% 17157 -
# 8 Extended offline Completed without error 00% 17069 -
# 9 Short offline Completed without error 00% 16989 -
#10 Short offline Completed without error 00% 16804 -
#11 Extended offline Completed without error 00% 16788 -
#12 Short offline Completed without error 00% 16541 -
#13 Short offline Completed without error 00% 16374 -
#14 Short offline Completed without error 00% 16206 -
#15 Short offline Completed without error 00% 16038 -
#16 Extended offline Completed without error 00% 15949 -
#17 Short offline Completed without error 00% 15870 -
#18 Short offline Completed without error 00% 15702 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 28 Celsius
Power Cycle Min/Max Temperature: 19/31 Celsius
Lifetime Min/Max Temperature: 2/50 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (28)
Index Estimated Time Temperature Celsius
29 2024-01-27 17:19 28 *********
... ..(476 skipped). .. *********
28 2024-01-28 01:16 28 *********
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 174708 Vendor specific
This is another drive from the same array:
Code:
root@freenas[/]# smartctl -x /dev/ada0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WCC4N6LTT28P
LU WWN Device Id: 5 0014ee 2605b09c5
Firmware Version: 82.00A82
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Jan 28 01:17:44 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (40500) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 406) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 179 177 021 - 6008
4 Start_Stop_Count -O--CK 100 100 000 - 104
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 100 253 000 - 0
9 Power_On_Hours -O--CK 076 076 000 - 17725
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 104
192 Power-Off_Retract_Count -O--CK 200 200 000 - 21
193 Load_Cycle_Count -O--CK 200 200 000 - 259
194 Temperature_Celsius -O---K 122 101 000 - 28
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 17724 -
# 2 Short offline Completed without error 00% 17626 -
# 3 Extended offline Completed without error 00% 17499 -
# 4 Short offline Completed without error 00% 17480 -
# 5 Short offline Completed without error 00% 17325 -
# 6 Short offline Completed without error 00% 17158 -
# 7 Extended offline Completed without error 00% 17069 -
# 8 Short offline Completed without error 00% 16990 -
# 9 Short offline Completed without error 00% 16805 -
#10 Extended offline Completed without error 00% 16788 -
#11 Short offline Completed without error 00% 16542 -
#12 Short offline Completed without error 00% 16374 -
#13 Short offline Completed without error 00% 16206 -
#14 Short offline Completed without error 00% 16038 -
#15 Extended offline Completed without error 00% 15950 -
#16 Short offline Completed without error 00% 15870 -
#17 Short offline Completed without error 00% 15703 -
#18 Short offline Completed without error 00% 15535 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 28 Celsius
Power Cycle Min/Max Temperature: 18/29 Celsius
Lifetime Min/Max Temperature: 2/49 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (54)
Index Estimated Time Temperature Celsius
55 2024-01-27 17:20 28 *********
... ..(476 skipped). .. *********
54 2024-01-28 01:17 28 *********
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 174776 Vendor specific
At this point, I think I have another test to do, swap the SATA cable again and see what happens, maybe when I swapped it first I managed to somehow connect the same cable to the same drive to a different port on the motherboard (I took the opportunity to improve a bit the cable management of the system while I had the system open). What do you think? Am I just wasting time or worse stressing the array for no reason? Why are the relevant SMART attributes stuck at 0?
