CRC errors - verify steps after replacing cable

A

asmodeus

Explorer
Joined
Jul 26, 2016
Messages
70
Hello,

I have a disk in my pool that FAULTED due to 'too many errors'. It is also reporting a UDMA_CRC_Error_Count of 1, Command_Timeout of 1 so I am going to power down my FreeNAS, replace the cable and see if that helps. What next steps do I perform after powering back up? Is 'replace disk' required now? Or can I somehow clear the faulted flag?

zpool status:
Code:
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub canceled on Tue Jun 11 09:20:52 2019
config:

        NAME                                                STATE     READ WRITE CKSUM
        tank                                                DEGRADED     0     0     0
          raidz2-0                                          DEGRADED     0     0     0
            gptid/c69917ab-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/7538cd43-8a92-11e9-8f89-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/c8255ea3-5e48-11e6-8818-0cc47a82d3e8.eli  FAULTED      6   108     0  too many errors
            gptid/c8f7431d-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/c9b20cb3-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/ca8fd5bf-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/cb546522-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0
            gptid/cc2d9b13-5e48-11e6-8818-0cc47a82d3e8.eli  ONLINE       0     0     0

errors: No known data errors


command timeout:
Code:
        (da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00 length 20480 SMID 668 terminated ioc 804b scsi 0 state c xfer 0
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00
(da2:mpr0:0:2:0): CAM status: SCSI Status Error
(da2:mpr0:0:2:0): SCSI status: Check Condition
(da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da2:mpr0:0:2:0): Retrying command (per sense data)
        (da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 468 terminated ioc 804b scsi 0 state c xfer 0
(da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command
(da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mpr0:0:2:0): CAM status: SCSI Status Error
(da2:mpr0:0:2:0): SCSI status: Check Condition
(da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da2:mpr0:0:2:0): Error 6, Retries exhausted
(da2:mpr0:0:2:0): Invalidating pack


smartctl output:
Code:
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate NAS HDD
Device Model:     ST4000VN000-2AH166
Serial Number:    WDH00MNH
LU WWN Device Id: 5 000c50 09bbc66d1
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 24 12:26:06 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 639) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       155120576
  3 Spin_Up_Time            0x0003   094   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1525862684
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24650 (114 44 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       34
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   059   040    Old_age   Always       -       36 (Min/Max 31/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       91
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       171
194 Temperature_Celsius     0x0022   036   041   000    Old_age   Always       -       36 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       24650 (67 113 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       79417164037
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       338669484870

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     24616         -
# 2  Extended offline    Completed without error       00%     24601         -
# 3  Short offline       Completed without error       00%     24568         -
# 4  Short offline       Completed without error       00%     24520         -
# 5  Short offline       Completed without error       00%     24474         -
# 6  Short offline       Completed without error       00%     24426         -
# 7  Short offline       Completed without error       00%     24378         -
# 8  Short offline       Completed without error       00%     24330         -
# 9  Short offline       Completed without error       00%     24282         -
#10  Extended offline    Completed without error       00%     24267         -
#11  Short offline       Completed without error       00%     24234         -
#12  Short offline       Completed without error       00%     24187         -
#13  Short offline       Completed without error       00%     24139         -
#14  Short offline       Completed without error       00%     24091         -
#15  Short offline       Completed without error       00%     24066         -
#16  Short offline       Completed without error       00%     24018         -
#17  Short offline       Completed without error       00%     23971         -
#18  Short offline       Completed without error       00%     23922         -
#19  Short offline       Completed without error       00%     23874         -
#20  Extended offline    Completed without error       00%     23859         -
#21  Short offline       Completed without error       00%     23826         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@nas.nyx.io /var/log
❯ smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate NAS HDD
Device Model:     ST4000VN000-2AH166
Serial Number:    WDH00MNH
LU WWN Device Id: 5 000c50 09bbc66d1
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 24 12:27:24 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 639) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       155120576
  3 Spin_Up_Time            0x0003   094   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1525862699
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24650 (161 213 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       34
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   059   040    Old_age   Always       -       36 (Min/Max 31/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       91
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       171
194 Temperature_Celsius     0x0022   036   041   000    Old_age   Always       -       36 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       24650 (115 26 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       79417164037
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       338669484870

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     24616         -
# 2  Extended offline    Completed without error       00%     24601         -
# 3  Short offline       Completed without error       00%     24568         -
# 4  Short offline       Completed without error       00%     24520         -
# 5  Short offline       Completed without error       00%     24474         -
# 6  Short offline       Completed without error       00%     24426         -
# 7  Short offline       Completed without error       00%     24378         -
# 8  Short offline       Completed without error       00%     24330         -
# 9  Short offline       Completed without error       00%     24282         -
#10  Extended offline    Completed without error       00%     24267         -
#11  Short offline       Completed without error       00%     24234         -
#12  Short offline       Completed without error       00%     24187         -
#13  Short offline       Completed without error       00%     24139         -
#14  Short offline       Completed without error       00%     24091         -
#15  Short offline       Completed without error       00%     24066         -
#16  Short offline       Completed without error       00%     24018         -
#17  Short offline       Completed without error       00%     23971         -
#18  Short offline       Completed without error       00%     23922         -
#19  Short offline       Completed without error       00%     23874         -
#20  Extended offline    Completed without error       00%     23859         -
#21  Short offline       Completed without error       00%     23826         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
A

asmodeus

Explorer
Joined
Jul 26, 2016
Messages
70
In case anyone else is wondering about it, I shut down my system, replaced the cable and booted up again. FreeNAS automatically went on with resilvering the disk after I unlocked the encrypted pool. Unfortunately I already have another CRC error, so that probably wasn't it (or the replacement cable is faulty too).
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "CRC errors - verify steps after replacing cable"

Similar threads

P
pool status unhealthy
Replies
16
Views
42K
sretalla
sretalla
madtulip
  • Locked
SMART error (FailedReadSmartSelfTestLog) detected on host
Replies
10
Views
6K
joeschmuck
joeschmuck
mbonato
zpool CKSUM errors while resilvering after replacing a drive
Replies
5
Views
4K
sretalla
sretalla
NugentS
Currently unreadable (pending) sectors
Replies
2
Views
4K
NugentS
NugentS
IceBoosteR
  • Locked
My first SMART error: What to do
Replies
12
Views
4K
IceBoosteR
IceBoosteR
Top