Hello,
I have a disk in my pool that FAULTED due to 'too many errors'. It is also reporting a UDMA_CRC_Error_Count of 1, Command_Timeout of 1 so I am going to power down my FreeNAS, replace the cable and see if that helps. What next steps do I perform after powering back up? Is 'replace disk' required now? Or can I somehow clear the faulted flag?
zpool status:
command timeout:
smartctl output:
I have a disk in my pool that FAULTED due to 'too many errors'. It is also reporting a UDMA_CRC_Error_Count of 1, Command_Timeout of 1 so I am going to power down my FreeNAS, replace the cable and see if that helps. What next steps do I perform after powering back up? Is 'replace disk' required now? Or can I somehow clear the faulted flag?
zpool status:
Code:
pool: tank state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub canceled on Tue Jun 11 09:20:52 2019 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/c69917ab-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/7538cd43-8a92-11e9-8f89-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/c8255ea3-5e48-11e6-8818-0cc47a82d3e8.eli FAULTED 6 108 0 too many errors gptid/c8f7431d-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/c9b20cb3-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/ca8fd5bf-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/cb546522-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 gptid/cc2d9b13-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0 errors: No known data errors
command timeout:
Code:
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00 length 20480 SMID 668 terminated ioc 804b scsi 0 state c xfer 0 (da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00 (da2:mpr0:0:2:0): CAM status: CCB request completed with an error (da2:mpr0:0:2:0): Retrying command (da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00 (da2:mpr0:0:2:0): CAM status: SCSI Status Error (da2:mpr0:0:2:0): SCSI status: Check Condition (da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da2:mpr0:0:2:0): Retrying command (per sense data) (da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 468 terminated ioc 804b scsi 0 state c xfer 0 (da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da2:mpr0:0:2:0): CAM status: CCB request completed with an error (da2:mpr0:0:2:0): Retrying command (da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da2:mpr0:0:2:0): CAM status: SCSI Status Error (da2:mpr0:0:2:0): SCSI status: Check Condition (da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da2:mpr0:0:2:0): Error 6, Retries exhausted (da2:mpr0:0:2:0): Invalidating pack
smartctl output:
Code:
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate NAS HDD Device Model: ST4000VN000-2AH166 Serial Number: WDH00MNH LU WWN Device Id: 5 000c50 09bbc66d1 Firmware Version: SC60 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5980 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Jun 24 12:26:06 2019 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 591) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 639) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 082 064 044 Pre-fail Always - 155120576 3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 34 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 092 060 045 Pre-fail Always - 1525862684 9 Power_On_Hours 0x0032 072 072 000 Old_age Always - 24650 (114 44 0) 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 34 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 064 059 040 Old_age Always - 36 (Min/Max 31/37) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 91 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 171 194 Temperature_Celsius 0x0022 036 041 000 Old_age Always - 36 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 24650 (67 113 0) 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 79417164037 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 338669484870 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 24616 - # 2 Extended offline Completed without error 00% 24601 - # 3 Short offline Completed without error 00% 24568 - # 4 Short offline Completed without error 00% 24520 - # 5 Short offline Completed without error 00% 24474 - # 6 Short offline Completed without error 00% 24426 - # 7 Short offline Completed without error 00% 24378 - # 8 Short offline Completed without error 00% 24330 - # 9 Short offline Completed without error 00% 24282 - #10 Extended offline Completed without error 00% 24267 - #11 Short offline Completed without error 00% 24234 - #12 Short offline Completed without error 00% 24187 - #13 Short offline Completed without error 00% 24139 - #14 Short offline Completed without error 00% 24091 - #15 Short offline Completed without error 00% 24066 - #16 Short offline Completed without error 00% 24018 - #17 Short offline Completed without error 00% 23971 - #18 Short offline Completed without error 00% 23922 - #19 Short offline Completed without error 00% 23874 - #20 Extended offline Completed without error 00% 23859 - #21 Short offline Completed without error 00% 23826 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@nas.nyx.io /var/log ❯ smartctl -a /dev/da2 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate NAS HDD Device Model: ST4000VN000-2AH166 Serial Number: WDH00MNH LU WWN Device Id: 5 000c50 09bbc66d1 Firmware Version: SC60 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5980 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Jun 24 12:27:24 2019 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 591) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 639) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 082 064 044 Pre-fail Always - 155120576 3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 34 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 092 060 045 Pre-fail Always - 1525862699 9 Power_On_Hours 0x0032 072 072 000 Old_age Always - 24650 (161 213 0) 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 34 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 064 059 040 Old_age Always - 36 (Min/Max 31/37) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 91 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 171 194 Temperature_Celsius 0x0022 036 041 000 Old_age Always - 36 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 24650 (115 26 0) 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 79417164037 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 338669484870 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 24616 - # 2 Extended offline Completed without error 00% 24601 - # 3 Short offline Completed without error 00% 24568 - # 4 Short offline Completed without error 00% 24520 - # 5 Short offline Completed without error 00% 24474 - # 6 Short offline Completed without error 00% 24426 - # 7 Short offline Completed without error 00% 24378 - # 8 Short offline Completed without error 00% 24330 - # 9 Short offline Completed without error 00% 24282 - #10 Extended offline Completed without error 00% 24267 - #11 Short offline Completed without error 00% 24234 - #12 Short offline Completed without error 00% 24187 - #13 Short offline Completed without error 00% 24139 - #14 Short offline Completed without error 00% 24091 - #15 Short offline Completed without error 00% 24066 - #16 Short offline Completed without error 00% 24018 - #17 Short offline Completed without error 00% 23971 - #18 Short offline Completed without error 00% 23922 - #19 Short offline Completed without error 00% 23874 - #20 Extended offline Completed without error 00% 23859 - #21 Short offline Completed without error 00% 23826 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.