Hello,
I have a disk in my pool that FAULTED due to 'too many errors'. It is also reporting a UDMA_CRC_Error_Count of 1, Command_Timeout of 1 so I am going to power down my FreeNAS, replace the cable and see if that helps. What next steps do I perform after powering back up? Is 'replace disk' required now? Or can I somehow clear the faulted flag?
zpool status:
command timeout:
smartctl output:
I have a disk in my pool that FAULTED due to 'too many errors'. It is also reporting a UDMA_CRC_Error_Count of 1, Command_Timeout of 1 so I am going to power down my FreeNAS, replace the cable and see if that helps. What next steps do I perform after powering back up? Is 'replace disk' required now? Or can I somehow clear the faulted flag?
zpool status:
Code:
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub canceled on Tue Jun 11 09:20:52 2019
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/c69917ab-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/7538cd43-8a92-11e9-8f89-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/c8255ea3-5e48-11e6-8818-0cc47a82d3e8.eli FAULTED 6 108 0 too many errors
gptid/c8f7431d-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/c9b20cb3-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/ca8fd5bf-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/cb546522-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
gptid/cc2d9b13-5e48-11e6-8818-0cc47a82d3e8.eli ONLINE 0 0 0
errors: No known data errors
command timeout:
Code:
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00 length 20480 SMID 668 terminated ioc 804b scsi 0 state c xfer 0
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command
(da2:mpr0:0:2:0): WRITE(16). CDB: 8a 00 00 00 00 01 30 af d8 c0 00 00 00 28 00 00
(da2:mpr0:0:2:0): CAM status: SCSI Status Error
(da2:mpr0:0:2:0): SCSI status: Check Condition
(da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da2:mpr0:0:2:0): Retrying command (per sense data)
(da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 468 terminated ioc 804b scsi 0 state c xfer 0
(da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command
(da2:mpr0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mpr0:0:2:0): CAM status: SCSI Status Error
(da2:mpr0:0:2:0): SCSI status: Check Condition
(da2:mpr0:0:2:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da2:mpr0:0:2:0): Error 6, Retries exhausted
(da2:mpr0:0:2:0): Invalidating pack
smartctl output:
Code:
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate NAS HDD
Device Model: ST4000VN000-2AH166
Serial Number: WDH00MNH
LU WWN Device Id: 5 000c50 09bbc66d1
Firmware Version: SC60
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jun 24 12:26:06 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 591) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 639) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 082 064 044 Pre-fail Always - 155120576
3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 34
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 092 060 045 Pre-fail Always - 1525862684
9 Power_On_Hours 0x0032 072 072 000 Old_age Always - 24650 (114 44 0)
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 34
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 064 059 040 Old_age Always - 36 (Min/Max 31/37)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 91
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 171
194 Temperature_Celsius 0x0022 036 041 000 Old_age Always - 36 (0 20 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 24650 (67 113 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 79417164037
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 338669484870
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 24616 -
# 2 Extended offline Completed without error 00% 24601 -
# 3 Short offline Completed without error 00% 24568 -
# 4 Short offline Completed without error 00% 24520 -
# 5 Short offline Completed without error 00% 24474 -
# 6 Short offline Completed without error 00% 24426 -
# 7 Short offline Completed without error 00% 24378 -
# 8 Short offline Completed without error 00% 24330 -
# 9 Short offline Completed without error 00% 24282 -
#10 Extended offline Completed without error 00% 24267 -
#11 Short offline Completed without error 00% 24234 -
#12 Short offline Completed without error 00% 24187 -
#13 Short offline Completed without error 00% 24139 -
#14 Short offline Completed without error 00% 24091 -
#15 Short offline Completed without error 00% 24066 -
#16 Short offline Completed without error 00% 24018 -
#17 Short offline Completed without error 00% 23971 -
#18 Short offline Completed without error 00% 23922 -
#19 Short offline Completed without error 00% 23874 -
#20 Extended offline Completed without error 00% 23859 -
#21 Short offline Completed without error 00% 23826 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
root@nas.nyx.io /var/log
❯ smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate NAS HDD
Device Model: ST4000VN000-2AH166
Serial Number: WDH00MNH
LU WWN Device Id: 5 000c50 09bbc66d1
Firmware Version: SC60
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jun 24 12:27:24 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 591) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 639) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 082 064 044 Pre-fail Always - 155120576
3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 34
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 092 060 045 Pre-fail Always - 1525862699
9 Power_On_Hours 0x0032 072 072 000 Old_age Always - 24650 (161 213 0)
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 34
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 064 059 040 Old_age Always - 36 (Min/Max 31/37)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 91
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 171
194 Temperature_Celsius 0x0022 036 041 000 Old_age Always - 36 (0 20 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 24650 (115 26 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 79417164037
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 338669484870
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 24616 -
# 2 Extended offline Completed without error 00% 24601 -
# 3 Short offline Completed without error 00% 24568 -
# 4 Short offline Completed without error 00% 24520 -
# 5 Short offline Completed without error 00% 24474 -
# 6 Short offline Completed without error 00% 24426 -
# 7 Short offline Completed without error 00% 24378 -
# 8 Short offline Completed without error 00% 24330 -
# 9 Short offline Completed without error 00% 24282 -
#10 Extended offline Completed without error 00% 24267 -
#11 Short offline Completed without error 00% 24234 -
#12 Short offline Completed without error 00% 24187 -
#13 Short offline Completed without error 00% 24139 -
#14 Short offline Completed without error 00% 24091 -
#15 Short offline Completed without error 00% 24066 -
#16 Short offline Completed without error 00% 24018 -
#17 Short offline Completed without error 00% 23971 -
#18 Short offline Completed without error 00% 23922 -
#19 Short offline Completed without error 00% 23874 -
#20 Extended offline Completed without error 00% 23859 -
#21 Short offline Completed without error 00% 23826 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
