dorfoninternet
Cadet
- Joined
- Jul 18, 2021
- Messages
- 3
I recently replaced 5 drives in my pool due to 4 of them failing S.M.A.R.T. tests and the 5th just having a sudden death. This was after a series of power outages over the last month or so. I also replaced the CableMatters SFF-8087::SFF-8087 with a genuine Supermicro cable. With so many drives failing (3 failed in one day) I thought it might be a bad cable. Though that CablesMatters cable had been in service for about 5 years without issue...
The day after getting through the last drive/cable replacement one of the new drives threw a checksum error. The pool immediately went into repairing mode. It finished the repair but the drive is still showing that 1 checksum error.
What are your thoughts? Run a zpool clear? Replace the drive? Put the original cable back?
pool: vol0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 16K in 18:21:38 with 0 errors on Sun Jul 18 18:21:39 2021
config:
The day after getting through the last drive/cable replacement one of the new drives threw a checksum error. The pool immediately went into repairing mode. It finished the repair but the drive is still showing that 1 checksum error.
What are your thoughts? Run a zpool clear? Replace the drive? Put the original cable back?
pool: vol0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 16K in 18:21:38 with 0 errors on Sun Jul 18 18:21:39 2021
config:
Code:
pool: vol0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 16K in 18:21:38 with 0 errors on Sun Jul 18 18:21:39 2021
config:
NAME STATE READ WRITE CKSUM
vol0 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/1224482e-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/1297ba9c-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/1315e6dd-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/6f2dd230-bb1b-11ea-ab54-0cc47a86a3e4 ONLINE 0 0 0
gptid/140c3e88-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/147bebeb-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/1b48ba9f-e26b-11eb-b4d8-0cc47abab4b1 ONLINE 0 0 1
gptid/157fce85-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/1684a5fc-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/179edafc-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/18b89709-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/716b97f6-e1b5-11eb-b4d8-0cc47abab4b1 ONLINE 0 0 0
gptid/1adcdb80-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/9f74095c-9736-11e9-ac69-0cc47a86a3e4 ONLINE 0 0 0
gptid/a30f7412-e6ed-11e9-bd5b-0cc47a86a3e4 ONLINE 0 0 0
gptid/46c3563d-7aa1-11ea-91d8-0cc47a86a3e4 ONLINE 0 0 0
gptid/1e596246-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/49f230f3-e05b-11eb-b4d8-0cc47abab4b1 ONLINE 0 0 0
gptid/20902071-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
gptid/21a978a1-e136-11e6-b6f4-0cc47a86a3e4 ONLINE 0 0 0
errors: No known data errorsCode:
root@freenas01:~ # glabel status
Name Status Components
gptid/439e1e3f-77bb-11eb-859b-0cc47abab4b1 N/A ada0p1
gptid/4390265c-77bb-11eb-859b-0cc47abab4b1 N/A ada1p1
gptid/1224482e-e136-11e6-b6f4-0cc47a86a3e4 N/A da0p2
gptid/1297ba9c-e136-11e6-b6f4-0cc47a86a3e4 N/A da1p2
gptid/1315e6dd-e136-11e6-b6f4-0cc47a86a3e4 N/A da2p2
gptid/6f2dd230-bb1b-11ea-ab54-0cc47a86a3e4 N/A da3p2
gptid/9f74095c-9736-11e9-ac69-0cc47a86a3e4 N/A da4p2
gptid/1adcdb80-e136-11e6-b6f4-0cc47a86a3e4 N/A da5p2
gptid/18b89709-e136-11e6-b6f4-0cc47a86a3e4 N/A da6p2
gptid/179edafc-e136-11e6-b6f4-0cc47a86a3e4 N/A da7p2
gptid/1684a5fc-e136-11e6-b6f4-0cc47a86a3e4 N/A da8p2
gptid/21a978a1-e136-11e6-b6f4-0cc47a86a3e4 N/A da9p2
gptid/20902071-e136-11e6-b6f4-0cc47a86a3e4 N/A da10p2
gptid/1e596246-e136-11e6-b6f4-0cc47a86a3e4 N/A da11p2
gptid/46c3563d-7aa1-11ea-91d8-0cc47a86a3e4 N/A da12p2
gptid/a30f7412-e6ed-11e9-bd5b-0cc47a86a3e4 N/A da13p2
gptid/140c3e88-e136-11e6-b6f4-0cc47a86a3e4 N/A da14p2
gptid/157fce85-e136-11e6-b6f4-0cc47a86a3e4 N/A da15p2
gptid/147bebeb-e136-11e6-b6f4-0cc47a86a3e4 N/A da16p2
gptid/49f230f3-e05b-11eb-b4d8-0cc47abab4b1 N/A da17p2
gptid/716b97f6-e1b5-11eb-b4d8-0cc47abab4b1 N/A da18p2
gptid/1b48ba9f-e26b-11eb-b4d8-0cc47abab4b1 N/A da19p2
gptid/9f65d774-9736-11e9-ac69-0cc47a86a3e4 N/A da4p1
gptid/6f1288ff-bb1b-11ea-ab54-0cc47a86a3e4 N/A da3p1
gptid/130d3302-e136-11e6-b6f4-0cc47a86a3e4 N/A da2p1
gptid/128f5d4d-e136-11e6-b6f4-0cc47a86a3e4 N/A da1p1
gptid/12194ec3-e136-11e6-b6f4-0cc47a86a3e4 N/A da0p1
Code:
root@freenas01:~ # smartctl --all /dev/da19
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red Pro
Device Model: WDC WD6003FFBX-68MU3N0
Serial Number:
LU WWN Device Id:
Firmware Version: 83.00A83
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Jul 18 23:23:12 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 87) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 648) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0004 133 133 054 Old_age Offline - 92
3 Spin_Up_Time 0x0007 100 100 024 Pre-fail Always - 0
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 5
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000a 100 100 067 Old_age Always - 0
8 Seek_Time_Performance 0x0004 128 128 020 Old_age Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 165
10 Spin_Retry_Count 0x0012 100 100 060 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 5
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 12
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 12
194 Temperature_Celsius 0x0002 136 136 000 Old_age Always - 44 (Min/Max 25/48)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 142 -
# 2 Short offline Completed without error 00% 118 -
# 3 Short offline Completed without error 00% 94 -
# 4 Short offline Completed without error 00% 72 -
# 5 Short offline Completed without error 00% 48 -
# 6 Short offline Completed without error 00% 35 -
# 7 Short offline Completed without error 00% 11 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
