Yet another degraded pool

[Glorious1]

My pools were doing a scrub today. Toward the end I got a notice that the volume was degraded. zpool status showed:

Code:

  pool: Ark
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
   Sufficient replicas exist for the pool to continue functioning in a
   degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
   repaired.
  scan: scrub repaired 1.19M in 12h24m with 0 errors on Fri Dec  1 13:24:57 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   Ark											 DEGRADED	 0	 0	 0
	 raidz1-0									  DEGRADED	 0	 0	 0
	   gptid/b3097641-89d1-11e4-8e67-0cc47a319aec  ONLINE	   0	 0	 0
	   gptid/b3fff80e-89d1-11e4-8e67-0cc47a319aec  FAULTED	 35   126	 0  too many errors
	   gptid/b4f85047-89d1-11e4-8e67-0cc47a319aec  ONLINE	   0	 0	 0
	   gptid/b5ec52cc-89d1-11e4-8e67-0cc47a319aec  ONLINE	   0	 0	 0

errors: No known data errors

I identified which drive it was and did a conveyance and a short test. Both failed with 90% remaining. Here is the drive smart info:

Code:

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Green
Device Model:	 WDC WD30EZRX-00SPEB0
Serial Number:	WD-WCC4E0748565
LU WWN Device Id: 5 0014ee 25eed58df
Firmware Version: 80.00A80
User Capacity:	3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5400 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Fri Dec  1 19:33:03 2017 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)   Offline data collection activity
				   was suspended by an interrupting command from host.
				   Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)   The previous self-test routine completed
				   without error or no self-test has ever
				   been run.
Total time to complete Offline
data collection:		(43200) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
				   Auto Offline data collection on/off support.
				   Suspend Offline collection upon new
				   command.
				   Offline surface scan supported.
				   Self-test supported.
				   Conveyance Self-test supported.
				   Selective Self-test supported.
SMART capabilities:			(0x0003)   Saves SMART data before entering
				   power-saving mode.
				   Supports SMART auto save timer.
Error logging capability:		(0x01)   Error logging supported.
				   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 ( 433) minutes.
Conveyance self-test routine
recommended polling time:	 (   5) minutes.
SCT capabilities:		   (0x7035)   SCT Status supported.
				   SCT Feature Control supported.
				   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   88
  3 Spin_Up_Time			0x0027   186   179   021	Pre-fail  Always	   -	   7700
  4 Start_Stop_Count		0x0032   097   097   000	Old_age   Always	   -	   3707
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   065   065   000	Old_age   Always	   -	   25738
 10 Spin_Retry_Count		0x0032   100   100   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   100   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   339
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   150
193 Load_Cycle_Count		0x0032   192   192   000	Old_age   Always	   -	   24980
194 Temperature_Celsius	 0x0022   125   113   000	Old_age   Always	   -	   27
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   2
198 Offline_Uncorrectable   0x0030   200   200   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed: read failure	   90%	 25737		 1114406229
# 2  Conveyance offline  Completed: read failure	   90%	 25737		 1114407944
# 3  Short offline	   Completed without error	   00%	 25490		 -
# 4  Short offline	   Completed without error	   00%	 25322		 -
# 5  Short offline	   Completed without error	   00%	 25155		 -
# 6  Short offline	   Completed without error	   00%	 24975		 -
# 7  Short offline	   Completed without error	   00%	 24808		 -
# 8  Extended offline	Completed without error	   00%	 24696		 -
# 9  Short offline	   Completed without error	   00%	 24425		 -
#10  Short offline	   Completed without error	   00%	 24257		 -
#11  Short offline	   Completed without error	   00%	 24089		 -
#12  Extended offline	Completed without error	   00%	 23978		 -
#13  Short offline	   Completed without error	   00%	 23681		 -
#14  Short offline	   Completed without error	   00%	 23513		 -
#15  Short offline	   Completed without error	   00%	 23346		 -
#16  Extended offline	Completed without error	   00%	 23234		 -
#17  Short offline	   Completed without error	   00%	 22950		 -
#18  Short offline	   Completed without error	   00%	 22782		 -
#19  Short offline	   Completed without error	   00%	 22619		 -
#20  Extended offline	Completed without error	   00%	 22508		 -
#21  Short offline	   Completed without error	   00%	 22236		 -

Long ago I had some problems and they were fixed by reseating the cables. So I shut down and did that. When it restarted, zpool status said the drive was resilvering. What? Then a few minutes later I got the critical degradation email again.

Is the drive a goner, or is there a chance still that it's a cable issue?

 

[danb35]

is there a chance still that it's a cable issue?

SMART tests are entirely internal to the drive--cable issues don't cause internal drive failures. If it's failing even a short test, it's in bad shape.

 

[Glorious1]

SMART tests are entirely internal to the drive--cable issues don't cause internal drive failures. If it's failing even a short test, it's in bad shape.

That's very useful information, thank you. My backup drive is resilvering now (man it is frighteningly slow) and two new backups on order.

 

[Chris Moore]

That's very useful information, thank you. My backup drive is resilvering now (man it is frighteningly slow) and two new backups on order.

The SMART test data you posted showed a couple pending sectors and I bet that is just the start, if you were to do a burn-in test on this drive it would probably turn up even more. Is it possible that it would be under warranty?

Edit: I looked up the model number and I see it is a WD Green and only came with a 2 year warranty, which is over based on the power on hours the SMART status showed. Sorry.

 

[Stux]

Dead green drive. Unsurprising :(

 

[Glorious1]

The SMART test data you posted showed a couple pending sectors and I bet that is just the start, if you were to do a burn-in test on this drive it would probably turn up even more. Is it possible that it would be under warranty?

Edit: I looked up the model number and I see it is a WD Green and only came with a 2 year warranty, which is over based on the power on hours the SMART status showed. Sorry.

Yes, you're all over this! Thank you. It's been running more or less continuously for 3 years, and off and on for years before that.

Dead green drive. Unsurprising :(

I replaced it with a Red ;)