Unrecoverable Disk Error question.

[mjk79]

Hey Guys,

I just ran into a critical disk error that I haven't seen before. I've replaced disks in the past but this one is weird.

Please take a look at the highlighted section of this screen shot and let me know what this could mean.

Thanks for the help!

 

[m0nkey_]

Can you post the output of zpool status using the code tags.

 

[Ericloewe]

And while we're at it, smartctl -a /dev/da12 - with [CODE] [/CODE] tags please.

 

[mjk79]

The pool just went into Degraded state about an hour ago. Now showing 3.86k under checksum column.

Code:

[root@freenas] ~# zpool status
  pool: Vol1
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
		attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
		using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 5h24m with 0 errors on Thu Oct  6 14:43:35 2016
config:

		NAME											STATE	 READ WRITE CKSUM
		Vol1											DEGRADED	 0	 0	 0
		  raidz2-0									  DEGRADED	 0	 0	 0
			gptid/87c2bb16-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8901ec6d-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/89b837ab-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/bd1d6f1c-cc5c-11e5-87c3-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8b4e0206-8e12-11e5-8e3d-d05099374ad7  DEGRADED	 0	 0 3.86K  too many errors
			gptid/8c20f16a-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8d5cc671-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8e8cc415-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8fc6ac11-8e12-11e5-8e3d-d05099374ad7  ONLINE	   0	 0	 0
			gptid/9a7ae7c5-3e7d-11e6-94bb-d05099374ad6  ONLINE	   0	 0	 0
		cache
		  gptid/91428c9e-8e12-11e5-8e3d-d05099374ad7	ONLINE	   0	 0	 0

errors: No known data errors

  pool: Vol_Backup
 state: ONLINE
  scan: scrub repaired 0 in 3h30m with 0 errors on Sat Oct  1 08:40:19 2016
config:

		NAME											STATE	 READ WRITE CKSUM
		Vol_Backup									  ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/792fcdbf-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/7c6665ad-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/7f52c80a-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/824f3ed1-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8526311f-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/87ef2ba2-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8ad79228-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8e00ce90-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8f6ab128-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/8ffe5dee-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/9091917e-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0
			gptid/91271704-dc1f-11e5-9238-d05099374ad7  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h9m with 0 errors on Fri Sep 30 03:54:09 2016
config:

		NAME										  STATE	 READ WRITE CKSUM
		freenas-boot								  ONLINE	   0	 0	 0
		  gptid/b01c2858-a80a-11e4-a607-d05099374ad6  ONLINE	   0	 0	 0

errors: No known data errors

Code:

[root@freenas] ~# smartctl -a /dev/da12
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68MYMN1
Serial Number:	WD-WX11D35C8ZEV
LU WWN Device Id: 5 0014ee 2b6cc911d
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Fri Oct  7 12:55:42 2016 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				( 4244) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   2) minutes.
Extended self-test routine
recommended polling time:		( 696) minutes.
Conveyance self-test routine
recommended polling time:		(   5) minutes.
SCT capabilities:			  (0x303d) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   222   195   021	Pre-fail  Always	   -	   7883
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   53
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   086   086   000	Old_age   Always	   -	   10356
 10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   53
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   50
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   152
194 Temperature_Celsius	 0x0022   116   101   000	Old_age   Always	   -	   36
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 10300		 -
# 2  Short offline	   Completed without error	   00%	 10086		 -
# 3  Extended offline	Completed without error	   00%	 10002		 -
# 4  Short offline	   Completed without error	   00%	  9918		 -
# 5  Short offline	   Completed without error	   00%	  9750		 -
# 6  Extended offline	Completed without error	   00%	  9666		 -
# 7  Short offline	   Completed without error	   00%	  9582		 -
# 8  Short offline	   Completed without error	   00%	  9343		 -
# 9  Extended offline	Completed without error	   00%	  9259		 -
#10  Short offline	   Completed without error	   00%	  9175		 -
#11  Short offline	   Completed without error	   00%	  9007		 -
#12  Extended offline	Completed without error	   00%	  8923		 -
#13  Short offline	   Completed without error	   00%	  8839		 -
#14  Short offline	   Completed without error	   00%	  8599		 -
#15  Extended offline	Completed without error	   00%	  8515		 -
#16  Short offline	   Completed without error	   00%	  8431		 -
#17  Short offline	   Completed without error	   00%	  8263		 -
#18  Extended offline	Completed without error	   00%	  8179		 -
#19  Short offline	   Completed without error	   00%	  8096		 -
#20  Short offline	   Completed without error	   00%	  7881		 -
#21  Extended offline	Completed without error	   00%	  7797		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[m0nkey_]

The disk looks fine. You can try clearing the error and run another scrub, see if the disk degrades again. If it does, it's time to replace it.

zpool clear <pool>
zpool scrub <pool>

 

[Robert Trevellyan]

Are you absolutely certain that gptid/8b4e0206-8e12-11e5-8e3d-d05099374ad7 == /dev/da12?

If so, given how healthy the disk looks, you might have some other kind of hardware issue, perhaps as simple as a bad cable.