Possible disk failure impending?

[bobb]

Hey guys,

I've been getting the following message through my daily email report since the easter weekend. I haven't cleared the CKSUM Error yet because I wanted to ask here first.
Can I clear this without any issues or is this an indication that disk gptid/25f7d03f-778d-11e6-b9ee-0cc47a4ce8f1 might be failing?
The report has been unchanged since it first occurred, i.e. it's always showing 1 in the CKSUM column for that specific disk.

Thanks,
bobb

Code:

Checking status of zfs pools:
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RaidZ2Volume  16.2T  14.0T  2.29T		 -	48%	85%  1.00x  ONLINE  /mnt
freenas-boot  14.9G  6.42G  8.45G		 -	  -	43%  1.00x  ONLINE  -

 pool: RaidZ2Volume
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
 scan: scrub repaired 0 in 28h47m with 0 errors on Mon Apr 30 04:47:30 2018
config:

	NAME											STATE	 READ WRITE CKSUM
	RaidZ2Volume									ONLINE	   0	 0	 0
	  raidz2-0									  ONLINE	   0	 0	 0
		gptid/23d8f352-778d-11e6-b9ee-0cc47a4ce8f1  ONLINE	   0	 0	 0
		gptid/248ad81f-778d-11e6-b9ee-0cc47a4ce8f1  ONLINE	   0	 0	 0
		gptid/a17db78a-2cfe-11e7-b54b-0cc47a4ce8f1  ONLINE	   0	 0	 0
		gptid/25f7d03f-778d-11e6-b9ee-0cc47a4ce8f1  ONLINE	   0	 0	 1
		gptid/471efa18-d508-11e6-8f8d-0cc47a4ce8f1  ONLINE	   0	 0	 0
		gptid/275e4458-778d-11e6-b9ee-0cc47a4ce8f1  ONLINE	   0	 0	 0

errors: No known data errors

-- End of daily output --

 

[Ericloewe]

Looks like the disk flipped a bit. It could be something more serious, but you'd need to provide more information ( smartctl -x /dev/whatever the device ID is that corresponds to that GPTID)

 

[joeschmuck]

Check out my little Hard Drive Troubleshooting Guide (in my link) and see if that helps.

 

[bobb]

Looks like the disk flipped a bit. It could be something more serious, but you'd need to provide more information ( smartctl -x /dev/whatever the device ID is that corresponds to that GPTID)

Thanks for taking the time to respond to this! I was thinking of a flipped bit, too. The error has remained at 1. The number 1 seems very specific and it seems unlikely to me that precisely this one bit would be "broken". Even if so, since I use WD RED drives, the drive should have already flagged this bit as broken (if it can't be read from) and ZFS should have already re-written the flipped bit to another one. Correct me if I'm wrong.

Below you'll find the output of smartctl -x /dev/da1. /dev/da1 is the affected drive. I have removed both the disk's serial number and the LU WWN Device Id. I don't think you'll need either one. Please let me know if you do.

Code:

[root@freenas] ~# smartctl -x /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD30EFRX-68AX9N0
Serial Number:	WD-CENSORED
LU WWN Device Id: CENSORED
Firmware Version: 80.00A80
User Capacity:	3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Fri May  4 21:59:21 2018 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection:		 (40380) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 ( 405) minutes.
Conveyance self-test routine
recommended polling time:	 (   5) minutes.
SCT capabilities:			(0x70bd)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAGS	VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate	 POSR-K   200   200   051	-	0
  3 Spin_Up_Time			POS--K   178   177   021	-	6091
  4 Start_Stop_Count		-O--CK   100   100   000	-	337
  5 Reallocated_Sector_Ct   PO--CK   200   200   140	-	0
  7 Seek_Error_Rate		 -OSR-K   200   200   000	-	0
  9 Power_On_Hours		  -O--CK   063   063   000	-	27262
 10 Spin_Retry_Count		-O--CK   100   100   000	-	0
 11 Calibration_Retry_Count -O--CK   100   100   000	-	0
 12 Power_Cycle_Count	   -O--CK   100   100   000	-	311
192 Power-Off_Retract_Count -O--CK   200   200   000	-	43
193 Load_Cycle_Count		-O--CK   200   200   000	-	293
194 Temperature_Celsius	 -O---K   122   108   000	-	28
196 Reallocated_Event_Count -O--CK   200   200   000	-	0
197 Current_Pending_Sector  -O--CK   200   200   000	-	0
198 Offline_Uncorrectable   ----CK   100   253   000	-	0
199 UDMA_CRC_Error_Count	-O--CK   200   200   000	-	0
200 Multi_Zone_Error_Rate   ---R--   200   200   000	-	0
							||||||_ K auto-keep
							|||||__ C event count
							||||___ R error rate
							|||____ S speed/performance
							||_____ O updated online
							|______ P prefailure warning

General Purpose Log Directory Version 1
SMART		   Log Directory Version 1 [multi-sector log support]
Address	Access  R/W   Size  Description
0x00	   GPL,SL  R/O	  1  Log Directory
0x01		   SL  R/O	  1  Summary SMART error log
0x02		   SL  R/O	  5  Comprehensive SMART error log
0x03	   GPL	 R/O	  6  Ext. Comprehensive SMART error log
0x06		   SL  R/O	  1  SMART self-test log
0x07	   GPL	 R/O	  1  Extended self-test log
0x09		   SL  R/W	  1  Selective self-test log
0x10	   GPL	 R/O	  1  SATA NCQ Queued Error log
0x11	   GPL	 R/O	  1  SATA Phy Event Counters log
0x21	   GPL	 R/O	  1  Write stream error log
0x22	   GPL	 R/O	  1  Read stream error log
0x80-0x9f  GPL,SL  R/W	 16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS	  16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS	   1  Device vendor specific log
0xbd	   GPL,SL  VS	   1  Device vendor specific log
0xc0	   GPL,SL  VS	   1  Device vendor specific log
0xc1	   GPL	 VS	  93  Device vendor specific log
0xe0	   GPL,SL  R/W	  1  SCT Command/Status
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 27245		 -
# 2  Short offline	   Completed without error	   00%	 27221		 -
# 3  Short offline	   Completed without error	   00%	 27197		 -
# 4  Short offline	   Completed without error	   00%	 27173		 -
# 5  Short offline	   Completed without error	   00%	 27149		 -
# 6  Short offline	   Completed without error	   00%	 27149		 -
# 7  Short offline	   Completed without error	   00%	 27101		 -
# 8  Short offline	   Completed without error	   00%	 27077		 -
# 9  Short offline	   Completed without error	   00%	 27053		 -
#10  Short offline	   Completed without error	   00%	 27029		 -
#11  Extended offline	Completed without error	   00%	 27014		 -
#12  Short offline	   Completed without error	   00%	 27005		 -
#13  Short offline	   Completed without error	   00%	 26981		 -
#14  Short offline	   Completed without error	   00%	 26957		 -
#15  Short offline	   Completed without error	   00%	 26933		 -
#16  Short offline	   Completed without error	   00%	 26909		 -
#17  Short offline	   Completed without error	   00%	 26885		 -
#18  Short offline	   Completed without error	   00%	 26861		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:				  3
SCT Version (vendor specific):	   258 (0x0102)
SCT Support Level:				   1
Device State:						Active (0)
Current Temperature:					28 Celsius
Power Cycle Min/Max Temperature:	 24/31 Celsius
Lifetime	Min/Max Temperature:	  2/42 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:	 2
Temperature Sampling Period:		 1 minute
Temperature Logging Interval:		1 minute
Min/Max recommended Temperature:	  0/60 Celsius
Min/Max Temperature Limit:		   -41/85 Celsius
Temperature History Size (Index):	478 (418)

Index	Estimated Time   Temperature Celsius
 419	2018-05-04 14:02	28  *********
 ...	..(476 skipped).	..  *********
 418	2018-05-04 21:59	28  *********

SCT Error Recovery Control:
		   Read:	 70 (7.0 seconds)
		  Write:	 70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID	  Size	 Value  Description
0x0001  2			0  Command failed due to ICRC error
0x0002  2			0  R_ERR response for data FIS
0x0003  2			0  R_ERR response for device-to-host data FIS
0x0004  2			0  R_ERR response for host-to-device data FIS
0x0005  2			0  R_ERR response for non-data FIS
0x0006  2			0  R_ERR response for device-to-host non-data FIS
0x0007  2			0  R_ERR response for host-to-device non-data FIS
0x0008  2			0  Device-to-host non-data FIS retries
0x0009  2			3  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2			4  Device-to-host register FISes sent due to a COMRESET
0x000b  2			0  CRC errors within host-to-device FIS
0x000f  2			0  R_ERR response for host-to-device data FIS, CRC
0x0012  2			0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4	  7427578  Vendor specific

Thank you!

 

[Ericloewe]

The number 1 seems very specific and it seems unlikely to me that precisely this one bit would be "broken".

It's not "one bit", it's "one error", meaning "one block". Doesn't really matter, just an FYI.

SMART looks fine. Run zpool clear and keep an eye on the disk.

 

[M H]

I've had this happen once or twice in 4 years with no effect or failure. Exactly "1". I cleared the error and everything kept on chugging.

 

[Ericloewe]

That's bit rot in action. No errors anywhere, but ZFS can tell that the bits were out of place.
Any other filesystem and you'd have corrupted data. With btrfs, you'd have corrupted data and a broken volume, most likely.

 

[Stux]

Time to thank the ZFS developers for preserving your data integrity ;)