Red Error Light for unrecoverable error but pool still shows healthy

Status
Not open for further replies.

Baenwort

Explorer
Joined
Feb 19, 2015
Messages
93
So I got the following after a reboot:

Code:
Dec  9 09:49:09 thing dhclient: New Routers (em0): 192.168.1.1
Dec  9 22:14:32 thing collectd[3046]: aggregation plugin: Unable to read the current rate of "thing.local/cpu-1/cpu-idle".
Dec  9 22:14:32 thing collectd[3046]: utils_vl_lookup: The user object callback failed with status 2.
Dec  9 22:32:52 thing collectd[3046]: aggregation plugin: Unable to read the current rate of "thing.local/cpu-2/cpu-interrupt".
Dec  9 22:32:52 thing collectd[3046]: utils_vl_lookup: The user object callback failed with status 2.
Dec 10 00:00:00 thing syslog-ng[1496]: Configuration reload request received, reloading configuration;
Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=7420478099537572347
Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=9857921623318705200
Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=18407950380888425658
Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222
Dec 10 00:41:47 thing	 (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 4a 20 00 00 00 d0 00 00 length 106496 SMID 502 terminated ioc 804b scsi 0 state 0 xfer 0
Dec 10 00:41:47 thing (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 4a 20 00 00 00 d0 00 00
Dec 10 00:41:47 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error
Dec 10 00:41:47 thing (da0:mps0:0:0:0): Retrying command
Dec 10 00:41:47 thing (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 49 70 00 00 00 b0 00 00
Dec 10 00:41:47 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error
Dec 10 00:41:47 thing (da0:mps0:0:0:0): SCSI status: Check Condition
Dec 10 00:41:47 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Dec 10 00:41:47 thing (da0:mps0:0:0:0): Info: 0x138284970
Dec 10 00:41:47 thing (da0:mps0:0:0:0): Error 5, Unretryable error
Dec 10 00:41:48 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222
Dec 10 01:17:22 thing	 (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b e0 00 00 a8 00 length 86016 SMID 346 terminated ioc 804b scsi 0 state 0 xfer 0
Dec 10 01:17:22 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b e0 00 00 a8 00
Dec 10 01:17:22 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error
Dec 10 01:17:22 thing (da0:mps0:0:0:0): Retrying command
Dec 10 01:17:22 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b 38 00 00 a8 00
Dec 10 01:17:22 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error
Dec 10 01:17:22 thing (da0:mps0:0:0:0): SCSI status: Check Condition
Dec 10 01:17:22 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Dec 10 01:17:22 thing (da0:mps0:0:0:0): Info: 0x38345b38
Dec 10 01:17:22 thing (da0:mps0:0:0:0): Error 5, Unretryable error
Dec 10 01:17:22 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222
Dec 10 02:50:08 thing	 (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd c0 00 00 98 00 length 77824 SMID 362 terminated ioc 804b scsi 0 state 0 xfer 0
Dec 10 02:50:08 thing (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd c0 00 00 98 00
Dec 10 02:50:08 thing (da1:mps0:0:2:0): CAM status: CCB request completed with an error
Dec 10 02:50:08 thing (da1:mps0:0:2:0): Retrying command
Dec 10 02:50:08 thing (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd 28 00 00 98 00
Dec 10 02:50:08 thing (da1:mps0:0:2:0): CAM status: SCSI Status Error
Dec 10 02:50:08 thing (da1:mps0:0:2:0): SCSI status: Check Condition
Dec 10 02:50:08 thing (da1:mps0:0:2:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Dec 10 02:50:08 thing (da1:mps0:0:2:0): Info: 0x9299dd28
Dec 10 02:50:08 thing (da1:mps0:0:2:0): Error 5, Unretryable error
Dec 10 02:50:09 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=18407950380888425658
Dec 10 03:45:03 thing ZFS: vdev state changed, pool_guid=12363175241056571006 vdev_guid=9903920484091501851
Dec 10 03:45:03 thing ZFS: vdev state changed, pool_guid=12363175241056571006 vdev_guid=4621403864633465360
Dec 10 07:34:15 thing	 (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 71 60 00 00 c8 00 length 102400 SMID 511 terminated ioc 804b scsi 0 state 0 xfer 0
Dec 10 07:34:15 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 71 60 00 00 c8 00
Dec 10 07:34:15 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error
Dec 10 07:34:15 thing (da0:mps0:0:0:0): Retrying command
Dec 10 07:34:15 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 70 a8 00 00 b8 00
Dec 10 07:34:15 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error
Dec 10 07:34:15 thing (da0:mps0:0:0:0): SCSI status: Check Condition
Dec 10 07:34:15 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Dec 10 07:34:15 thing (da0:mps0:0:0:0): Info: 0xfcb70a8
Dec 10 07:34:15 thing (da0:mps0:0:0:0): Error 5, Unretryable error
Dec 10 07:34:15 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222


I also got an alert stating:

Code:
CRITICAL: Dec. 10, 2017, 12:42 a.m. - The volume cousin_itt state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.


However, when I look at the pool it still shows healthy in the "Storage" tab and when I check the Volume Status I find the attached. It shows no unrecoverable error.

I did check the SMART attributes and got the below:

Code:
smartctl -a /dev/da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX11D26H46XE
LU WWN Device Id: 5 0014ee 20d7abe02
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun Dec 10 20:11:20 2017 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				( 4244) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   2) minutes.
Extended self-test routine
recommended polling time:		( 696) minutes.
Conveyance self-test routine
recommended polling time:		(   5) minutes.
SCT capabilities:			  (0x303d) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   207   197   021	Pre-fail  Always	   -	   8641
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   16
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   100   253   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   086   086   000	Old_age   Always	   -	   10868
 10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   16
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   11
193 Load_Cycle_Count		0x0032   197   197   000	Old_age   Always	   -	   9658
194 Temperature_Celsius	 0x0022   129   119   000	Old_age   Always	   -	   23
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 10865		 -
# 2  Short offline	   Completed without error	   00%	 10841		 -
# 3  Short offline	   Completed without error	   00%	 10817		 -
# 4  Extended offline	Interrupted (host reset)	  80%	 10809		 -
# 5  Short offline	   Completed without error	   00%	 10793		 -
# 6  Short offline	   Completed without error	   00%	 10769		 -
# 7  Short offline	   Completed without error	   00%	 10745		 -
# 8  Short offline	   Completed without error	   00%	 10721		 -
# 9  Short offline	   Completed without error	   00%	 10697		 -
#10  Short offline	   Completed without error	   00%	 10673		 -
#11  Extended offline	Completed without error	   00%	 10651		 -
#12  Short offline	   Completed without error	   00%	 10625		 -
#13  Extended offline	Completed without error	   00%	 10603		 -
#14  Short offline	   Completed without error	   00%	 10577		 -
#15  Short offline	   Completed without error	   00%	 10553		 -
#16  Short offline	   Completed without error	   00%	 10529		 -
#17  Short offline	   Completed without error	   00%	 10505		 -
#18  Short offline	   Completed without error	   00%	 10481		 -
#19  Short offline	   Completed without error	   00%	 10457		 -
#20  Extended offline	Completed without error	   00%	 10435		 -
#21  Short offline	   Completed without error	   00%	 10409		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.




Now, I think my next step should be replacing the disk that showed errors I think?
 

Attachments

  • FreeNAS storage error.png
    FreeNAS storage error.png
    11.6 KB · Views: 209

garm

Wizard
Joined
Aug 19, 2017
Messages
1,535
You got checksum errors on da0 and zfs corrected it using the mirror. Could be a fluke and the drive isn’t that old. I would keep an eye on it to make sure it doesn’t grow. Your last extended smart test was interrupted, could be a good idea to run one now.
Maybe you should have a new drive burned in an waiting also, but that’s up to you.
 

Baenwort

Explorer
Joined
Feb 19, 2015
Messages
93
So I ran a another extended smart test and it completed successfully. No further occurrence of the error.

It seems from your post that there isn't much I can do but wait for a failure large enough to leave a mark that I can use to do a warranty return?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,535
You need a failed SMART test I think for a RMA (never had one under warranty).

Could be something other than the drive, a cable glitch or cosmic radiation ^^
 
Status
Not open for further replies.
Top