Volume went from DEGRADED to ONLINE - but zpool status shows resilvering of 166 MB, and CHKSUM of 2?

Status
Not open for further replies.
victorhooi

victorhooi

Contributor
Joined
Mar 16, 2012
Messages
184
Hi,

I have a FreeNAS 11.0 system with 4 x 8TB Seagate Archive drives in a RAID-Z1 configuration.

A few days ago, I got an alert in FreeNAS:

> Critical: <Datetime> - Device /dev/ad0, unable to open device.
> Critical: <Datetime> - The volume datastore state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

I looked in the Disks view in the FreeNAS GUI to get the serial number of /dev/ada0. I then powered down the NAS, in order to look at the disk, and confirm which one it was.

I assumed at this time that the disk was kaput and needed to be replaced.

I powered back up the NAS - and then this time, the volume "datastore" seems to be ONLINE.

In alerts, I see:

> CRITICAL: June 29, 2017, 9:10 a.m. - The volume datastore state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

I checked the output of zpool status:

Code:
% sudo zpool status datastore
...
  pool: datastore
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 166M in 0h0m with 0 errors on Thu Jun 29 09:08:39 2017
config:

	NAME											STATE	 READ WRITE CKSUM
	datastore									   ONLINE	   0	 0	 0
	  raidz1-0									  ONLINE	   0	 0	 0
		gptid/1b019b58-5db5-11e6-92fe-10604b92dc14  ONLINE	   0	 0	 0
		gptid/1bd01f4e-5db5-11e6-92fe-10604b92dc14  ONLINE	   0	 0	 0
		gptid/1b586c61-5db5-11e6-92fe-10604b92dc14  ONLINE	   0	 0	 0
		gptid/1c918aec-5db5-11e6-92fe-10604b92dc14  ONLINE	   0	 0	 2

errors: No known data errors


Checking the physical devices to gptid:
Code:
% glabel status
									  Name  Status  Components
gptid/1c918aec-5db5-11e6-92fe-10604b92dc14	 N/A  ada0p2
gptid/1b586c61-5db5-11e6-92fe-10604b92dc14	 N/A  ada1p2
gptid/1bd01f4e-5db5-11e6-92fe-10604b92dc14	 N/A  ada2p2
gptid/1b019b58-5db5-11e6-92fe-10604b92dc14	 N/A  ada3p2
gptid/8134852c-59f1-11e7-9c60-000c29c82ac0	 N/A  da0p1


It seems the disk with 2 in the CKHSUM column is indeed ada0.

What happened here?

My understanding is that resilvering is when ZFS uses the checksum from the other disks to rebuild the damaged data.

What are the suggested next steps?

Regards,
Victor
 
victorhooi

victorhooi

Contributor
Joined
Mar 16, 2012
Messages
184
The hardware is a HP MicroServer Gen8 - so the HDDs just slide in and connect to a backplane at the rear. Hmm, I hadn't thought of cable issues.

I checked the SMART health status - it currently says passed:
Code:
% sudo smartctl -H /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-BETA1 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

I just started a long SMART test - apparently it will take 926 (approx 15 hours) to complete:
Code:
% sudo smartctl -t long /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-BETA1 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 926 minutes for test to complete.
Test will complete after Fri Jun 30 12:43:33 2017

Use smartctl -X to abort test.


This is the output of smartctl -a:
Code:
% sudo smartctl -a /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-BETA1 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate Archive HDD
Device Model:	 ST8000AS0002-1NA17Z
Serial Number:	Z840L2JP
LU WWN Device Id: 5 000c50 0918234a4
Firmware Version: RT17
User Capacity:	8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5980 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Thu Jun 29 21:20:21 2017 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  ( 249)	Self-test routine in progress...
					90% of test remaining.
Total time to complete Offline
data collection:		 (	0) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 926) minutes.
Conveyance self-test routine
recommended polling time:	 (   2) minutes.
SCT capabilities:			(0x30b5)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   116   099   006	Pre-fail  Always	   -	   107435344
  3 Spin_Up_Time			0x0003   092   091   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   17
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   068   060   030	Pre-fail  Always	   -	   172110958621
  9 Power_On_Hours		  0x0032   092   092   000	Old_age   Always	   -	   7797
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   17
183 Runtime_Bad_Block	   0x0032   100   100   000	Old_age   Always	   -	   0
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   064   056   045	Old_age   Always	   -	   36 (Min/Max 34/36)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   0
192 Power-Off_Retract_Count 0x0032   093   093   000	Old_age   Always	   -	   15469
193 Load_Cycle_Count		0x0032   089   089   000	Old_age   Always	   -	   23763
194 Temperature_Celsius	 0x0022   036   044   000	Old_age   Always	   -	   36 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   116   099   000	Old_age   Always	   -	   107435344
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   7088 (150 35 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   15761098589
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   22884561182

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Self-test routine in progress 90%	  7797		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
victorhooi

victorhooi

Contributor
Joined
Mar 16, 2012
Messages
184
The long test I scheduled just completed:
Code:
% sudo smartctl -a /dev/ada0
Password:
Password:
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-BETA1 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate Archive HDD
Device Model:	 ST8000AS0002-1NA17Z
Serial Number:	Z840L2JP
LU WWN Device Id: 5 000c50 0918234a4
Firmware Version: RT17
User Capacity:	8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5980 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Jul  1 04:02:23 2017 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
										was completed without error.
										Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(	0) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.																						 [30/931]
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   1) minutes.
Extended self-test routine
recommended polling time:		( 926) minutes.
Conveyance self-test routine
recommended polling time:		(   2) minutes.
SCT capabilities:			  (0x30b5) SCT Status supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   114   099   006	Pre-fail  Always	   -	   71048736
  3 Spin_Up_Time			0x0003   092   091   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   17
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   068   060   030	Pre-fail  Always	   -	   172116699358
  9 Power_On_Hours		  0x0032   092   092   000	Old_age   Always	   -	   7828
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   17
183 Runtime_Bad_Block	   0x0032   100   100   000	Old_age   Always	   -	   0
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   063   056   045	Old_age   Always	   -	   37 (Min/Max 34/38)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   0
192 Power-Off_Retract_Count 0x0032   093   093   000	Old_age   Always	   -	   15470
193 Load_Cycle_Count		0x0032   089   089   000	Old_age   Always	   -	   23764
194 Temperature_Celsius	 0x0022   037   044   000	Old_age   Always	   -	   37 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   114   099   000	Old_age   Always	   -	   71048736
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   7118 (133 33 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   15811283829
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   22936925065

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	  7822		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

This is super weird - the drive to be magically fine now.

Is there still any reason to treat it with suspicion?
 
A

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Freenas may have lost connection to HDD due to connection issue or possibly a bug.
I would suggest you do a scrub. Some resilvering may happen during the process and it may just be normal.
If scrub is successful then I would not worry too much. But I would still keep an eye for future event such as this one.
 
Status
Not open for further replies.

Similar threads

K
  • Locked
zpool status issues
Replies
6
Views
1K
Rattlebattle79
R
zs3223
  • Locked
Zpool status degraded
Replies
5
Views
2K
SweetAndLow
SweetAndLow
B
Spare Failed During Drive Replacement, pool shows degraded still after successful replacement
Replies
0
Views
715
boltworks
B
N
  • Locked
Degraded Pool; Volume UNAVAIL - What to do next
Replies
15
Views
2K
Stux
Stux
K
  • Locked
critical unrecoverable error, suggestion
Replies
2
Views
1K
Ericloewe
Ericloewe
Top