SOLVED From DEGRADED to HEALTHY without help?

[SebbaG]

Hi folks,

recently I encountered a strange behaviour. I'm running an old version of FreeNAS-9.3-STABLE-201412090314 and I recently discovered my ZFS-Pool in a DEGRADED state (RaidZ2 though). So I ordered a replacement disk and it came one week afterwards. When I was looking at my ZFS-Pool again, to figure out which of the drives I had to exchange, all of a sudden FreeNAS showed the pool in a HEALTHY state again.

Can anyone explain this behaviour? I was really confused: So I continued with:

Code:

zpool status nas-pool

pool: nas-pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
   attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
   using 'zpool clear' or replace the device with 'zpool replace'.
[...]

What I did next is to clear the errors and start scrubbing again:

Code:

zpool clear nas-pool
zpool scrub nas-pool

So far it's showing the following errors on one drive. But I will take another 16h to complete the scrub.

Code:

  nas-pool  ONLINE  0  0  0
	raidz2-0  ONLINE  0  0  0
	gptid/1fda6c4c-c57e-11e6-a1c8-000c29e892e3.eli  ONLINE  0  0  272  (repairing)
	[...]
errors: No known data errors

What do you think? Is the drive damaged and needs to be replaced? I'm really a bit confused now!

Thanks in advance and
greetings
Sebastian

 

[bestboy]

Well, I think the instructions are kind of clear: "Determine if the device needs to be replaced"

Lookup the failing device via
glabel list

and then check its SMART data via
smartctrl -a /dev/${device}

If the device is bad, you need to replace it.

Edit: Oh and your pool is still "healthy" because of ZFS. There was enough redundancy to avoid data loss and mitigate the problem:
"One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected."

 

[SebbaG]

hi,

first of all thx @bestboy for your answer.
I just wondered because it changed state from Degrated to Healthy, but ok.
Nevertheless it repaired around 20.2 GB, so I wonder where this comes from.

Code:

pool: nas-pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
   attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
   using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 20.2G in 14h20m with 0 errors on Sun Sep  3 00:03:01 2017
config:
   NAME  STATE  READ WRITE CKSUM
   nas-pool  ONLINE  0  0  0
	raidz2-0  ONLINE  0  0  0
	gptid/1fda6c4c-c57e-11e6-a1c8-000c29e892e3.eli  ONLINE  0  0 1.07M
	[...]

So second thing is, that I can't see any of the drives having a bad smarctl state. On all drives the test passed. And on the specific drive (/dev/da1) This is the output:

Code:

smartctl -a /dev/da7
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:  ST8000VN0022-2EL112
Serial Number: 
Firmware Version: SC61
User Capacity:  8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  7200 rpm
Form Factor:  3.5 inches
Device is:  Not in smartctl database [for details use: -P showall]
ATA Version is:  ACS-3 (unknown minor revision code: 0x006d)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Sun Sep  3 16:04:39 2017 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)   Offline data collection activity
		   was completed without error.
		   Auto Offline Data Collection: Enabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  567) seconds.
Offline data collection
capabilities:		 (0x7b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  1) minutes.
Extended self-test routine
recommended polling time:	 ( 711) minutes.
Conveyance self-test routine
recommended polling time:	 (  2) minutes.
SCT capabilities:	 (0x50bd)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  079  064  044  Pre-fail  Always  -  87847544
  3 Spin_Up_Time  0x0003  084  084  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  30
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  091  060  045  Pre-fail  Always  -  1364204043
  9 Power_On_Hours  0x0032  093  093  000  Old_age  Always  -  6155 (137 148 0)
 10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  29
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  4295032833
189 High_Fly_Writes  0x003a  100  100  000  Old_age  Always  -  0
190 Airflow_Temperature_Cel 0x0022  058  050  040  Old_age  Always  -  42 (Min/Max 39/47)
191 G-Sense_Error_Rate  0x0032  098  098  000  Old_age  Always  -  4090
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  13
193 Load_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  422
194 Temperature_Celsius  0x0022  042  050  000  Old_age  Always  -  42 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a  003  001  000  Old_age  Always  -  87847544
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0
240 Head_Flying_Hours  0x0000  100  253  000  Old_age  Offline  -  6072 (150 177 0)
241 Total_LBAs_Written  0x0000  100  253  000  Old_age  Offline  -  20600312535
242 Total_LBAs_Read  0x0000  100  253  000  Old_age  Offline  -  59017752571

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I'm not an expert, but I don't see the drive failing because the "Reallocated_Sector_Ct" is 0. As for the other values, it almost looks similiar on the other drives as well. Any idea?

PS: I issued a "smartctl -t long /dev/da7" smart-Test but I have to wait for the results!

 

[bestboy]

The drive does not look too bad to me.
All the Raw_Read_Errors seem to have been recovered by ECC.
Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable and UDMA_CRC_Error_Count are 0.
And the Seek_Error_Rate is also showing 0 errors (1364204043 = 0x0000, 0x51501A0B)
The only value that could indicate a problem is Command_Timeout.

From all this I'd guess the drive is ok and the problem is with its connection. I'd suggest to take a good look at the cable and maybe switch/replace it.

 

[bestboy]

Oh and BTW: You should setup regular SMART checks in the FreeNAS GUI

No self-tests have been logged.

We really hate to read this line of smartctrl output :)

 

[SebbaG]

hi bestboy,

thx again for your comment! I checked the cables and indeed it was a bit loosy, so let's see if the problem is fixed or whether it occurs again in the near future. Another interesting thing is the fact you mentioned about smart-tests. Actually I was thinking to run them on a regular basis, because I configured it to do so. (See screenshot below) smart-services is enabled.

Could it be, that those test do not run correctly, because my disks are not recognized as /dev/ata[0-n] but as /dev/da[0-n] instead? (Just I guess into the wild)
Maybe someone has an idea or hint?

Thx and greetings
sebastian

 

[Stux]

hi bestboy,

thx again for your comment! I checked the cables and indeed it was a bit loosy, so let's see if the problem is fixed or whether it occurs again in the near future. Another interesting thing is the fact you mentioned about smart-tests. Actually I was thinking to run them on a regular basis, because I configured it to do so. (See screenshot below) smart-services is enabled.

View attachment 20445

Could it be, that those test do not run correctly, because my disks are not recognized as /dev/ata[0-n] but as /dev/da[0-n] instead? (Just I guess into the wild)
Maybe someone has an idea or hint?

Thx and greetings
sebastian

You need to select disks to run the tests over in the edit dialog.

 

[SebbaG]

You need to select disks to run the tests over in the edit dialog.

:) OMG, seems I have been tricked by the UI. :( oops, I thought when it listed the drives it will run those test on them, but it seems I was wrong. Just discovered the difference between blue and white highlighting (When you select them they become blue) :)

Thank you for this great hint!

 

[Stux]

And you also need to recheck the right disks are selected when you shuffle disks around...