Zpool clear not working

[Wizzard]

Hello.
So problem is that freenas alert me about errors. I checked zpool status and found that there are writing errors, check smartctl but it was ok, so i decided to watch this disk and clear errors, i make
#zpool clear whole

So number of errors was reset to 0 but status anyway says that pool degraded:

Code:

  pool: whole
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub repaired 0 in 0 days 00:40:53 with 0 errors on Thu Nov 28 00:23:54 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        whole                                           DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/7a30eef0-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/7c1cd2f8-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/7e116df1-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/801ec82b-aef1-11e9-85f3-00259092f5da  FAULTED      0     0     0  too many errors
            gptid/8207240e-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0

errors: No known data errors

I even run scrub after it, and when it finished, now new errors 0 0 0.
How i could make this disk online?
I clear errors, run scrub no errors, run smart no errors, but disk still FAULTED.... why?

 

[Chris Moore]

Could be a bad drive. Let us do some testig. Please show us the results of smartctl -x /dev/XXX, substituting the drive number in question for the XXX. It might be a good idea to look at all the drives.

 

[Jailer]

You have a faulted disk that has dropped from your pool. You'll need to identify what disk that is and likely replace it.

 

[Wizzard]

Problem of this whole situation that iam not sure that problem in disk hardware...
smartctl:

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM004-2CV104
Serial Number:    ZFN25NGV
LU WWN Device Id: 5 000c50 0b6548a47
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Thu Nov 28 20:11:51 2019 +07
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 497) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   073   064   006    -    19924557
  3 Spin_Up_Time            PO----   097   096   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    17
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   086   060   045    -    445934123
  9 Power_On_Hours          -O--CK   097   097   000    -    2995 (176 103 0)
10 Spin_Retry_Count        PO--C-   100   100   097    -    0
12 Power_Cycle_Count       -O--CK   100   100   020    -    17
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   001   000    -    0 339 43565
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   065   062   040    -    35 (Min/Max 30/37)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    98
193 Load_Cycle_Count        -O--CK   100   100   000    -    136
194 Temperature_Celsius     -O---K   035   040   000    -    35 (0 28 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   073   064   000    -    19924557
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   174   140   000    -    43570
240 Head_Flying_Hours       ------   100   253   000    -    2989h+50m+04.491s
241 Total_LBAs_Written      ------   100   253   000    -    7114875167
242 Total_LBAs_Read         ------   100   253   000    -    4169627834
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    512  Current Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      24  Device vendor specific log
0xa2       GPL     VS    8160  Device vendor specific log
0xa6       GPL     VS     192  Device vendor specific log
0xa8-0xa9  GPL,SL  VS     136  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xb0       GPL     VS    9048  Device vendor specific log
0xbd       GPL     VS       8  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL,SL  VS      16  Device vendor specific log
0xc3       GPL,SL  VS       8  Device vendor specific log
0xc4       GPL,SL  VS      24  Device vendor specific log
0xd1       GPL     VS     264  Device vendor specific log
0xd3       GPL     VS    1920  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      2987         -
# 2  Short offline       Completed without error       00%      2956         -
# 3  Short offline       Completed without error       00%      2932         -
# 4  Short offline       Completed without error       00%      2908         -
# 5  Extended offline    Completed without error       00%      2897         -
# 6  Short offline       Completed without error       00%      2860         -
# 7  Short offline       Completed without error       00%      2836         -
# 8  Extended offline    Completed without error       00%      2825         -
# 9  Short offline       Completed without error       00%      2788         -
#10  Short offline       Completed without error       00%      2766         -
#11  Short offline       Completed without error       00%      2742         -
#12  Extended offline    Completed without error       00%      2733         -
#13  Short offline       Completed without error       00%      2694         -
#14  Short offline       Completed without error       00%      2670         -
#15  Extended offline    Interrupted (host reset)      00%      2660         -
#16  Short offline       Completed without error       00%      2622         -
#17  Short offline       Completed without error       00%      2598         -
#18  Short offline       Completed without error       00%      2574         -
#19  Extended offline    Completed without error       00%      2565         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    35 Celsius
Power Cycle Min/Max Temperature:     30/37 Celsius
Lifetime    Min/Max Temperature:     28/38 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         3 minutes
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     14/55 Celsius
Min/Max Temperature Limit:           10/60 Celsius
Temperature History Size (Index):    128 (116)

Index    Estimated Time   Temperature Celsius
117    2019-11-23 15:07    36  *****************
...    ..( 12 skipped).    ..  *****************
   2    2019-11-24 03:54    36  *****************
   3    2019-11-24 04:53    35  ****************
   4    2019-11-24 05:52    36  *****************
...    ..( 15 skipped).    ..  *****************
  20    2019-11-24 21:36    36  *****************
  21    2019-11-24 22:35    35  ****************
  22    2019-11-24 23:34    36  *****************
  23    2019-11-25 00:33    36  *****************
  24    2019-11-25 01:32    35  ****************
  25    2019-11-25 02:31    35  ****************
  26    2019-11-25 03:30    35  ****************
  27    2019-11-25 04:29    36  *****************
  28    2019-11-25 05:28    35  ****************
  29    2019-11-25 06:27    36  *****************
  30    2019-11-25 07:26    35  ****************
...    ..(  4 skipped).    ..  ****************
  35    2019-11-25 12:21    35  ****************
  36    2019-11-25 13:20    36  *****************
  37    2019-11-25 14:19    36  *****************
  38    2019-11-25 15:18    35  ****************
  39    2019-11-25 16:17    35  ****************
  40    2019-11-25 17:16    36  *****************
...    ..( 12 skipped).    ..  *****************
  53    2019-11-26 06:03    36  *****************
  54    2019-11-26 07:02    35  ****************
  55    2019-11-26 08:01    35  ****************
  56    2019-11-26 09:00    35  ****************
  57    2019-11-26 09:59    36  *****************
...    ..(  8 skipped).    ..  *****************
  66    2019-11-26 18:50    36  *****************
  67    2019-11-26 19:49    35  ****************
  68    2019-11-26 20:48    36  *****************
  69    2019-11-26 21:47    36  *****************
  70    2019-11-26 22:46    35  ****************
  71    2019-11-26 23:45    35  ****************
  72    2019-11-27 00:44    36  *****************
  73    2019-11-27 01:43    35  ****************
  74    2019-11-27 02:42    35  ****************
  75    2019-11-27 03:41    36  *****************
...    ..(  3 skipped).    ..  *****************
  79    2019-11-27 07:37    36  *****************
  80    2019-11-27 08:36    35  ****************
  81    2019-11-27 09:35    35  ****************
  82    2019-11-27 10:34    36  *****************
  83    2019-11-27 11:33    36  *****************
  84    2019-11-27 12:32    36  *****************
  85    2019-11-27 13:31    35  ****************
  86    2019-11-27 14:30    35  ****************
  87    2019-11-27 15:29    35  ****************
  88    2019-11-27 16:28    36  *****************
...    ..(  3 skipped).    ..  *****************
  92    2019-11-27 20:24    36  *****************
  93    2019-11-27 21:23    35  ****************
  94    2019-11-27 22:22    36  *****************
...    ..(  5 skipped).    ..  *****************
100    2019-11-28 04:16    36  *****************
101    2019-11-28 05:15    37  ******************
...    ..(  2 skipped).    ..  ******************
104    2019-11-28 08:12    37  ******************
105    2019-11-28 09:11    36  *****************
...    ..(  2 skipped).    ..  *****************
108    2019-11-28 12:08    36  *****************
109    2019-11-28 13:07    35  ****************
...    ..(  5 skipped).    ..  ****************
115    2019-11-28 19:01    35  ****************
116    2019-11-28 20:00    34  ***************

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              17  ---  Lifetime Power-On Resets
0x01  0x010  4            2995  ---  Power-on Hours
0x01  0x018  6      7114875167  ---  Logical Sectors Written
0x01  0x020  6       195056887  ---  Number of Write Commands
0x01  0x028  6      4169609836  ---  Logical Sectors Read
0x01  0x030  6        25010673  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            2993  ---  Spindle Motor Power-on Hours
0x03  0x010  4            2993  ---  Head Flying Hours
0x03  0x018  4             136  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              98  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4           43565  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              35  ---  Current Temperature
0x05  0x010  1              35  ---  Average Short Term Temperature
0x05  0x018  1              34  ---  Average Long Term Temperature
0x05  0x020  1              38  ---  Highest Temperature
0x05  0x028  1              31  ---  Lowest Temperature
0x05  0x030  1              36  ---  Highest Average Short Term Temperature
0x05  0x038  1              31  ---  Lowest Average Short Term Temperature
0x05  0x040  1              34  ---  Highest Average Long Term Temperature
0x05  0x048  1              33  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              55  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1              13  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4           43720  ---  Number of Hardware Resets
0x06  0x010  4              32  ---  Number of ASR Events
0x06  0x018  4           43580  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) supported [please try: '-l defects']

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2         1277  Device-to-host register FISes sent due to a COMRESET
0x0001  2         1272  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2         1274  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2           20  R_ERR response for host-to-device non-data FIS

 

[garm]

That’s not a happy spinner, replace asap. If it was me I would leave the server (alt pool) offline until you can start resilvering a burned in drive

 

[cobrakiller58]

Problem of this whole situation that iam not sure that problem in disk hardware...
smartctl:

Code:

0x06  0x018  4           43580  ---  Number of Interface CRC Errors

uh isn't that a little high...? have you tried this drive on a new cable and/or port?

 

[Wizzard]

uh isn't that a little high...? have you tried this drive on a new cable and/or port?

No, but for two days and zpool status still shows 0 errors. However it also says that disk in pool, so it working but no additional errors.

 

[garm]

That’s not how it works.. you want a nasty surprise and risk all your data? Run zpool scrub poolName, but seriously don’t!! Just do what I said earlier..

 

[cobrakiller58]

it is not currently accessing the drive because it is in a faulted state.

 

[Wizzard]

it is not currently accessing the drive because it is in a faulted state.

Why then i get such message? If i want to reuse this drive and watch how it perform?

Code:

% sudo zpool replace whole  gptid/801ec82b-aef1-11e9-85f3-00259092f5da
invalid vdev specification
use '-f' to override the following errors:
/dev/gptid/801ec82b-aef1-11e9-85f3-00259092f5da is part of active pool 'whole'

That’s not how it works.. you want a nasty surprise and risk all your data? Run zpool scrub poolName, but seriously don’t!! Just do what I said earlier..

You could see last scrub date time in initial message and it is same. Scrub was completed after failed disk, after error clean and after smartcheck 0 errors.

 

[Wizzard]

uh isn't that a little high...? have you tried this drive on a new cable and/or port?

Yes this could be a problem i will replace cable, other 4 disks show 0 in this param.

 

[Jailer]

The drive is FAULTED. That means it's dropped out of the pool because it's dead or dying. Replace the drive.

How many people do you need to tell you that before you realize what you need to do?

 

[Wizzard]

The drive is FAULTED. That means it's dropped out of the pool because it's dead or dying. Replace the drive.

How many people do you need to tell you that before you realize what you need to do?

Hey man, calm down. I need to understand why, what the point to change drive if cable failed?

 

[Wizzard]

Problem solved for now.
Cable replaced, after reboot, zpool status updated itself now its online 0 errors. I will watch this drive and interface error number.

 

[garm]

if your not gonna do anything else about this, run the scrub

 

[Wizzard]

if your not gonna do anything else about this, run the scrub

Done, 0 errors.... strange thing...

 

[Wizzard]

So if anyone has interest, 2 weeks pass, and state:

Code:

       NAME                                            STATE     READ WRITE CKSUM
        whole                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/7a30eef0-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/7c1cd2f8-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/7e116df1-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0
            gptid/801ec82b-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     9
            gptid/8207240e-aef1-11e9-85f3-00259092f5da  ONLINE       0     0     0

It was usual load, several smart tests with OK result. Yes Probably this disk not perfect, but far from trash. This 9 CKSUM errors happen one and half week ago and still not changed.