The volume [Name] state is DEGRADED: One or more devices could not be opened.

[nissnn]

Hello Everybody,

Today I was cleaning my NAS Build and when powering it back on afterwards one drive was not showing up and this error was showing:

Code:

The volume Speicherplatz state is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

I shut down the system and checked all the cabling, when powering on again the drive showed up but I still get the same error.

Running zpool Status returns this:

Code:

 pool: Speicherplatz
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: none requested
config:

        NAME                                                STATE     READ WRITE CKSUM
        Speicherplatz                                       DEGRADED     0     0     0
          raidz1-0                                          DEGRADED     0     0     0
            gptid/d3490947-4009-11e9-b436-bcaec5dcfcad.eli  ONLINE       0     0     0
            15070291121131822168                            UNAVAIL      0     0     0  was /dev/gptid/d46c4f36-4009-11e9-b436-bcaec5dcfcad.eli
            gptid/d5b48372-4009-11e9-b436-bcaec5dcfcad.eli  ONLINE       0     0     0
            gptid/d6f078d8-4009-11e9-b436-bcaec5dcfcad.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Thu Mar 14 03:45:03 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada3p2    ONLINE       0     0     0

errors: No known data errors

The drive that was not showing up and is now unavailable seems to be fine:

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Enterprise Capacity 3.5 HDD
Device Model:     ST10000NM0086-2AA101
Serial Number:    ZA21ET7T
LU WWN Device Id: 5 000c50 074424642
Firmware Version: SN05
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Mar 20 20:42:40 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 950) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   080   064   044    Pre-fail  Always       -       108588180
  3 Spin_Up_Time            0x0003   091   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   082   060   045    Pre-fail  Always       -       171839681
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       687
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       37
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   064   064   000    Old_age   Always       -       36
190 Airflow_Temperature_Cel 0x0022   065   053   040    Old_age   Always       -       35 (Min/Max 34/35)
191 G-Sense_Error_Rate      0x0032   098   098   000    Old_age   Always       -       5539
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       142
194 Temperature_Celsius     0x0022   035   047   000    Old_age   Always       -       35 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   080   064   000    Old_age   Always       -       108588180
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       683h+37m+44.452s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       110853972576
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       138308350991

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       684         -
# 2  Extended offline    Interrupted (host reset)      00%       655         -
# 3  Extended offline    Completed without error       00%       636         -
# 4  Extended offline    Aborted by host               20%       609         -
# 5  Extended offline    Completed without error       00%       589         -
# 6  Extended offline    Completed without error       00%       560         -
# 7  Extended offline    Interrupted (host reset)      60%       493         -
# 8  Extended offline    Interrupted (host reset)      80%       466         -
# 9  Extended offline    Interrupted (host reset)      90%       441         -
#10  Extended offline    Interrupted (host reset)      80%       422         -
#11  Extended offline    Completed without error       00%        16         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Onlining it doesn't work I just get:

Code:

warning: device '15070291121131822168' onlined, but remains in faulted state

Is it possible that the drive is just not properly recognised because it was missing once and can I somehow reconnect it?

Thanks for reading,

nissnn

 

[Mr. Slumber]

Which FreeNAS version do you use?

 

[nissnn]

Hi, thanks for replying!
I'm on version 11.2-U2.1.
Sorry should have included that.

 

[Mr. Slumber]

Today I was cleaning my NAS Build

Sorry for asking, but you used compressed air for that? Not a "swiffer" kind of thing? I had a client once who did just that and really killed his system due to static charges... ;) I'm sure it's something else but just asking.

 

[nissnn]

Yes, just blew all the dust out ;) I disconnected the drives and I think one wasn't properly connected afterwards, so it wasn't recognised when I first booted up again.

 

[Mr. Slumber]

I disconnected the drives and I think one wasn't properly connected afterwards

Not sure but maybe this video may be of a little help. When I was building my penultimate FreeNAS I tried very many things on purpose to see how bulletproof FreeNAS is (it's very very much! :)) and so, beneath other things, I ripped a live sata cable out, rebooted, had to reattach the drive but everything was working. Keep my fingers crossed for you that nothing really destructive happened to your system. :)

 

[nissnn]

Okay, thanks! I guess I'll just run a few tests on that drive and then wipe and resilver it...

 

[Mr. Slumber]

Good luck! :)

 

[Mr. Slumber]

Did it work out?

 

[nissnn]

Resilvering successfully finished tonight and everything seems to be working fine!

Thanks for your help!!

 

[Mr. Slumber]

Cool! Happpy it worked out! :)