Drive failure - pool offline. How to restore data?

[chrisb81]

Hi all,

I've been using TrueNAS Core this year after upgrading from FreeNAS. I ended up setting up a new pool when I did this - using three 2TB drives. I'm running into trouble though.

A couple of weeks after running TrueNAS, the machine became completely unresponsive - nothing even showing on the local terminal. I had to hard reset it. When it came back online, everything seemed fine.

Now, this is just a testbench server for me at the moment, and I backed up a ton of data from my laptops onto it and then hadn't looked back on the machine again until a couple more weeks had passed. It was in the same state - totally unresponsive. It also seemed to be running pretty hot/using a lot of electricity (I have it on a UPS and the utilisation was pretty high), so I forcibly shut the machine down until I had time to look at it.

So now I've switched it back on - it takes a *long* time to boot up, complaining about read errors on one of the drives. When it finally boots up and I'm able to see the web interface, my pool is showing as offline. I also have errors on two drives according to the notifications:

/dev/ada1, 8 Offline uncorrectable sectors.
/dev/ada1, 8 Currently unreadable (pending) sectors.
/dev/ada4, 760 Offline uncorrectable sectors.
/dev/ada4, 760 Currently unreadble (pending) sectors.

It seems like I have two dead disks in my array? I have a few problems:

1. This was the first time setting up a ZFS arrray, I think I set it up as RAIDZ but honestly I can't really remember. The GUI is just showing the pool as offline and isn't giving me any further details. Is there any way to find out?

2. These 2TB drives were brand new. Two dead disks out of three? Is it possible the unreadable status is due to something else, and the disks aren't actually toast?

3. I don't mind setting this all up from scratch, but I'd like to see if I can retrieve any data from the disks first. Bearing in mind there are unreadable sectors, is there any way to just force it online and copy off what is readable?

Thanks for any help,

Chris.

 

[Etorix]

Please post a detailed description of your hardware and the output of the shell command zpool status -v, preferably within </> Code tags for readability (these are under the … menu in the forum editor). This will tell the layout of the pool.

 

[sretalla]

These 2TB drives were brand new. Two dead disks out of three? Is it possible the unreadable status is due to something else, and the disks aren't actually toast?

It's certainly possible (although not probable) that the errors are due to cabling or a SATA controller issue.

Although the numbers look suspicious, it would just be representative of the amount of data being requested (if all requests are failing).

It will certainly be necessary to look into the SMART data, so we should start with that after the zpool status -v as requested above:

smartctl -a /dev/ada1
smartctl -a /dev/ada4

Also in code tags please.

 

[chrisb81]

Thanks for the replies, I appreciate the help! zpool status - v gave the following info:

Code:

  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:44 with 0 errors on Sun Feb 21 03:45:44 2021
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

errors: No known data errors

I'll be honest, I don't really understand this - am I expecting to see my offline pool listed here?

ADA1 appears to be the drive showing errors, according to the SMART data pasted below. Also note, the time of the last two errors on ADA1 seem to indicate this drive isn't as new as I thought! This is my fault - I think I must have confused myself over which drives in the machine are new.

smartctl -a /dev/ada1:

Code:

smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1CH164
Serial Number:    W1E6E69X
LU WWN Device Id: 5 000c50 06e9fa625
Firmware Version: CC27
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Feb 22 20:21:27 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  592) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 226) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   092   092   006    Pre-fail  Always       -       212002914
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       8
  7 Seek_Error_Rate         0x000f   066   060   030    Pre-fail  Always       -       17199918101
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       10317
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       22
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       570
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1 1 1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   057   043   045    Old_age   Always   In_the_past 43 (Min/Max 27/43 #165)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       366773
194 Temperature_Celsius     0x0022   043   057   000    Old_age   Always       -       43 (0 13 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       7629h+27m+43.834s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       286149412
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       50807670

SMART Error Log Version: 1
ATA Error Count: 593 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 593 occurred at disk power-on lifetime: 10317 hours (429 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:10:57.879  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:54.278  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:50.652  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:47.027  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:43.472  READ DMA EXT

Error 592 occurred at disk power-on lifetime: 10317 hours (429 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:10:54.278  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:50.652  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:47.027  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:43.472  READ DMA EXT
  c8 00 00 80 03 40 e0 00      00:10:43.471  READ DMA

Error 591 occurred at disk power-on lifetime: 10317 hours (429 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:10:50.652  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:47.027  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:43.472  READ DMA EXT
  c8 00 00 80 03 40 e0 00      00:10:43.471  READ DMA
  c8 00 b0 d0 01 40 e0 00      00:10:43.469  READ DMA

Error 590 occurred at disk power-on lifetime: 10317 hours (429 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:10:47.027  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:10:43.472  READ DMA EXT
  c8 00 00 80 03 40 e0 00      00:10:43.471  READ DMA
  c8 00 b0 d0 01 40 e0 00      00:10:43.469  READ DMA
  c8 00 08 c8 01 40 e0 00      00:10:43.469  READ DMA

Error 589 occurred at disk power-on lifetime: 10317 hours (429 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:10:43.472  READ DMA EXT
  c8 00 00 80 03 40 e0 00      00:10:43.471  READ DMA
  c8 00 b0 d0 01 40 e0 00      00:10:43.469  READ DMA
  c8 00 08 c8 01 40 e0 00      00:10:43.469  READ DMA
  c8 00 08 c0 01 40 e0 00      00:10:43.469  READ DMA

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

smartctl -a /dev/ada4:

Code:

smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1CH164
Serial Number:    W1E6CTLJ
LU WWN Device Id: 5 000c50 06e9ec282
Firmware Version: CC27
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Feb 22 20:28:15 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  584) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 212) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   101   099   006    Pre-fail  Always       -       3479104
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       448166
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       6181
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       22
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   058   044   045    Old_age   Always   In_the_past 42 (Min/Max 27/42 #1)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       145
194 Temperature_Celsius     0x0022   042   056   000    Old_age   Always       -       42 (0 13 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1649h+44m+54.384s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       20778747
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       358346

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[sretalla]

So ada1 is in bad shape.

For ada4, it's showing no tests were run.

smartctl -t long /dev/ada4

Then wait for the time returned from the command to come back and check with -a again.

Also check zpool import

 

[chrisb81]

Thanks sretalla!

zpool import responds with the following:

Code:

   pool: HOMES
     id: 8183500318984732181
  state: FAULTED
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        HOMES                                           FAULTED  corrupted data
          raidz1-0                                      DEGRADED
            gptid/80a47cf4-2de2-11eb-b1c2-00101833c49b  UNAVAIL  cannot open
            gptid/80de5a9d-2de2-11eb-b1c2-00101833c49b  ONLINE
            gptid/80ecf36f-2de2-11eb-b1c2-00101833c49b  ONLINE

I've just run smartctl -t long /dev/ada4, it's telling me it should be done in about 3 hours 30 minutes, so I'll post those results as soon as I have them.

Thanks!

Chris.

 

[chrisb81]

OK - and here is the result of smartctl -a /dev/ada4:

Code:

smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1CH164
Serial Number:    W1E6CTLJ
LU WWN Device Id: 5 000c50 06e9ec282
Firmware Version: CC27
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Feb 23 16:00:39 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  584) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 212) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   101   099   006    Pre-fail  Always       -       3508680
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       23
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       448636
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       6187
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       23
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   057   044   045    Old_age   Always   In_the_past 43 (Min/Max 27/45 #1)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       153
194 Temperature_Celsius     0x0022   043   056   000    Old_age   Always       -       43 (0 13 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1655h+42m+08.243s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       20778747
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       375247

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6186         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Should I be able to bring the pool back online by replacing the defective disk?

 

[sretalla]

Should I be able to bring the pool back online by replacing the defective disk?

So there may actually not be an issue with that disk (ada4), so it's only ada1 that's bad.

Check the cabling to ada4.

Then you should be able to import the pool with the -f switch (zpool import -f HOMES) as a first step... I have seen many different outcomes depending on the health of the other disks in the pool, so we may be looking at other options like readonly mounting if that's no good.

 

[chrisb81]

Then you should be able to import the pool with the -f switch (zpool import -f HOMES) as a first step... I have seen many different outcomes depending on the health of the other disks in the pool, so we may be looking at other options like readonly mounting if that's no good.

Attempting to import the pool gave the following error:

cannot import 'HOMES': I/O error

Check the cabling to ada4.

I'll check this in the morning and will run through the import again. Thanks again for your help :)

 

[chrisb81]

OK, well I checked the cabling - all looks good, so I ran zpool import -f HOMES again and got the same error. It tells me to destroy and re-create the pool from a backup source, which I unfortunately don't have!

Is there anything I can do from here? It seems to me that I have (hopefully) two of the three drives in decent shape, so I'm hoping I might at least be able to get some data back.

 

[Hellione]

You said your drives were brand new? Your SMART says 6.000+ and 10.000+ hours for your ada1 and ada4.
What is your third drive? Is this completely not reachable / offline? Tried another sata port? Does it begin to spin up?
If not, i think your data is gone. ada1 has uncorrectable read errors + pending + reallocated sectors. ada? is gone. ada4 is good.
You made some major mistakes. 1. no backup 2. no proper drives 3. no recommended setup (raidz2)

 

[chrisb81]

Thanks for your reply! Appreciate you taking the time to write that up... I'll try to give a response to your points:

You said your drives were brand new? Your SMART says 6.000+ and 10.000+ hours for your ada1 and ada4.

I briefly mentioned this in the post with the SMART data, I did buy new drives for this machine, but I had some older drives in it as well. All are 2TB. My memory of the setup isn't 100%, but somehow or other I must have created the pool with some of the older drives. I honestly thought I had used the new drives for the pool, all I can think is I got my drive assignments mixed up.

What is your third drive? Is this completely not reachable / offline? Tried another sata port? Does it begin to spin up?

As far as I'm aware, the third drive is totally fine - zpool import seems to show it as two drives online, one drive offline, unless I'm misreading it? All the drives seem to spin up and are connected, although I must admit I haven't tried switching SATA ports.

You made some major mistakes. 1. no backup 2. no proper drives 3. no recommended setup (raidz2)

I'm interested in your points here, because when I set up this machine up it was the first time I'd used ZFS (and hence has been a bit of a learning experience), although I've been familiar with RAID for a very long time... so, to answer each point:

1. This was a testbench machine, and I was really setting it up just to try it out - it's hopefully going to be my home NAS, and I am stretched in terms of what equipment I can afford - but I have several rackmount chassis with hotswap drives, UPS and a rack to put it all in, so I've just been trying to make the most of it. I of course totally agree with the need to back up, but I'm really keen to see if I can resurrect even part of what's there, if only just to understand more deeply how this works.

2. No proper drives... I'll be honest, and maybe I'm taking your point too literally, but could you clarify what you mean by 'proper drives'? I mean I know I can buy more expensive drives, (SAS, etc), but I have to work inside the limits of what I've got to hand, and then use that in the best way I can.

3. In my first post, I mentioned that I actually couldn't remember how I originally set these three drives up. I notice that zpool import states raidz1-0, but reading through various information on the web, I couldn't figure out if this meant I had a RAIDZ1 configuration or if this was just some code for the position of the pool. So, and I'm sorry if I'm being a bit stupid here, but when you say 'no recommended setup (raidz2)', I don't know if you're saying that raidz2 is what I should have used, or raidz2 is what I have used?

Sorry for the long reply, for the main part if anyone has any ideas of further steps I can take to even just see what's on the drives, I'd be really grateful. Hopefully I don't think I've really lost anything major, I archived off a bunch of stuff that was taking up space on my laptops and I think I already had backups of most of it. But I would like to understand if it's possible to get this pool back online, even if just a one-off, readonly kind of situation. It's going to help me with my plans going forwards and make sure I have everything set up right when I finally go nuclear and start storing all my data onto NAS rather than scattered across my multiple machines!

Cheers,

Chris.

 

[Hellione]

ok, i would try to switch sata cables (data+power) and boot up. could be a problem of ports/cables. maybe the unavailable drive goes online.
or take all drives, plug it into another machine, including boot drive, and look if all drives become online again.
the problem is, you have raidz1, and more than 1 bad drives. the name raidz1-0 is just a name, but with 3 drives it can only be raidz1.
minimum setup for raidz1 is 3 drives. 2 data drives + 1 parity drive. there can be more data drives, but not less.
a raidz2 would consist of minimum of 4 drives. 2 data drives + 2 parity drives.
practically, parity is distributed over the whole vdev, so the drives are not explicit for data or parity as in unraid for example, but the number of drives planned for parity is the raidz1,2 or 3 level. that says, 1,2 or 3 drives from your vdev can fail at the same time, and nothing is lost. you might experience a slow down, and if configured, you will get a alert mail from the system at this time. i think you did not configure this at your system, and one drive already failed some time ago. now you have 1 unavailable drive (means not reachable, for whatever reason, could be disconnected, exploded,...). at this time the redundancy was gone. and you have a 2nd drive with bad sectors, unreadable sectors, pending sectors, which cant be read.

2)i would use drives made for nas/server/raid purpose. reason is different firmware. a desktop drive thinks it is the only one. so if you want to read a block that is not readable, the drive does this: "oh my god, i cant read, try again, omg i cant read, try again.... " and so on. if you ever used windows, system hangs for seconds or minutes, or crashes. the raid drive would do this: "omfg i cant read, hey controller go on, do your job, i cant read this". Firmware features like Time-Limited Error Recovery (TLER) or Command Completion Time Limit (CCTL) or Error Recovery Control (ERC) are the main reason for "proper" drives. Some also have better mechanics, built to run 24/7, or absorb vibrations, and so on.
There is no reason for SAS, you can also use "proper" SATA drives.

But you can use desktop drives for sure. As i would not trust them, only use with raidz2 or raidz3. The chance one or more is failing over time, is higher.