Help recovering DEGRADED zpool mirror

juanjico · Apr 15, 2019

Hello,

(sorry for my bad English)

I have an old FreeNAS-9.10.2-U6 (561f0d7a1) running on an old HP Prolian MIcroserver (first gen). Using two desktop Seagate 4TB disk in mirror setup. This is the current status:

Code:

  pool: zfs1
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 46h25m with 1 errors on Tue Mar 26 01:26:29 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        zfs1                                            DEGRADED 1.65M     0 7.57K
          mirror-0                                      DEGRADED 6.58M     0 30.3K
            gptid/03cc6e09-df0f-11e5-9697-009c02a7fa32  DEGRADED     0     0 6.61M  too many errors
            2975835688515449866                         UNAVAIL      0     0     0  was /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32
            gptid/b4dd3975-4f65-11e8-ab2f-009c02a7fa32  ONLINE       0     0 6.61M

errors: Permanent errors have been detected in the following files:

        /mnt/zfs1/jails/OpenVPN/tmp
        <0xffffffffffffffff>:<0x0>

Now, the long history.

Last December, I got an email from the FreeNAS server getting lot of smart error in one of the disk. The disk was totally unusable. So, I just physically remove it from the server (my bad, yes). Then installed a new disk (the same, Seagate 4TB), and issued an add command. Resilver started.

This is the first problem: I have a ghost device (2975835688515449866) which I can't remove. The detach command returns:

Code:

[root@freenas] ~# zpool detach zfs1 2975835688515449866
cannot detach 2975835688515449866: no valid replicas
[root@freenas] ~# zpool detach zfs1 /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32
cannot detach /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32: no valid replicas

More problems: The READ, WRITE and CHSUM columns show a lot of errors. But CKSUM columns for the two disk show the same value ALWAYS.

I already checked both disk using smart test, both short and long tests returns no error. This is the results for the two disk:

Code:

[root@freenas] ~# smartctl -a /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM004-2CV104
Serial Number:    ZFN14JSY
LU WWN Device Id: 5 000c50 0aff03e78
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Apr 15 21:13:26 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 484) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   064   006    Pre-fail  Always       -       240686392
  3 Spin_Up_Time            0x0003   097   097   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       16
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1542408167
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       8474 (177 116 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       16
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       0 0 2
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   072   061   040    Old_age   Always       -       28 (Min/Max 25/29)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       276
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       352
194 Temperature_Celsius     0x0022   028   040   000    Old_age   Always       -       28 (0 20 0 0 0)
195 Hardware_ECC_Recovered  0x001a   084   064   000    Old_age   Always       -       240686392
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       8439h+10m+11.326s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       20354682795
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       101287520165

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8458         -
# 2  Short offline       Completed without error       00%      8445         -
# 3  Short offline       Completed without error       00%      8410         -
# 4  Short offline       Completed without error       00%      8362         -
# 5  Short offline       Completed without error       00%      8314         -
# 6  Short offline       Completed without error       00%      8266         -
# 7  Short offline       Completed without error       00%      8218         -
# 8  Short offline       Completed without error       00%      8170         -
# 9  Extended offline    Completed without error       00%      8134         -
#10  Short offline       Completed without error       00%      8051         -
#11  Short offline       Completed without error       00%      8004         -
#12  Short offline       Completed without error       00%      7955         -
#13  Short offline       Completed without error       00%      7907         -
#14  Short offline       Completed without error       00%      7859         -
#15  Short offline       Completed without error       00%      7811         -
#16  Extended offline    Completed without error       00%      7779         -
#17  Short offline       Completed without error       00%      7715         -
#18  Short offline       Completed without error       00%      7667         -
#19  Short offline       Completed without error       00%      7619         -
#20  Short offline       Completed without error       00%      7571         -
#21  Short offline       Completed without error       00%      7523         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@freenas] ~# smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM004-2CV104
Serial Number:    ZFN0GWZD
LU WWN Device Id: 5 000c50 0a4b23fe1
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Apr 15 21:15:55 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 473) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   075   064   006    Pre-fail  Always       -       29218056
  3 Spin_Up_Time            0x0003   097   097   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       8
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1560721189
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       8252 (215 250 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       8
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       0 0 2
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   072   051   040    Old_age   Always       -       28 (Min/Max 26/29)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       233
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       333
194 Temperature_Celsius     0x0022   028   049   000    Old_age   Always       -       28 (0 20 0 0 0)
195 Hardware_ECC_Recovered  0x001a   075   064   000    Old_age   Always       -       29218056
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       8219h+20m+48.654s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       47000063073
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       56545103875

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8236         -
# 2  Short offline       Completed without error       00%      8223         -
# 3  Short offline       Completed without error       00%      8188         -
# 4  Short offline       Completed without error       00%      8143         -
# 5  Short offline       Completed without error       00%      8092         -
# 6  Short offline       Completed without error       00%      8044         -
# 7  Short offline       Completed without error       00%      7997         -
# 8  Short offline       Completed without error       00%      7948         -
# 9  Extended offline    Completed without error       00%      7901         -
#10  Short offline       Completed without error       00%      7830         -
#11  Short offline       Completed without error       00%      7781         -
#12  Short offline       Completed without error       00%      7733         -
#13  Short offline       Completed without error       00%      7685         -
#14  Short offline       Completed without error       00%      7637         -
#15  Short offline       Completed without error       00%      7589         -
#16  Extended offline    Completed without error       00%      7556         -
#17  Short offline       Completed without error       00%      7493         -
#18  Short offline       Completed without error       00%      7445         -
#19  Short offline       Completed without error       00%      7397         -
#20  Short offline       Completed without error       00%      7349         -
#21  Short offline       Completed without error       00%      7301         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I'm not an expert, but I think that disk are ok. So, why such big CKSUM errors ?

I have dismantled the server, cleaning all dust and connectors, and rebuild it to prevent failed cables or connectors (the server is very old). But still getting same errors.

But, I have check the data and seems to be correct. I can't find any corrupted file.

The only problem is the directory tmp from OpenVPN jail, which is listed as permanent error. It's like a black hole. You can copy or create any file, but ls return 0 files.

I already do a scrub command several times, but always ends with 0 recovery and 1 error, which I think is the tmp directory.

I don't know what is the <0xffffffffffffffff>:<0x0> .... searching in Google don't give any clue.

More information: a zpool clear command only start a resilver and ends with same situation. Rebooting the server starts a resilver again and end with same situation. Resilver always go from first disk in list (the old good disk) to the second disk on list (new and good disk).

So, I'm totally stuck with this. I'm out of ideas.

Any help, please ???

Thanks!

Apollo · Apr 15, 2019

"tmp" would be a temp file and maybe only available without anything upon reboot.
The reason you can't detach is because you added the other volume making it a 3 way mirror (I think).
For now, I would worry about re-establishing redundancy which might be already jeopardized.
You vould get your hand on a 4TB WD RED or similar and do a replacement.
Then upon resilvering, you will get more errors, but what counts at the end is eploring the list of potentially corrupted files with "zpool status -v" and decide how much you have lost.

PhiloEpisteme · Apr 15, 2019

I think generally you want to do as much pool management through the UI. There are settings etc that FreeNAS relies upon that you may not know about when issuing commands yourself.

From the looks of the status of your drives I think @Apollo is correct. It looks like you added a third drive to your mirror rather than replacing the single bad drive.

I would suggest you replace that drive ASAP following the User Guide's instructions. The more you resilver and require those disks to work extra hard the increased likelihood you have of another disk failure.

Do you have backups of your data?

SweetAndLow · Apr 15, 2019

This pool is a complete loss. You have metadata corruption from checksum errors on multiple disks. Is that the exact output of zpool status or did you indent a line before making your comment? I think the drive was added as a stripe not a mirror and you definitely did not replace the bad drive.

Apollo · Apr 15, 2019

SweetAndLow said:
This pool is a complete loss. You have metadata corruption from checksum errors on multiple disks. Is that the exact output of zpool status or did you indent a line before making your comment? I think the drive was added as a stripe not a mirror and you definitely did not replace the bad drive.

Not necessarely a loss, depending on the OP steps taken to replace the drives, but it is possible.
As long as the pool redundancy can be restored and scrub is performed after resilver is complete.

juanjico · Apr 16, 2019

Apollo said:
You vould get your hand on a 4TB WD RED or similar and do a replacement.

Just ordered a new WD RED 4TB disk. But, replace what device ? The ghost device ? The old with "too many errors" ?

juanjico · Apr 16, 2019

PhiloEpisteme said:
I think generally you want to do as much pool management through the UI. There are settings etc that FreeNAS relies upon that you may not know about when issuing commands yourself.

Yes, but when the old disk start failling, the GUI was very slow and nearly inaccessible, so I started to work on the command line.

PhiloEpisteme said:
I would suggest you replace that drive ASAP following the User Guide's instructions. The more you resilver and require those disks to work extra hard the increased likelihood you have of another disk failure.

Same question: what drive must I replace ? The ghost device ? Can I replace a removed device ?

PhiloEpisteme said:
Do you have backups of your data?

No. Losing the data will not start a thermonuclear war, but I prefer to not lose it.

juanjico · Apr 16, 2019

SweetAndLow said:
This pool is a complete loss. You have metadata corruption from checksum errors on multiple disks. Is that the exact output of zpool status or did you indent a line before making your comment? I think the drive was added as a stripe not a mirror and you definitely did not replace the bad drive.

But, I can't find any corrupted file in the pool, except for the permanent errors shown at 'zpool -v'. All data seems to be there.

juanjico · Apr 16, 2019

The pool is resilvering again. I don't know why. I can't remember the times that a resilver started by itself. This takes 3 days.

The new disk will arrive tomorrow, so I must wait for the resilver is completed. Or, maybe, can I replace the disk while resilvering ?

PhiloEpisteme · Apr 16, 2019

juanjico said:
But, I can't find any corrupted file in the pool, except for the permanent errors shown at 'zpool -v'. All data seems to be there.

The corrupt files could be in the metadata somewhere or in some part of a file you didn't check. How did you check the files for corruption?

juanjico said:
Yes, but when the old disk start failling, the GUI was very slow and nearly inaccessible, so I started to work on the command line.

That is unfortunate. If you are forced to use the command line you can find good help here on the forums. There are lots of folks that know the correct commands to use if you are totally unable to use the GUI. FreeNAS uses ZFS but expect specific flags etc that google etc may not reveal.

Apollo · Apr 16, 2019

@juanjico, The CLI is I believe the only place you can check for information about the corrupted files.

The command is:
zpool status -v

This will give you the entire list of damagaed files and whatnot of the pool since the last srcub, or resilvering, I believe.

The reason you have so many errors maybe simply due to the disk having been detached so long ago and too much data was changes or maybe it detached a few times but a few sectors are causing those errors to compound.
The resilvering is a normal process and you shouldn't try to prevent it.
Resilvering can take as long as a scrub would normally do, but sometimes, the resilvering takes place only on the damaged area and can be very quick to complete.

When the new drive arrive, DON"T REMOVE any of the old ones.
Turn system down by sending the Turn-off command via GUI, don't force it by power button. It may take a while to shotdown but this is the clean way to do it if you want to avoid more troubles, unless you have a hot swappable enclosure.

Just insert the new drive and when you are in GUI do the following:

1) Go to "Storage" => Select the pool.
2) Select "Status"
3) In the list of disks, select the one that say "UNAVAILABLE", and chose the option the "Replace" option.
4) The new disk should appear in the list. Select it and proceed ahead.

At this time , resilvering should now start on the new drive.

It maybe wise to increase the priority of the resilvering:
This is an option located under "Taks" => "Resilver Priority"
Enable it.
When you do, you system will be even less responsive because the resilvering will be force to take precedence and lower priority taks will be delayed.
I think as a result, it should put less stress on your drive as the heads shouldn't wander doing other tasks that are not resilver related.
It should speed the resilvering process significantly.

Apollo · Apr 16, 2019

@juanjico, When resilvering is in progress, you should try minimizing the use of the system as much as you can. I think reestablishing your redundancy is the primary concern.
You can monitor resilvering progress through Web GUI or CLI with:
zpool status
but if you do, don't over react in the sense that resilvering progress is not necessarily linear. By that I mean, you will see some fast progress saying you are 5hr or similar away from completion and once in a while you might get 100's of hours or even days. Depending on the data and possibly fragmentation, and especially when resilvering has been interupted, it usually doesn't restart fromthe beginning if it was interrupted, instead, the internal counter will cause ETA to be wrong.
Anyway, the point I am trying to make is that, you should not attempt interfering with the resilvering process at any cost because you think something is taking too long to proceed.

I don't know your specs, but you could use "Netdata" to give you some idea of what is happening with the drives. It is actually quite useful thing to do, but maybe you want to keep that for another time.

I think you should let it resilver and forget about it. No point causing adrenaline rush in these conditions.

juanjico · Apr 17, 2019

Thanks to all for your help..

Just received the new disk, powered off the system using the GUI, installed the disk, power on and go to GUI.

The new disk appears in View disks. So, I used the GUI to do the replace of the ghost disk. Now, this is the status from command line:

Code:

[root@freenas] ~# zpool status -v zfs1
  pool: zfs1
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Apr 17 13:43:03 2019
        438M scanned out of 3.04T at 2.00M/s, 442h58m to go
        812M resilvered, 0.01% done
config:

        NAME                                              STATE     READ WRITE CKSUM
        zfs1                                              DEGRADED   110     0     5
          mirror-0                                        DEGRADED   440     0    20
            gptid/03cc6e09-df0f-11e5-9697-009c02a7fa32    ONLINE       0     0   460
            replacing-1                                   UNAVAIL      0     0     0
              2975835688515449866                         UNAVAIL      0     0     0  was /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32
              gptid/d07286d1-6105-11e9-a0f5-009c02a7fa32  ONLINE       0     0     0  (resilvering)
            gptid/b4dd3975-4f65-11e8-ab2f-009c02a7fa32    ONLINE       0     0   460  (resilvering)

errors: Permanent errors have been detected in the following files:

        /mnt/zfs1/jails/OpenVPN/tmp
        <0xffffffffffffffff>:<0x0>

Things seems to go correctly, but for now the resilver process is very slow. But, I will just wait until it finish.

I still curious why CKSUM error number are the same at the old and previous new disk. The today's new disk don't have CKSUM errors.

I will update here when the resilver process end.

Thanks!!!!

Apollo · Apr 17, 2019

As I have mentioned already, there may be some moment where ETA for resilvering is indicating as exceedingly unreasonable amount of time.
This is normal and should speed up later on.
Resilvering is happening on two of the disks.
Errors on "zfs1" is not good I believe, but as long as resilvering is in effect we need to wait.
Just be patient.

juanjico · Apr 17, 2019

@Apollo yes, thanks. The resilvering process already speed up, now "only" 45 hours to go.

Waiting patiently.

Apollo · Apr 17, 2019

Can you post anothe rof the status again?
Have you setup increase priority for resilvering in GUI?

juanjico · Apr 17, 2019

Apollo said:
Can you post anothe rof the status again?
Have you setup increase priority for resilvering in GUI?

Current status:

Code:

[root@freenas] ~# zpool status -v zfs1
  pool: zfs1
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Apr 17 13:43:03 2019
        191G scanned out of 3.04T at 17.6M/s, 47h13m to go
        383G resilvered, 6.15% done
config:

        NAME                                              STATE     READ WRITE CKSUM
        zfs1                                              DEGRADED 1.68K     0     5
          mirror-0                                        DEGRADED 6.72K     0    20
            gptid/03cc6e09-df0f-11e5-9697-009c02a7fa32    ONLINE       0     0 6.74K
            replacing-1                                   UNAVAIL      0     0     0
              2975835688515449866                         UNAVAIL      0     0     0  was /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32
              gptid/d07286d1-6105-11e9-a0f5-009c02a7fa32  ONLINE       0     0     0  (resilvering)
            gptid/b4dd3975-4f65-11e8-ab2f-009c02a7fa32    ONLINE       0     0 6.74K  (resilvering)

errors: Permanent errors have been detected in the following files:

        /mnt/zfs1/jails/OpenVPN/tmp
        <0xffffffffffffffff>:<0x0>
[root@freenas] ~#

I don't have "Resilver Priority" under "Tasks". I have Cron Jobs, Init/Shutdown Scripts, Rsync Tasks and SMART Tests.

Apollo · Apr 17, 2019

My bad, you are on Freenas 9. Just let it run its course then.

juanjico · Apr 20, 2019

Apollo said:
My bad, you are on Freenas 9. Just let it run its course then.

Hello,

Resilver process ended, but old ghost disk is still there:

Code:

[root@freenas] ~# zpool status -v zfs1
  pool: zfs1
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 5.73T in 70h22m with 1 errors on Sat Apr 20 12:05:33 2019
config:

        NAME                                              STATE     READ WRITE CKSUM
        zfs1                                              DEGRADED     0     0 13.6K
          mirror-0                                        DEGRADED     0     0 54.4K
            gptid/03cc6e09-df0f-11e5-9697-009c02a7fa32    DEGRADED     0     0 54.4K  too many errors
            replacing-1                                   DEGRADED 54.4K     0     0
              2975835688515449866                         UNAVAIL      0     0     0  was /dev/gptid/9d79ca97-47f5-11e8-b3a4-009c02a7fa32
              gptid/d07286d1-6105-11e9-a0f5-009c02a7fa32  ONLINE       0     0     0
            gptid/b4dd3975-4f65-11e8-ab2f-009c02a7fa32    ONLINE       0     0 54.4K

errors: Permanent errors have been detected in the following files:

        /mnt/zfs1/jails/OpenVPN/tmp
        <0xffffffffffffffff>:<0x0>
[root@freenas] ~#

Is it normal that replaced disk still listed ?

Apollo · Apr 20, 2019

Let's take this slowly.
So far, we have a lot of checksum errors, not unexpected.
But only one errors during resilvering so this looks good.

What does the command returns:
zpool status -v

if only the one error listed in your screenshot is all there is, then we are in the clear.
Then, what I think we should do is take the following step, only if the one error is the only error listed, otherwise we need to reevaluate things:

1) Clear the status on the previously degraded pool:
zpool clear zfs1

2) Run scrub to verify we don't have problem with the pool after it has been repaired:
zpool scrub zfs1

3) Wait for scrub to complete and check all status are without any type of errors. This will guaranty we have a healthy system.

4) Evaluate what steps to take to remove the unavailable disk.

Maybe someone will comment on this approach and provide a different procedure.[/cmd]

Important Announcement for the TrueNAS Community.

Help recovering DEGRADED zpool mirror

Dabbler

Wizard

Guru

Sweet'NASty

Wizard

Dabbler

Dabbler

Dabbler

Dabbler

Guru

Wizard

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Help recovering DEGRADED zpool mirror"

Similar threads