resilver question on mirrored drives

[djsensui]

I'm currently having issue with my freenas 9.10.
Suddenly i'm having alerts that my disks are being resilvered.

Code:

NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE      31     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/a60d10a0-33c0-11e8-bc03-000c29765447  ONLINE       0     0     0
            gptid/c75756e7-a4c4-11e6-97d6-000c29765447  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/d664dad6-a4c4-11e6-97d6-000c29765447  ONLINE       0     0    33
            gptid/d715ad71-a4c4-11e6-97d6-000c29765447  ONLINE       0     0     0
          mirror-2                                      ONLINE      31     0     8
            gptid/dc3311ff-18b7-11e9-93b9-000c29765447  ONLINE       0     0    67  (resilvering)
            gptid/eb190cbd-a4c4-11e6-97d6-000c29765447  ONLINE      31     0    30  (resilvering)
          mirror-3                                      ONLINE       0     0     0
            gptid/ff3f0754-a4c4-11e6-97d6-000c29765447  ONLINE       0     0     0
            gptid/fff412af-a4c4-11e6-97d6-000c29765447  ONLINE       0     1     8  (resilvering)
          mirror-4                                      ONLINE       0     0     0
            gptid/1319baba-a4c5-11e6-97d6-000c29765447  ONLINE       0     0     0
            gptid/3cf45e81-1b0c-11e9-93b9-000c29765447  ONLINE       0     0     0  (resilvering)
        logs
          mirror-5                                      ONLINE       0     0     0
            gptid/0ad3c651-a4dc-11e6-849d-000c29765447  ONLINE       0     0     0
            gptid/0b59661a-a4dc-11e6-849d-000c29765447  ONLINE       0     0     0

Resilvering keeps on restarting.
On my mirror-2, although the disk are both resilvering, the pool is still working. and those 2 disks are having problem like "unreadable pending sectors".

It is safe to replace it both without the whole pool being destroyed?

What would be my safe way of replacing drives without data corruptions.

 

[Basil Hendroff]

Do you have backups of the data on those mirrors? If not, you might want to think about copying that data off while the brains trust on this forum consider what the next step might be.

 

[Chris Moore]

Suddenly

Is it actually sudden? How long have you been having disk errors that you ignored?

those 2 disks are having problem like "unreadable pending sectors".

How old are those disks?

What would be my safe way of replacing drives without data corruptions.

The following disks most likely need to be replaced.

Code:

            gptid/d664dad6-a4c4-11e6-97d6-000c29765447  ONLINE       0     0    33

            gptid/dc3311ff-18b7-11e9-93b9-000c29765447  ONLINE       0     0    67  (resilvering)
            gptid/eb190cbd-a4c4-11e6-97d6-000c29765447  ONLINE      31     0    30  (resilvering)

            gptid/fff412af-a4c4-11e6-97d6-000c29765447  ONLINE       0     1     8  (resilvering)

            gptid/3cf45e81-1b0c-11e9-93b9-000c29765447  ONLINE       0     0     0  (resilvering)
 

Unfortunately, you will probably loose data between these two and you can't replace either of them until one of them finishes relivering:

Code:

            gptid/dc3311ff-18b7-11e9-93b9-000c29765447  ONLINE       0     0    67  (resilvering)
            gptid/eb190cbd-a4c4-11e6-97d6-000c29765447  ONLINE      31     0    30  (resilvering)
 

It would be useful to look at the output of smartctrl -x for each of the drives in your pool.
Are you able to ssh in with some terminal program?

 

[djsensui]

Unfortunately, you will probably loose data between these two and you can't replace either of them until one of them finishes relivering:

Code:

            gptid/dc3311ff-18b7-11e9-93b9-000c29765447  ONLINE       0     0    67  (resilvering)
            gptid/eb190cbd-a4c4-11e6-97d6-000c29765447  ONLINE      31     0    30  (resilvering)

If I replace both of this at the same time, does the whole pool will be destroyed?

 

[Chris Moore]

If I replace both of this at the same time, does the whole pool will be destroyed?

Yes. You must have one of them in a stable state. Then you could replace the other, but you can never replace both disks in a mirror at the same time. A mirror only provides one disk of parity. That means only one disk can fail without loosing the mirror. If you loose a vdev in the pool, the entire pool is lost.
I have seen a mirror recover where each disk had part of the information needed such that a good set of data was able to be recovered on one of the disks. Just let the resilver keep working. If it can be saved, it will do it, given time.

 

[SweetAndLow]

You conveniently left out the full output from zpool status on purpose. You should post that output and it will probably say your pool need to be restored from backup.

What is your hardware, do you have a controller or cables going bad? Smart data from those disks is also important.

 

[djsensui]

You conveniently left out the full output from zpool status on purpose. You should post that output and it will probably say your pool need to be restored from backup.

What is your hardware, do you have a controller or cables going bad? Smart data from those disks is also important.

the bottom part indicated that there is a permanent errors in certain files which i dont mind losing thats why I left it out.

My hardware is Supermicro SSG-6028R-E1CR12L. Its an 12 bay 2U server.

 

[Chris Moore]

If I replace both of this at the same time, does the whole pool will be destroyed?

Was this able to recover?