ZFS pool issues - is it going to recover?

[bjbishop92]

Hi everyone,

I have been having issues with my storage pool - The pool started having some failures one at a time.
I have added in 3 spares which were used up instantly to recover the pool and has been resilvering.
Storage wise all the disks look to be okay from SMART data.
This is part of one of the 3 vdevs in operation (ZFS RaidZ2).

Is my storage going to recover? Or am i better off remounting the pool as read only and pulling out as much as I can?

 

[Johnny Fartpants]

Can you show us a bit more please. Ideally the entire output of zpool status? Thanks

 

[bjbishop92]

Here it is:

Code:

root@MEDIA01[~]# zpool status -x
  pool: pool0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jun 28 14:18:31 2023
        126T scanned at 2.17G/s, 120T issued at 2.05G/s, 936T total
        811G resilvered, 12.79% done, 4 days 17:08:16 to go
config:

        NAME                                              STATE     READ WRITE CKSUM
        pool0                                             DEGRADED     0     0 55.6M
          raidz2-0                                        DEGRADED     0     0     0
            gptid/ecdd12a7-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/ef9c122b-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/eccf81f5-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/f5290531-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/f8791385-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/f6e030b4-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/fa211f28-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/f4678488-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/f7a6be78-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/fe5f8ca0-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/fceff086-83ce-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            1275254717285153159                           REMOVED      0     0     0  was /dev/gptid/001b3f2c-83cf-11eb-bcbe-3cecef6bb44e
            gptid/05f4b306-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/092168b2-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/09138e64-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/0834d23e-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/09e58fbd-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/0b36d59f-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/0c3db946-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
            gptid/0ca8bd09-83cf-11eb-bcbe-3cecef6bb44e    ONLINE       0     0     0
          raidz2-1                                        ONLINE       0     0     0
            gptid/917d161e-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/90e6cd3d-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/929040a7-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/967624e6-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/95e01f57-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/9c64b980-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/9b3091d0-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/9c57cb2f-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/9ffb0ac8-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/a2ce42d5-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/a639b245-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/a77217fa-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/aad8c284-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/ab5fd5ed-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0    16
            gptid/acce8bcf-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/aea52707-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/369ad69e-7745-11ec-b840-3cecef799940    ONLINE       0     0     0
            gptid/b19198f4-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/b1faaa73-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
            gptid/b292c179-bec2-11eb-a10e-3cecef6bb44e    ONLINE       0     0     0
          raidz2-2                                        DEGRADED     0     0  111M
            gptid/0772b37d-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/0a88fc35-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/0e331cc4-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/0b296d67-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1375fecb-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/140f7c44-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1945a633-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/19c81f1d-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1c3c4e7a-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/15d2d20e-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1cadf6e3-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1af5f84b-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1dd240d5-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/1fbc91fe-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            spare-14                                      DEGRADED 3.38K 3.38K 38.9K
              11217627118659683482                        UNAVAIL      2    59     0  was /dev/da3p2
              418563821624915554                          REMOVED      0     0     0  was /dev/gptid/efbd3321-1574-11ee-a051-3cecef799940
              gptid/ef46d6bd-1578-11ee-a051-3cecef799940  ONLINE       0     0     0
            gptid/2505e336-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            gptid/27af194d-edb6-11eb-b9dd-3cecef6bb44e    ONLINE       0     0     0
            15941238648645753733                          REMOVED      0     0     0  was /dev/da1p2
            replacing-18                                  ONLINE       0     0     1
              gptid/296f082a-edb6-11eb-b9dd-3cecef6bb44e  ONLINE       0     0     0
              gptid/225c9330-1181-11ee-be68-3cecef799940  ONLINE       0     0     0
            spare-19                                      DEGRADED     0     0 69.3K
              11729002520101200174                        REMOVED      0     0     0  was /dev/da0p2
              gptid/efcd4811-1574-11ee-a051-3cecef799940  ONLINE       0     0     0
        spares
          418563821624915554                              UNAVAIL   was /dev/gptid/efbd3321-1574-11ee-a051-3cecef799940
          12290758471181271420                            INUSE     was /dev/gptid/efcd4811-1574-11ee-a051-3cecef799940
          14522713778886610251                            INUSE     was /dev/gptid/ef46d6bd-1578-11ee-a051-3cecef799940

errors: 55770327 data errors, use '-v' for a list

 

[danb35]

20-wide vdevs? Ouch.

 

[Johnny Fartpants]

I think we need full hardware specs before we open this can of worms. With so many disks dropping there is a chance its a hardware issue. How old is the kit? What drives are you using? How quickly did all your failed drives start dropping? PS: when you want to replace a drive you don’t add it as a hot-spare you offline and replace the drive. Hot-spare are there when you have a perfectly working pool and they are essentially a standby for when a disk fails.

 

[Whattteva]

Nearly 5 days to resilver, lawlz. 20-wide vdev and only RAIDZ2 is like asking for trouble. I don't even want to see how long it will take to completely upgrade a whole vdev. Hopefully, you can afford to not use the pool at all, cause that resilvering will take even longer if you have to use the pool at the same time.

 

[Johnny Fartpants]

I run 6 x 15disk Z3 and with 8-10TB drives with the pool at full capacity that took approx 48-72 hours to resilver. Now using 18TB drives and at 50% capacity that takes approx a week so one would assume at full capacity about two weeks. Personally I don’t have a massive problem with that as each vdev is Z3 and I closely monitor drives and have four hot-spares. However 20 disk Z2 would be even outside my comfort zone and I’m curious to hear how this pool got into the state it’s in.

 

[Etorix]

Is my storage going to recover? Or am i better off remounting the pool as read only and pulling out as much as I can?

You've had many issues before, judging by the numbers after "spare" and "replacing", and you may well recover from these as well.
Officially replace the removed/unavailable drives from the GUI, and add two further drives to replace the two drives which have not yet been taken over.

But if you can afford to backup the pool, destroy it and recreate with a safer layout (more vdevs, each no wider than 10-12), that's a much better option.