Degraded RAID 10 array will not complete resilver

[loknar28]

I have a RAID10 array running on 9.10 where one of the drives has caused the array to degrade. I tried a replace but it will not finish the resilvering process. It appears at time like the progress indicator goes backwards or ends up starting over. When I try to offline or remove the bad drive it "thinks" for awhile before returning to show how it has ignored my request. Any insight would be appreciated.

My specs are below

ASRock C2750D4I AVOTON

Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM

6 x WD Re Datacenter drives(WD3000FYYZ)

Silverstone DS380 Case w/Silverstone SFX 650SX-G

Code:

[root@freenas] ~# zpool status -v
  pool: UberStore
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Dec  8 07:56:09 2018
        82.5M scanned out of 3.39T at 2.95M/s, 335h37m to go
        85.5M resilvered, 0.00% done
config:

    NAME                                              STATE     READ WRITE CKSUM
    UberStore                                         DEGRADED     8     0     0
      mirror-0                                        ONLINE       0     0     0
        gptid/d319d248-3de5-11e8-8d87-d05099c1cb7b    ONLINE       0     0     0
        gptid/6b4cc187-41e9-11e8-8cc3-d05099c1cb7b    ONLINE       0     0     0
      mirror-1                                        DEGRADED     0     0     0
        8125711904661813974                           OFFLINE      0     0     0  was /dev/gptid/a4933167-95e5-11e8-b7cb-d05099c1cb7b
        gptid/e2322185-9624-11e8-b7f0-d05099c1cb7b    ONLINE       0     0     0
      mirror-2                                        ONLINE       8     0     0
        gptid/80ae8e73-ed18-11e8-b7f0-d05099c1cb7b    ONLINE       0     0    16  (resilvering)
        replacing-1                                   ONLINE       8     0     0
          gptid/b66d6c45-f6ae-11e8-8e87-d05099c1cb7b  ONLINE       8     0     3  (resilvering)
          gptid/6cd952fd-f03b-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    16  (resilvering)
          gptid/2ba45419-f38d-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    16  (resilvering)

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Wed Nov 28 03:45:19 2018
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      ada5p2    ONLINE       0     0     0

errors: No known data errors
[root@freenas] ~#

 

[loknar28]

A few days ago I had posted a duplicate of this thread without a pool status output for reference. I have since deleted the other thread since I had not received any replies there yet. If anyone who believes they may have an idea of what could be the problem But needs more information please let me know what reports I need to pull.

 

[Allan Jude]

When you do admin commands like trying to attach or detact additional drives, the resilver/scrub process will start over.

You have a few things going on here:
A) You look to be replacing the failed disk with 2 disks. This is a perfectly acceptable thing to do, but I expect not what you intended. When this resilver completes, you'll have a 3-deep mirror
B) The replace process likely will not detach the replaced drive at this point because of the checksum failures you've encountered. You appear to have errors on both sides of the mirror.

Recommendations:
Once the current process completes (stop poking it and let is run), run another scrub.

Once that is done:
zpool detach UberStore <disks you want to remove>


Note: Don't believe the 'remaining time to scrub/resilver' until it has been running for an hour or so. The speed at the very start is slow.
Also: on newer FreeNAS, sequential scrub/resilver is much faster.

 

[loknar28]

Thank you for your quick response. I had originally run though a complete resilver that took a very long time and when it was done there where a huge number of errors detected. I had been unable to get a replace to work so I had done the Zpool detach command to remove the bad drive. After adding a new drive to the mirror I was able to restore the mirror but large amounts of data were missing. So I added the faulty drive back to the mirror. Now I have three drives in the mirror which indeed was not the intended result.

 

[loknar28]

I figured adding the faulty drive back to the mirror would give it a chance to grab any missing data and replicate it throughout the array. I was just surprised that the data would not be available within the rest of the mirror already since it is striped as well as mirrored.
The other mirror is only degraded since I had to pull one to put the faulty drive back in after back in. Since the new drive was now part of the mirror having problems I did not want to remove it.

 

[loknar28]

GUI View of Resilver

 

[loknar28]

Still resilvering 10 days later. How long is typical?

 

[Mr_N]

Never had to resilver a drive but that sounds like its been going too long, considering my scrub takes like ~6 hrs and a long drive test takes ~10hrs i'm pretty confident you have a problem :(

 

[loknar28]

That has been established

 

[cobrakiller58]

Have the number of errors increased on your pool? if you do not have a full backup of your data I suggest you begin copying everything you do not wish to lose somewhere safe just in case this goes south.

 

[loknar28]

Here is a new zpool status -xv
So as you can see over the last 10 days or so(See first post for first zpool status), believing that this is the same resilver(it is), the array has seen an additional 12 read errors.
Although it does say resilver in progress since Thu Dec 20 16:43:54 2018(today) and the amount of data scanned does reset with the date reported.

I suppose it could be restarting over and over, but none of the outputs have ever comeback with a smaller error count than the last.

Code:

[root@freenas] ~# zpool status -xv
  pool: UberStore
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Dec 20 16:43:54 2018
        35.5M scanned out of 3.39T at 7.10M/s, 139h11m to go
        33.5M resilvered, 0.00% done
config:

    NAME                                              STATE     READ WRITE CKSUM
    UberStore                                         DEGRADED    20     0     0
      mirror-0                                        ONLINE       0     0     0
        gptid/d319d248-3de5-11e8-8d87-d05099c1cb7b    ONLINE       0     0     0
        gptid/6b4cc187-41e9-11e8-8cc3-d05099c1cb7b    ONLINE       0     0     0
      mirror-1                                        DEGRADED     0     0     0
        8125711904661813974                           OFFLINE      0     0     0  was /dev/gptid/a4933167-95e5-11e8-b7cb-d05099c1cb7b
        gptid/e2322185-9624-11e8-b7f0-d05099c1cb7b    ONLINE       0     0     0
      mirror-2                                        ONLINE      20     0     0
        gptid/80ae8e73-ed18-11e8-b7f0-d05099c1cb7b    ONLINE       0     0    40  (resilvering)
        replacing-1                                   ONLINE      20     0     0
          gptid/b66d6c45-f6ae-11e8-8e87-d05099c1cb7b  ONLINE      20     0    22  (resilvering)
          gptid/6cd952fd-f03b-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    40  (resilvering)
          gptid/2ba45419-f38d-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    40  (resilvering)

errors: No known data errors
[root@freenas] ~#

 

[cobrakiller58]

I believe you will have to rebuild your pool from a backup, I would not bet on that resilver finishing. As far as I understand hypothetically if you had not removed the failing drive from the pool and did the replacement it may have been able to reconstruct the mirror even if there are read errors on both sides of the mirror as long as it was not on the same data. Personally I would destroy the pool, burn-in all my drives replace any failing ones, burn-in the new ones and rebuild my pool from my back-up.

 

[loknar28]

Is there a burn-in Procedure specific to FreeNAS? I have new drives to replace the failed ones but I would prefer to first try and save the data since the array primarily consists of several VMWare guests that I will have to rebuild.

 

[loknar28]

I want to say that I really like FreeNAS so please don’t think I am being critical here. I would just like to understand something that has me a bit perplexed.
M Why is there a need for this replace process. With a typical off the shelf NAS RAID Array, when you have a failing drive you replace it and then it rebuilds either from the mirror and/or stripe data. With Free/TrueNAS ZFS I understand that there is an index or other validation archive of some sort it uses as well. It just seems counter intuitive to require a resilver to validate a failed or failing drive before replacing it.

 

[pro lamer]

In your case some data is lost from one drive and some from the other. This is why you need three disks temporarily.

Sent from my phone

 

[cobrakiller58]

For burn-in there is a resource https://forums.freenas.org/index.php?resources/hard-drive-burn-in-testing.92/

All raid array's resilver the new drive when replacing the failed drive in this case (I believe) there were read errors on the drive it was copy data from to the new drive you installed. That is the exact scenario that people warn about with Raid5/RaidZ1 and large drives, it isn't a common occurrence as far as I know but an increasing possibility as the drive sizes/stored data increases. Mirror aren't as susceptible to it as they resilver more quickly but it is still possible and some people mitigate the possibility by doing 3-way mirrors (or more if they can justify it) then there is the fact the Raid is NOT a back-up solution but a way to keep your data accessible as long as possible. That is where the 3-2-1 back-up concept comes into play to help ensure you do not loss data. I don't know much about the intricacies of ZFS but I understand it uses something(someone help me out here) to ensure data integrity, so it will know if data that is being copied has been corrupted but if the other copy of the data is unreadable it has no way to correct it.

 

[pro lamer]

they resilver more quickly

This detail seems moot to me. See the below

The speed of the resilver is affected by the hardware and pool design, but with good design, the limit is how fast the new drive can write data and that doesn't care if it is a mirror or RAIDz.

Sent from my phone

 

[loknar28]

I think I will be blowing away the array and rebuilding.
Having said that, am I missing the point with ZFS. I understand RAID is not a backup solution, but it should provide reliable redundancy for drive failures. Is there a better storage design for Speed with reasonable redundancy. I had built a RAID 10 array for what I thought would be the best balance for this purpose. Sure its a ton of wasted storage space but that was the trade off. I am running this as a FC connected QLogic SAN for a couple of dual connected (2x4Gbps) vmware hosts. The array is built from 6x4TB WD Re drives and I keep a 7th (hot spare) for failures. Any constructive advice would be appreciated as I proceed forward. Also, is there a good reason to move from 9.10 to 11.x. I like the interface of 9.10 better for the most part.

 

[cobrakiller58]

The array is built from 6x4TB WD Re drives and I keep a 7th (hot spare) for failures.

You had a hot spare in the machine? did you have email notifications setup? I'd like to find out how you got to your current situation.
I don't currently use my machine for hosting VM's but others do who would be better for advising you in that regard but essentially I believe you have the right idea. ZFS cannot protect your pool from everything, multiple hardware failures, severe user errors and natural disasters are things I don't think any filesystem can protect against.

 

[Apollo]

Here is a new zpool status -xv
So as you can see over the last 10 days or so(See first post for first zpool status), believing that this is the same resilver(it is), the array has seen an additional 12 read errors.
Although it does say resilver in progress since Thu Dec 20 16:43:54 2018(today) and the amount of data scanned does reset with the date reported.

I suppose it could be restarting over and over, but none of the outputs have ever comeback with a smaller error count than the last.

Code:

[root@freenas] ~# zpool status -xv
  pool: UberStore
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Dec 20 16:43:54 2018
        35.5M scanned out of 3.39T at 7.10M/s, 139h11m to go
        33.5M resilvered, 0.00% done
config:

    NAME                                              STATE     READ WRITE CKSUM
    UberStore                                         DEGRADED    20     0     0
      mirror-0                                        ONLINE       0     0     0
        gptid/d319d248-3de5-11e8-8d87-d05099c1cb7b    ONLINE       0     0     0
        gptid/6b4cc187-41e9-11e8-8cc3-d05099c1cb7b    ONLINE       0     0     0
      mirror-1                                        DEGRADED     0     0     0
        8125711904661813974                           OFFLINE      0     0     0  was /dev/gptid/a4933167-95e5-11e8-b7cb-d05099c1cb7b
        gptid/e2322185-9624-11e8-b7f0-d05099c1cb7b    ONLINE       0     0     0
      mirror-2                                        ONLINE      20     0     0
        gptid/80ae8e73-ed18-11e8-b7f0-d05099c1cb7b    ONLINE       0     0    40  (resilvering)
        replacing-1                                   ONLINE      20     0     0
          gptid/b66d6c45-f6ae-11e8-8e87-d05099c1cb7b  ONLINE      20     0    22  (resilvering)
          gptid/6cd952fd-f03b-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    40  (resilvering)
          gptid/2ba45419-f38d-11e8-b7f0-d05099c1cb7b  ONLINE       0     0    40  (resilvering)

errors: No known data errors
[root@freenas] ~#

I would wait for the resilvering to complete.
While there are a lot of READ errors detected, it is not critical when ZFS can restore redundancy.
So far, "ZFS status" is not reporting any corruption to the data. This means you are in the clear for now.
You are also resilvering several disks at once and your setup (I am not familiar with it in details) is not made up of a fast CPU.
On my Xeon E3, I had a resilver on a 4TB RED take about 4 days to complete, if memory serves.

In a nut shell, I would leave the resilvering move forward until completion.
As already mentioned, don't rely on the ETA provided by "ZFS status".

So far you have no errors reported and this is what matters for now.
Just be patient. If you have some services such as Jails running, I would possibly considered stopping them, at least the one that would cause constant access to the volume, unless there are required, such as Netcloud and the like.
Having infrequent access to the volume should allow for a more efficient resilvering.