ZFS Help: Never Ending Resilver

[craiga_uk]

Hi All,

My ZFS volume is degraded, as a drive had died.
I replaced the drive a couple of weeks ago, and very quickly (within around 12 hours) the Zpool Status command was showing that the resilver was 100% complete.
I did some Googling, which suggested that this wasn't unusual and to give it some time.
I've checked in today (it's a remote server used as a secondary backup, so not totally critical) and it's STILL resilvering.
A zpool status shows the following:

Code:

pool: Data
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 550h23m, 100.00% done, 0h0m to go
config:

        NAME                       STATE     READ WRITE CKSUM
        Data                       DEGRADED     0     0    31
          raidz1                   DEGRADED     0     0    62
            ada0p2                 ONLINE       0     0     0
            replacing              DEGRADED     0     0     0
              8280195129692024807  UNAVAIL      0     0     0  was /dev/ada1p2/o                                              ld
              ada1p2               ONLINE       0     0     0  62.1T resilvered
            ada2p2                 ONLINE       0     0     0
            ada3p2                 ONLINE       0     0     0
            ada4p2                 ONLINE       0     0     0
            ada5p2                 ONLINE       0     0     0
            ada6p2                 ONLINE       0     0     0
            ada7p2                 ONLINE       0     0     0
            ada8p2                 ONLINE       0     0     0
            ada9p2                 ONLINE       0     0     0

Any tips on what to do? Is it still ACTUALLY resilvering, or is it b0rked? :o)
Thanks,
Craig
Oh, it's FreeNAS 8.0.3-RELEASE-P1-x64 (9591)

 

[cyberjock]

First, the output of your zpool status must be included within CODE tags. The text formatting is VERY important.

Second, if your file system is corrupted, it will resilver indefinitely if it happens to get stuck in a loop. This is probably(not not for certain) what happened.

Third, 8.0.3 has nasty security issues. I'd highly recommend you upgrade to 8.3.2. The newer ZFS code may fix your problem. No guarantees though!

Fourth, if what I read above is right you might have as many as 10 disks in a RAIDZ1. This is extremely dangerous as RAIDZ1 is "dead" even with just a few disks. You've put 10 in a single pool, so you were pretty much guaranteed to never be able to resilver a pool of your size if a disk failed. If you don't have backups I'd start making them immediately.

 

[craiga_uk]

Sorry Cyberjock - reformatted.

Actually, this server is designed as a 2nd backup of a main file server (which has some huge 24TB arrays) so the primary design of this box is to get the most storage from the disks as possible, hence the z1 config.

We wanted emergency redundancy, but 2 disks dying isn't an issue.

So, from what you're saying, if there are any checksum errors (which zfs is designed to detect and work around) detected during a resilver then the whole array is a write off?

 

[craiga_uk]

Last newbie question - if I had split the pools down could I have still used them as a single mount point/Rsync target?
If I need to rebuild the box then I'll do it whichever way would make recovery possible, and provide the most disk space. Advice gratefully received!

 

[cyberjock]

We wanted emergency redundancy, but 2 disks dying isn't an issue.

So, from what you're saying, if there are any checksum errors (which zfs is designed to detect and work around) detected during a resilver then the whole array is a write off?

And that's why RAIDZ1 is "dead"(link in my sig). If one disk fails you are all but guaranteed to have a URE while rebuilding the pool. ZFS is designed to detect and correct for errors with which it has sufficient redundancy to do the repair. You have no redundancy if you have a failed disk. So it has no choice but to accept any errors it finds as it cannot fix them.

So when you need redundancy the most is precisely when you don't have it.

What you really have is a backup server that, if it experiences a disk failure, the 'recovery' is to destroy the pool, recreate a new one, then let backups repopulate. The chances of being able to rebuild a 10 disk pool with no redundancy is pretty close to zero(as you are seeing first hand).

Even as a backup server, I'd never do anything less than RAIDZ2, and to be honest I'd probably just go with an 11 disk RAIDZ3. Partly because when you do need to do a recovery from the pool you will work those disks pretty hard. And that's precisely when you are more likely to see a failure. So you are much more likely to see failures when doing the recovery, which really sucks because that's the precise time when you are relying on your server.

And its not like an extra disk really is adding that much more cost. You've already paid for the server and all of the disks. Why avoid that extra $100 or whatever for 1 more disk when it adds so much to the picture.

Overall, just a very bad pool design in my opinion.

 

[craiga_uk]

Hi Cyberjock,

Sorry, I'm now actually on a remote session with the FreeNAS box in question and can confirm it's actually a RaidZ2 config.
I couldn't see your signature earlier as was browsing from an iPhone (the mobile forum view seems to chop off sigs) but will read your article now.

Unfortunately 11 disks in this machine isn't possible due to drive bays, power and SATA controller layouts. 10 drives is the max.

So, do you think it's worth doing the FreeNAS upgrade, or should I just start over? (The only downside is the time it takes to copy 20TB of data back to the box).

If I start over, what's the best drive config/layout for 10 x 3TB disks bearing in mind I need to get at least 21TB of usable storage for RSync backups (ideally in 1 mount point)

 

[cyberjock]

As a backup server I'd feel pretty comfortable with a RAIDZ2. As a primary I'd do a RAIDZ3 or 2 vdevs or RAIDZ2. You can have a little more leeway with backup servers because if they crash you just have to redo the backups. Of course, you DO need them to be reliable enough that they aren't crashing regularly or you might see them crash at the same time as the primary server. If you plan to do RAIDZ2 you should be setting up SMART monitoring, SMART testing, and regular scrubs. Keep in mind you should try not to let zpools go beyond 80% full. Things get nasty and performance can suffer badly if they get too full.

Honestly, if its stuck resilvering the file system is probably having a problem and an upgrade isn't likely to fix it. If the other servers are in good health(I'd verify this!!!) then I'd upgrade to 8.3.2 or 9.1.1, redo the pool as a RAIDZ2 and redo the backups. I'd try to make sure that they are actually backing up properly and such after you make the new pool though. Verify every step of the way. :)

 

[Ed Dack]

I had a similar problem with resilvering. It would reach almost 100% and then start again. Over and over again.

I solved it by taking the drive offline a with following command

zpool offline data0 ada3p2

waiting for the resilvering to timeout and then forcing a scrub

zpool scrub data0

When the scrub completed I re-enabled the drive

zpool online data0 ada3p2

and it resilvered sucessfully. Job done ;)