Lost one disk, resilvered, but lost data

[ubellavance]

Hi,

We lost a disk in a FreeNAS machine last week. We replaced the drive itself, then did the zpool replace command. Here is the output of the zpool status command for this volume:

Code:

[root@atq-nasr-1] ~# zpool status volume1
  pool: volume1
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver completed after 0h1m with 10180 errors on Mon Feb 11 09:47:04 2013
config:

        NAME                        STATE     READ WRITE CKSUM
        volume1                     DEGRADED     0     0 38.0K
          raidz1                    DEGRADED     0     0  103K
            ada2p2                  UNAVAIL      0     0     0  cannot open
            replacing               DEGRADED     0     0     0
              18279147940933226290  UNAVAIL      0     0     0  was /dev/ada1p2
              ada1                  ONLINE       0     0     0  3.87M resilvered
            ada0p2                  ONLINE       0     0     0
            ada3p2                  ONLINE       0     0     0
            ada4p2                  ONLINE       0     0     0

Now it looks like we have lost data on the volume. I have a backup of part of it, but not all. Using -v shows me files and directories that seems to have been lost. Is it normal to loose data with one drive failure on a raidz volume? Would it be possible to recover some of the data?

Thanks,

 

[ben]

Were you performing scheduled scrubs on that volume? Without regular scrubs to repair data that became corrupted, when you lose a disk those errors can become unrecoverable.

 

[ubellavance]

Hmm, no. No scrubs were performed. I thought that ZFS was doing integrity checks on each writes. Honestly, I did wonder why scrub was available. Is scrubbing very io-intensive?

 

[ben]

A scrub is needed to repair bit rot and the like, and it also checks for errors through the whole filesystem (not just recently written parts). And yes, a scrub will impact performance if the box is under load. That's why the usual scrub recommendation is something like sunday at 2am every 5 weeks.

 

[ubellavance]

Ok, so there is no way I can recover my lost data?

 

[ben]

Do you have backups? If not, you might be out of luck. What does zpool status say after the replacement is done? DEGRADED usually means some functionality can be sustained.

 

[cyberjock]

This may be the result of not using dual redundancy. If you lose 1 disk and any other disk has any bad sectors or bit-rot you WILL have data/file system corruption.

I don't recommend mirrors personally because of the same issue. If you lose 1 disk in the mirror and the other disk has any bad sectors or bitrot you will lose data.

I'm not sure if 3 disk mirrors are supported, but at that point you are paying 3x over for the disk space so I wouldn't recommend that anyway.

Edit: Scrubs are a MUST if you use ZFS. Without it your data may be corrupted and you won't know it until you read it. If you already have a bad disk you are in the "no redundancy and bad data" category.

 

[ben]

3-way mirrors are supported through the command line. cyberjock is right though: if data protection is your priority, two-disk redundancy, regular scrubs, and backups are all required.

 

[ubellavance]

I installed 8.3 on an usb stick and booted on this one instead. I did an auto-import and I can now see (at least some, if not all) my data that disappeared when booted on 8.0.1. Am I lucky? I have copied some files via scp to my computer and I can open them.

 

[ben]

Very lucky indeed. Watch out for more subtle forms of corruption than lost files though. If the checksums don't match, SOMETHING must have changed.