Unable to import zpool, missing disk

[jallis]

I had a raidz1 pool go down with two disks dying on me.

I managed to recover all but 300k worth of data from one of the disks, and image it onto a new disk (hoping the 300k that was replaced with 0x00 would not make a big deal)

Putting in the recovered disk, and a fresh empy disk I try to import the volume:

Code:

[root@freenas] ~# zpool import
  pool: storage
  id: 6529383252281524190
  state: FAULTED
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
   devices and try again.
   The pool may be active on another system, but can be imported using
   the '-f' flag.
  see: http://illumos.org/msg/ZFS-8000-3C
 config:

   storage  FAULTED  corrupted data
    raidz1-0  FAULTED  corrupted data
    gptid/bd193c14-7ac7-11e3-9f0e-0025225aafce  ONLINE
    4259204251121798783  UNAVAIL  cannot open
    gptid/64928b4e-7aac-11e3-a88d-0025225aafce  ONLINE
    gptid/28249605-789c-11e3-9bd2-0025225aafce  ONLINE
[root@freenas] ~#

Shouldn't I be able to import the volume with a missing disk, in the same way as a raidz1 can survive a one-disk-failure?

Will I have to use my backup or is there a way to get this volume up again?

 

[Bidule0hm]

The 300 kB lost is on one of the 3 online disks or on the unavailable one?

 

[jallis]

The 300 kB lost is on one of the 3 online disks or on the unavailable one?

There was 300k lost in the cloning of one of the disks that is now online, however I have not looked into where on the disk the errors were located. (If it was on the 2 GB swap partition it should not make any difference at all)

The disk that is currently UNAVAIL was in such a bad state that I was just planning on replacing it with an empty disk and rebuild pool based on the three online disks. However, it seems that I am unable to replace the UNAVAIL disk before I import the volume, and import fails...

 

[rs225]

Have you tried zpool import -f storage
?

Was the UNAVAIL disk failed for a while, and then the 300k disk just failed?

Odds are low that the missing 300k is enough to finish off the pool, especially because metadata is stored twice. However, that doesn't mean it is impossible depending on what exactly has happened.

 

[Bidule0hm]

Then I'd say try the -f flag ;)

 

[Mirfster]

Hope the best for you, but personally I am doubtful the outcome will be good. At the very least, you can understand why RaidZ1 is not recommended...

BTW, what are your System Hardware Specs? I hope that you are at the very least using ECC?

 

[jallis]

Have you tried zpool import -f storage
?

Was the UNAVAIL disk failed for a while, and then the 300k disk just failed?

Odds are low that the missing 300k is enough to finish off the pool, especially because metadata is stored twice. However, that doesn't mean it is impossible depending on what exactly has happened.

I tried just "zpool import -f", but that was obviously wrong syntax.
"zpool import -f storage" gives me this:

Code:

[root@freenas] ~# zpool import -f storage
cannot import 'storage': I/O error
   Destroy and re-create the pool from
   a backup source.
[root@freenas] ~#

Maybe the 300k makes a difference after all..

I don't know how long the system was running in degraded state before the second disk failed.

I have been waiting for a chance to migrate from raidz1 to raidz2, and maybe this is the perfect opertunity :)

 

[jallis]

Hope the best for you, but personally I am doubtful the outcome will be good. At the very least, you can understand why RaidZ1 is not recommended...

BTW, what are your System Hardware Specs? I hope that you are at the very least using ECC?

I know the recommended HW for Freenas and this small personal system is not at that level. Just standard RAM "from here and there" and too little of it.

 

[rs225]

Was the UNAVAIL disk UNAVAIL when you discovered the problem, or was it DEGRADED?

If DEGRADED, it might be worth trying to image it, as it may actually be in sync with the pool.

The 300k is probably not the problem with that drive; it seems far more likely to me the problem is the drive is out of sync with the pool. That is how you end up with metadata that is so 'corrupt' the pool won't import.

 

[jallis]

Was the UNAVAIL disk UNAVAIL when you discovered the problem, or was it DEGRADED?

If DEGRADED, it might be worth trying to image it, as it may actually be in sync with the pool.

Not sure what the status was. The system froze and did not answer ping packets. When rebooting the system would not boot as long as the faulted disk was connected.
The UNAVAIL disk is in such a bad state that I'm not able to read a single sector from it, so imaging it is not the way forward.

The 300k is probably not the problem with that drive; it seems far more likely to me the problem is the drive is out of sync with the pool. That is how you end up with metadata that is so 'corrupt' the pool won't import.

This time around I'll make sure to have a working e-mail report working, and raidz2 instead of raidz1.

 

[SweetAndLow]

The system is not going to mount that pool. Start over.

Sent from my Nexus 5X using Tapatalk