3-drive ZFS tank in failed state, how to recover

[mooreaa]

Hi guys,

I had one of my tanks go down (3x segate 1TB stack). I hear the click of death on one of the drives so I'm writing that one off normally I should be able to recover and I have a spare 1.5TB that I was going to use to correct but I'm really stuck.

1) The system would not boot with that stack plugged in. I would get a Fatal trap 12: page fault while in kernel mode fault during boot at a g_io_deliver. Fault code: supervisor read data, page not present.

So, I booted the system w/o those drives plugged in, and was able to remove the tank (detach w/o erase).

2) When I attempted to import that volume from the GUI it would error out. So I tried from the command line and i get this:

Code:

root@freenas:/ # zpool import
   pool: Tank1
     id: 11188256837084795573
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
config:

    Tank1                                           UNAVAIL  insufficient replicas
      raidz1-0                                      FAULTED  corrupted data
        2440255289171413541                         FAULTED  corrupted data
        3317451196795012811                         UNAVAIL  cannot open
        gptid/b9da1cfb-a5c3-11e7-89e0-605040302010  ONLINE

From the GUI i see two of the three drives (ada2 and ada6) listed there and am not seeing any issues there, however, I do see an alarm:
CRITICAL: Dec. 16, 2018, 1:06 a.m. - Device: /dev/ada6, 4 Currently unreadable (pending) sectors
CRITICAL: Dec. 16, 2018, 1:06 a.m. - Device: /dev/ada6, 4 Offline uncorrectable sectors


Am i totally hosed? I am not sure why the second drive says faulted/corrupted data. As far as I know it was only a single drive failure and the likely hood of two failures at the same time seem low but maybe this is what happened?

Appreciate any support on what I might be able to do. Could I manually try to use another tool to replicate the data on ada6? What options do I have?

 

[danb35]

As far as I know it was only a single drive failure

It appears that one disk has completely failed, and the second has suffered data corruption to the point that it is no longer a valid memver of the pool. If you can bring the missing disk back online, you might be able to save the data. You can also see if ZFS will force the import using zpool import -Fn Tank1.