ZRAID1 drive marked as UNAVAIL and can't import

[davidred]

Hi All,

I've been through all the forum posts I can find here and various blogs about this problem but unfortunately I'm still no closer to getting my pool imported.

This is a home NAS and all the hardware is consumer grade. I've setup 2 ZRAID1 (3x2TB drives) in two separate pools (vol0 & vol1). I've been running this setup for ~12 months and only yesterday did the problem happen without any prior warning. vol0 is unaffected and is accessible without issue, vol1 however is not so lucky.

One of the drives in vol1 is marked as UNAVAIL corrupted data;

Code:

[root@server] ~# zpool import
   pool: vol1
     id: 15892612696792570768
  state: ONLINE
status: One or more devices contains corrupted data.
action: The pool can be imported using its name or numeric identifier.
   see: http://illumos.org/msg/ZFS-8000-4J
config:

    vol1                                            ONLINE
      raidz1-0                                      ONLINE
        gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180  ONLINE
        542711347190070112                          UNAVAIL  corrupted data
        gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180  ONLINE

Attempting to import the pool gets this error;

Code:

[root@server] ~# zpool import vol1
cannot import 'vol1': I/O error
Destroy and re-create the pool from
a backup source.

I've run smartctl on the drive (/dev/ada1) in question and it looks fine.

Code:

 9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       13327
....
SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     13321         -
# 2  Short offline       Completed without error       00%     11994         -
# 3  Short offline       Completed without error       00%     11994         -

Here is more details on the drives;

Code:

[root@server] ~# camcontrol devlist
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus3 target 0 lun 0 (ada2,pass2)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus5 target 0 lun 0 (ada3,pass3)
<HL-DT-ST DVDRAM GSA-H62N CL01>    at scbus5 target 1 lun 0 (pass4,cd0)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus6 target 0 lun 0 (ada4,pass5)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus6 target 1 lun 0 (ada5,pass6)
<General USB Flash Disk 1.0>       at scbus7 target 0 lun 0 (pass7,da0)

Code:

[root@server] ~# gpart show
=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada3  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada4  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada5  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>      63  15656897  da0  MBR  (7.5G)
        63   1930257    1  freebsd  [active]  (942M)
   1930320        63       - free -  (31k)
   1930383   1930257    2  freebsd  (942M)
   3860640      3024    3  freebsd  (1.5M)
   3863664     41328    4  freebsd  (20M)
   3904992  11751968       - free -  (5.6G)

=>      0  1930257  da0s1  BSD  (942M)
        0       16         - free -  (8.0k)
       16  1930241      1  !0  (942M)

No output from zpool import -nfF

Code:

[root@server] ~# zpool import -nfF vol1
[root@server] ~# 

I'd have thought a failure of one drive should not have caused the entire pool to become unusable. I've used mdraid on Linux for years before this and have had drives fail without causing the volume to fail, and replacing them was easy. I can't even get the pool imported to remove that drive successfully.

I'd appreciate any advice or further areas to look into, much thanks in advance!

Cheers

 

[DrKK]

Let's have a closer look at things.

Give us the ***FULL*** smartctl -x /dev/ada1. The whole thing. Don't touch anything, except deleting your serial number if you like.

Give us the full hardware specs. Motherboard, RAM, CPU.

Give us the full output of zpool status.

We'll get this solved, but we need more information.

 

[DrKK]

A couple of comments: A couple of "short" SMART tests, like you have run, are worse than useless in checking the health of the drives in a case like this.

Also, what is your scrubbing schedule? etc.

 

[davidred]

Thanks for the quick reply, here is the details you're after;

Code:

smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZAJ640016
LU WWN Device Id: 5 0014ee 25ced0f60
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Tue Aug  5 14:29:59 2014 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40260) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 388) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   175   166   021    -    6241
  4 Start_Stop_Count        -O--CK   100   100   000    -    86
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   082   082   000    -    13329
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    85
192 Power-Off_Retract_Count -O--CK   200   200   000    -    33
193 Load_Cycle_Count        -O--CK   157   157   000    -    129693
194 Temperature_Celsius     -O---K   120   107   000    -    30
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   001   000    -    1378477
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

ATA_READ_LOG_EXT (addr=0x00:0x00, page=0, n=1) failed: 48-bit ATA commands not implemented for legacy controllers
Read GP Log Directory failed

SMART Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00           SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x09           SL  R/W      1  Selective self-test log
0x80-0x9f      SL  R/W     16  Host vendor specific log
0xa0-0xa7      SL  VS      16  Device vendor specific log
0xa8-0xb7      SL  VS       1  Device vendor specific log
0xbd           SL  VS       1  Device vendor specific log
0xc0           SL  VS       1  Device vendor specific log
0xe0           SL  R/W      1  SCT Command/Status
0xe1           SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log (GP Log 0x03) not supported

SMART Error Log Version: 1
No Errors Logged

SMART Extended Self-test Log (GP Log 0x07) not supported

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     13321         -
# 2  Short offline       Completed without error       00%     11994         -
# 3  Short offline       Completed without error       00%     11994         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     30/31 Celsius
Lifetime    Min/Max Temperature:      0/42 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (120)

Index    Estimated Time   Temperature Celsius
121    2014-08-05 06:33    30  ***********
...    ..( 17 skipped).    ..  ***********
139    2014-08-05 06:51    30  ***********
140    2014-08-05 06:52    31  ************
141    2014-08-05 06:53    30  ***********
...    ..(100 skipped).    ..  ***********
242    2014-08-05 08:34    30  ***********
243    2014-08-05 08:35    22  ***
...    ..( 12 skipped).    ..  ***
256    2014-08-05 08:48    22  ***
257    2014-08-05 08:49     ?  -
258    2014-08-05 08:50    23  ****
259    2014-08-05 08:51    22  ***
...    ..(  2 skipped).    ..  ***
262    2014-08-05 08:54    22  ***
263    2014-08-05 08:55    21  **
264    2014-08-05 08:56    21  **
265    2014-08-05 08:57    20  *
...    ..(  4 skipped).    ..  *
270    2014-08-05 09:02    20  *
271    2014-08-05 09:03    19  -
...    ..(  4 skipped).    ..  -
276    2014-08-05 09:08    19  -
277    2014-08-05 09:09    18  -
278    2014-08-05 09:10     ?  -
279    2014-08-05 09:11    18  -
280    2014-08-05 09:12    17  -
...    ..(  4 skipped).    ..  -
285    2014-08-05 09:17    17  -
286    2014-08-05 09:18    18  -
...    ..(  9 skipped).    ..  -
296    2014-08-05 09:28    18  -
297    2014-08-05 09:29    19  -
...    ..( 12 skipped).    ..  -
310    2014-08-05 09:42    19  -
311    2014-08-05 09:43    20  *
312    2014-08-05 09:44    19  -
313    2014-08-05 09:45    20  *
...    ..( 63 skipped).    ..  *
377    2014-08-05 10:49    20  *
378    2014-08-05 10:50    19  -
379    2014-08-05 10:51    19  -
380    2014-08-05 10:52    20  *
381    2014-08-05 10:53    20  *
382    2014-08-05 10:54    19  -
383    2014-08-05 10:55    20  *
...    ..(  2 skipped).    ..  *
386    2014-08-05 10:58    20  *
387    2014-08-05 10:59    19  -
388    2014-08-05 11:00    20  *
389    2014-08-05 11:01    19  -
390    2014-08-05 11:02    19  -
391    2014-08-05 11:03    20  *
392    2014-08-05 11:04    20  *
393    2014-08-05 11:05    19  -
...    ..( 16 skipped).    ..  -
410    2014-08-05 11:22    19  -
411    2014-08-05 11:23    20  *
412    2014-08-05 11:24    19  -
413    2014-08-05 11:25    19  -
414    2014-08-05 11:26    20  *
415    2014-08-05 11:27     ?  -
416    2014-08-05 11:28    18  -
...    ..(  7 skipped).    ..  -
424    2014-08-05 11:36    18  -
425    2014-08-05 11:37    19  -
...    ..(  2 skipped).    ..  -
428    2014-08-05 11:40    19  -
429    2014-08-05 11:41    20  *
...    ..(  3 skipped).    ..  *
433    2014-08-05 11:45    20  *
434    2014-08-05 11:46    21  **
435    2014-08-05 11:47    21  **
436    2014-08-05 11:48    21  **
437    2014-08-05 11:49    22  ***
...    ..(  7 skipped).    ..  ***
445    2014-08-05 11:57    22  ***
446    2014-08-05 11:58    23  ****
447    2014-08-05 11:59    23  ****
448    2014-08-05 12:00    23  ****
449    2014-08-05 12:01     ?  -
450    2014-08-05 12:02    23  ****
...    ..(  5 skipped).    ..  ****
456    2014-08-05 12:08    23  ****
457    2014-08-05 12:09    24  *****
...    ..(  6 skipped).    ..  *****
464    2014-08-05 12:16    24  *****
465    2014-08-05 12:17    25  ******
...    ..(  6 skipped).    ..  ******
472    2014-08-05 12:24    25  ******
473    2014-08-05 12:25    26  *******
474    2014-08-05 12:26    25  ******
...    ..( 12 skipped).    ..  ******
   9    2014-08-05 12:39    25  ******
  10    2014-08-05 12:40    26  *******
...    ..(  5 skipped).    ..  *******
  16    2014-08-05 12:46    26  *******
  17    2014-08-05 12:47     ?  -
  18    2014-08-05 12:48    27  ********
...    ..(  3 skipped).    ..  ********
  22    2014-08-05 12:52    27  ********
  23    2014-08-05 12:53    28  *********
  24    2014-08-05 12:54    27  ********
...    ..(  3 skipped).    ..  ********
  28    2014-08-05 12:58    27  ********
  29    2014-08-05 12:59     ?  -
  30    2014-08-05 13:00    28  *********
...    ..(  5 skipped).    ..  *********
  36    2014-08-05 13:06    28  *********
  37    2014-08-05 13:07    29  **********
...    ..( 27 skipped).    ..  **********
  65    2014-08-05 13:35    29  **********
  66    2014-08-05 13:36    30  ***********
...    ..(  8 skipped).    ..  ***********
  75    2014-08-05 13:45    30  ***********
  76    2014-08-05 13:46     ?  -
  77    2014-08-05 13:47    31  ************
  78    2014-08-05 13:48    30  ***********
...    ..( 12 skipped).    ..  ***********
  91    2014-08-05 14:01    30  ***********
  92    2014-08-05 14:02    31  ************
  93    2014-08-05 14:03    30  ***********
...    ..(  4 skipped).    ..  ***********
  98    2014-08-05 14:08    30  ***********
  99    2014-08-05 14:09     ?  -
100    2014-08-05 14:10    30  ***********
...    ..( 19 skipped).    ..  ***********
120    2014-08-05 14:30    30  ***********

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

ATA_READ_LOG_EXT (addr=0x11:0x00, page=0, n=1) failed: 48-bit ATA commands not implemented for legacy controllers
Read SATA Phy Event Counters failed

Hardware Specs:

Code:

RAM: 4Gb
Motherboard: Asus P5KPL/EPU
CPU: Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz (1864.84-MHz K8-class CPU)

Code:

[root@server] ~# zpool status
  pool: vol0
state: ONLINE
  scan: scrub repaired 0 in 8h39m with 0 errors on Sun Jul 13 08:39:47 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        vol0                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/933e08e3-6b04-11e3-b904-10bf48b6e789  ONLINE       0     0     0
            gptid/93ef0157-6b04-11e3-b904-10bf48b6e789  ONLINE       0     0     0
            gptid/94a33d5e-6b04-11e3-b904-10bf48b6e789  ONLINE       0     0     0

errors: No known data errors
[root@server] ~#

Scrub schedule is the default FreeNAS setting, which I believe is every Sunday.

 

[DrKK]

Sir, I think we have your problem.

Attribute 199, UDMA_CRC error count? 98% of the time that's a bad SATA cable to the drive. I'd say there's an 80% chance that everything will be fine the instant you replace the SATA cable.

 

[DrKK]

Just for the record, you also have a high "load cycle count". This is because you're using a "green" drive in a NAS. This drive will eventually wear out prematurely--you will other posts in the forum where we talk about the problems using a WD green in a NAS. One of the "green" features is a cycle retract after a short period of time, then, the NAS reloads it, time and time and time and time again.

 

[DrKK]

And also for the record, you need to use the GUI in the FreeNAS to set up a regimen of SMART tests. We recommend scheduling a couple "long" tests per month. You have not been doing those.

But other than that, everything about this drive suggests it is fine. Replace the cable, you'll be fine.

 

[davidred]

Thanks DrKK for all your help so far. I replaced the cable and also changed SATA ports the drive was on but the pool status is still the same. Note I've disconnected the other 3 drives that made up the pool vol0 which is why it's not appearing;

Code:

[root@server] ~# zpool status
no pools available
[root@server] ~# zpool import
   pool: vol1
     id: 15892612696792570768
  state: ONLINE
status: One or more devices contains corrupted data.
action: The pool can be imported using its name or numeric identifier.
   see: http://illumos.org/msg/ZFS-8000-4J
config:

        vol1                                            ONLINE
          raidz1-0                                      ONLINE
            gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180  ONLINE
            542711347190070112                          UNAVAIL  corrupted data
            gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180  ONLINE
[root@server] ~# zpool import vol1
cannot import 'vol1': I/O error
        Destroy and re-create the pool from
        a backup source.
[root@server] ~#

This event has made me realise it's time to get some better hardware, so I've already put an order in for new drives to replace the WD Green drives I already have :)

 

[danb35]

On the load cycle count, you can set the drives to not park the heads as often--Google for wdidle3, which is WD's utility to adjust that setting. That won't reduce the current count, of course, but will reduce the rate of increase.

Note that your other hardware isn't up to spec either--you're using a consumer-grade motherboard which doesn't support ECC, and half the minimum recommended RAM for ZFS. I doubt either of these caused your problem, but they aren't going to help.

I wonder what would happen if you disconnected the faulted drive, and then tried importing the pool.

 

[davidred]

I tried that earlier today and it didn't help unfortunately. It's late here (Australia) so will attempt a few more things tomorrow like live CDs (PCBSD in particular).

 

[cyberjock]

Honestly, I think your pool was eaten by your <8GB of RAM in conjunction with your bad SATA cable.

At this point I can only wish you luck with getting your pool back. :(

I've done pool recovery before, and this isn't looking promising at all.

 

[DrKK]

Yeah unfortunately, I thought of this also---when you get into a fail condition with insufficient hardware, you can be in trouble. :(

Sorry sir, I really thought it would come back no problem.

 

[davidred]

Unfortunately no luck with the LiveCDs (PCBSD, Ubuntu 14 Desktop, mfsbsd, OmniOS & Wheezy rescue). Ubuntu wouldn't even recognise the ZFS pool as it was created with a newer version of ZFS, which is a shame as I've seen success by other people using it.

Without the 'bad' drive connected, all indications are that it should import but no luck;

Code:

[root@server] ~# zpool import
   pool: vol1
     id: 15892612696792570768
  state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://illumos.org/msg/ZFS-8000-2Q
config:

        vol1                                            DEGRADED
          raidz1-0                                      DEGRADED
            gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180  ONLINE
            542711347190070112                          UNAVAIL  cannot open
            gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180  ONLINE
[root@server] ~# zpool import -f vol1
cannot import 'vol1': I/O error
        Destroy and re-create the pool from
        a backup source.
[root@server] ~#

I'm curious if anyone thinks that importing with the -X command would help at all?

 

[cyberjock]

Generally if you're getting I/O errors you are talking data recovery specialist or at least someone that can go looking for where the bad transaction is and roll back to before it. I've recovered from the exact error you have, but that error can be indicative of so many problems. If you are looking to throw money at trying to get your files back send me a PM. It's not cheap, but I have about a 90% success rate. But to be honest you probably won't like my price and your chances of getting your data back are significantly diminished by the fact you had <8GB of RAM.

 

[davidred]

I've ordered some new hardware to replace the existing setup. I'll be moving the content on my good pool (vol0) across to the new hardware which will leave me with 3 identical HDDs to what is in vol1. I'll then do a disk dupe to image those drives and then will do some more potentially destructive things.

But until then I ran with some more informative commands and found a few interesting things. Hopefully the follow log spam isn't too much!

I ran a zdb -l on each of the device partitions;

Code:

[root@server] ~# zdb -l /dev/ada0p2
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3429033
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 4574289677792951915
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3429033
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 4574289677792951915
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3429033
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 4574289677792951915
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3429033
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 4574289677792951915
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
[root@server] ~#

Code:

[root@server] ~# zdb -l /dev/ada1p2
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 542711347190070112
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 542711347190070112
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 542711347190070112
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 542711347190070112
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
[root@server] ~#

Code:

[root@server] ~# zdb -l /dev/ada2p2
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 7895556280047347563
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 7895556280047347563
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 7895556280047347563
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 5000
    name: 'vol1'
    state: 0
    txg: 3456630
    pool_guid: 15892612696792570768
    hostid: 3855458985
    hostname: 'server.red32.local'
    top_guid: 13596152721278901337
    guid: 7895556280047347563
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 13596152721278901337
        nparity: 1
        metaslab_array: 34
        metaslab_shift: 35
        ashift: 12
        asize: 5994739924992
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4574289677792951915
            path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5bd6e536-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 233
            create_txg: 4
            removed: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 542711347190070112
            path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            phys_path: '/dev/gptid/1e453333-ff49-11e3-9916-00248ceadc5e'
            whole_disk: 1
            DTL: 230
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 7895556280047347563
            path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            phys_path: '/dev/gptid/5d92bafc-6d5b-11e3-93a7-000fea4f7180'
            whole_disk: 1
            DTL: 232
            create_txg: 4
    features_for_read:
[root@server] ~#

 

[davidred]

If you notice, the TXG on ada0p2 is 3429033 while the TXG on the other 2 partitions is 3456630. The corrupted partition reported by zpool import is ada1. So in summary;

Code:

ada0p2 = 3429033
ada1p2 = 3456630 (Corrupted)
ada2p2 = 3456630

I tried to do an import at the earlier TXG level with no luck;

Code:

[root@server] ~# zpool import -N -o readonly=on -f -R /pool -F -T 3429033 vol1
cannot import 'vol1': one or more devices is currently unavailable

I then tried some zdb commands which gave lots of output but still no success;

Code:

[root@server] ~# zdb -AAA -L -t 3429033 -bcdmu -e 15892612696792570768

Uberblock:
    magic = 0000000000bab10c
    version = 5000
    txg = 3429033
    guid_sum = 5607834575682738463
    timestamp = 1406970453 UTC = Sat Aug  2 19:07:33 2014


Metaslabs:
    vdev          0
    metaslabs   174   offset                spacemap          free     
    ---------------   -------------------   ---------------   -------------
    metaslab      0   offset            0   spacemap     37   free    1.77G
    metaslab      1   offset    800000000   spacemap     63   free    1.15G
    metaslab      2   offset   1000000000   spacemap     64   free    1.27G
    metaslab      3   offset   1800000000   spacemap     71   free    6.90G
    metaslab      4   offset   2000000000   spacemap     72   free    8.25G
    metaslab      5   offset   2800000000   spacemap     73   free    8.69G
    metaslab      6   offset   3000000000   spacemap     74   free    7.89G
    metaslab      7   offset   3800000000   spacemap     75   free    7.39G
    metaslab      8   offset   4000000000   spacemap     76   free    5.85G
    metaslab      9   offset   4800000000   spacemap     77   free    5.34G
    metaslab     10   offset   5000000000   spacemap     80   free    5.39G
    metaslab     11   offset   5800000000   spacemap     81   free    4.73G
    metaslab     12   offset   6000000000   spacemap     83   free    3.82G
    metaslab     13   offset   6800000000   spacemap     84   free    4.14G
    metaslab     14   offset   7000000000   spacemap     85   free    5.36G
    metaslab     15   offset   7800000000   spacemap     86   free    4.42G
    metaslab     16   offset   8000000000   spacemap     87   free    3.69G
    metaslab     17   offset   8800000000   spacemap     90   free    6.56G
    metaslab     18   offset   9000000000   spacemap     91   free    8.71G
    metaslab     19   offset   9800000000   spacemap     92   free    7.39G
    metaslab     20   offset   a000000000   spacemap     93   free    5.14G
    metaslab     21   offset   a800000000   spacemap     94   free    5.08G
    metaslab     22   offset   b000000000   spacemap     95   free    3.96G
    metaslab     23   offset   b800000000   spacemap     96   free    4.63G
    metaslab     24   offset   c000000000   spacemap     97   free    4.45G
    metaslab     25   offset   c800000000   spacemap     98   free    3.08G
    metaslab     26   offset   d000000000   spacemap     99   free    3.69G
    metaslab     27   offset   d800000000   spacemap    100   free    4.02G
    metaslab     28   offset   e000000000   spacemap    101   free    3.61G
    metaslab     29   offset   e800000000   spacemap    104   free    3.09G
    metaslab     30   offset   f000000000   spacemap    105   free    3.12G
    metaslab     31   offset   f800000000   spacemap    106   free    2.59G
    metaslab     32   offset  10000000000   spacemap    107   free    3.04G
    metaslab     33   offset  10800000000   spacemap     36   free    3.30G
    metaslab     34   offset  11000000000   spacemap    108   free    4.66G
    metaslab     35   offset  11800000000   spacemap    109   free    3.23G
    metaslab     36   offset  12000000000   spacemap    110   free    3.06G
    metaslab     37   offset  12800000000   spacemap    111   free    4.37G
    metaslab     38   offset  13000000000   spacemap    112   free    5.36G
    metaslab     39   offset  13800000000   spacemap    113   free    4.64G
    metaslab     40   offset  14000000000   spacemap    114   free    4.48G
    metaslab     41   offset  14800000000   spacemap    115   free    3.56G
    metaslab     42   offset  15000000000   spacemap    116   free    4.30G
    metaslab     43   offset  15800000000   spacemap    117   free    3.83G
    metaslab     44   offset  16000000000   spacemap    118   free    4.88G
    metaslab     45   offset  16800000000   spacemap     82   free    4.00G
    metaslab     46   offset  17000000000   spacemap    121   free    4.78G
    metaslab     47   offset  17800000000   spacemap    122   free    4.54G
    metaslab     48   offset  18000000000   spacemap    123   free     9.9G
    metaslab     49   offset  18800000000   spacemap    124   free    13.5G
    metaslab     50   offset  19000000000   spacemap     89   free    11.2G
    metaslab     51   offset  19800000000   spacemap    125   free    11.6G
    metaslab     52   offset  1a000000000   spacemap    126   free    11.5G
    metaslab     53   offset  1a800000000   spacemap    140   free    10.8G
    metaslab     54   offset  1b000000000   spacemap    141   free    14.7G
    metaslab     55   offset  1b800000000   spacemap    142   free    4.87G
    metaslab     56   offset  1c000000000   spacemap    143   free    4.25G
    metaslab     57   offset  1c800000000   spacemap    144   free    4.26G
    metaslab     58   offset  1d000000000   spacemap    145   free    4.78G
    metaslab     59   offset  1d800000000   spacemap    146   free    4.21G
    metaslab     60   offset  1e000000000   spacemap    147   free    4.34G
    metaslab     61   offset  1e800000000   spacemap    148   free    5.94G
    metaslab     62   offset  1f000000000   spacemap    103   free    1.43G
    metaslab     63   offset  1f800000000   spacemap    150   free    5.90G
    metaslab     64   offset  20000000000   spacemap    151   free    1.28G
    metaslab     65   offset  20800000000   spacemap    152   free    1.25G
    metaslab     66   offset  21000000000   spacemap     35   free    5.04G
    metaslab     67   offset  21800000000   spacemap     79   free    1.29G
    metaslab     68   offset  22000000000   spacemap    159   free    2.57G
    metaslab     69   offset  22800000000   spacemap    160   free    8.93G
    metaslab     70   offset  23000000000   spacemap    161   free    8.93G
    metaslab     71   offset  23800000000   spacemap    133   free    2.56G
    metaslab     72   offset  24000000000   spacemap    134   free    1.25G
    metaslab     73   offset  24800000000   spacemap    135   free    17.8G
    metaslab     74   offset  25000000000   spacemap    136   free    8.91G
    metaslab     75   offset  25800000000   spacemap    137   free    1.30G
    metaslab     76   offset  26000000000   spacemap    138   free    1.42G
    metaslab     77   offset  26800000000   spacemap    139   free    1.64G
    metaslab     78   offset  27000000000   spacemap    164   free    1.62G
    metaslab     79   offset  27800000000   spacemap    120   free    1.42G
    metaslab     80   offset  28000000000   spacemap    165   free    1.35G
    metaslab     81   offset  28800000000   spacemap    166   free    1.40G
    metaslab     82   offset  29000000000   spacemap    167   free    1.35G
    metaslab     83   offset  29800000000   spacemap     88   free    1.21G
    metaslab     84   offset  2a000000000   spacemap    168   free    1.20G
    metaslab     85   offset  2a800000000   spacemap    169   free    1.29G
    metaslab     86   offset  2b000000000   spacemap    170   free    7.38G
    metaslab     87   offset  2b800000000   spacemap    171   free    4.36G
    metaslab     88   offset  2c000000000   spacemap    172   free    1.49G
    metaslab     89   offset  2c800000000   spacemap    173   free    5.96G
    metaslab     90   offset  2d000000000   spacemap    174   free    20.0G
    metaslab     91   offset  2d800000000   spacemap    177   free    3.72G
    metaslab     92   offset  2e000000000   spacemap    178   free    17.6G
    metaslab     93   offset  2e800000000   spacemap    179   free    1.34G
    metaslab     94   offset  2f000000000   spacemap    180   free    1.65G
    metaslab     95   offset  2f800000000   spacemap    102   free    4.66G
    metaslab     96   offset  30000000000   spacemap    181   free    4.83G
    metaslab     97   offset  30800000000   spacemap    182   free    2.17G
    metaslab     98   offset  31000000000   spacemap    183   free    1.31G
    metaslab     99   offset  31800000000   spacemap     78   free     9.7G
    metaslab    100   offset  32000000000   spacemap    185   free    25.0G
    metaslab    101   offset  32800000000   spacemap    186   free    23.5G
    metaslab    102   offset  33000000000   spacemap    187   free    23.0G
    metaslab    103   offset  33800000000   spacemap    188   free    3.24G
    metaslab    104   offset  34000000000   spacemap    189   free    2.07G
    metaslab    105   offset  34800000000   spacemap    190   free    2.48G
    metaslab    106   offset  35000000000   spacemap    191   free    3.03G
    metaslab    107   offset  35800000000   spacemap    194   free    3.44G
    metaslab    108   offset  36000000000   spacemap    195   free    24.0G
    metaslab    109   offset  36800000000   spacemap    196   free    26.4G
    metaslab    110   offset  37000000000   spacemap    197   free    25.8G
    metaslab    111   offset  37800000000   spacemap    163   free    3.02G
    metaslab    112   offset  38000000000   spacemap    119   free    17.8G
    metaslab    113   offset  38800000000   spacemap    202   free    18.7G
    metaslab    114   offset  39000000000   spacemap    203   free    24.1G
    metaslab    115   offset  39800000000   spacemap    204   free    3.87G
    metaslab    116   offset  3a000000000   spacemap    198   free    7.84G
    metaslab    117   offset  3a800000000   spacemap    205   free    2.48G
    metaslab    118   offset  3b000000000   spacemap    206   free    25.1G
    metaslab    119   offset  3b800000000   spacemap    207   free    23.3G
    metaslab    120   offset  3c000000000   spacemap    208   free    17.8G
    metaslab    121   offset  3c800000000   spacemap    209   free    19.8G
    metaslab    122   offset  3d000000000   spacemap    210   free    25.8G
    metaslab    123   offset  3d800000000   spacemap    215   free    7.37G
    metaslab    124   offset  3e000000000   spacemap    176   free    11.6G
    metaslab    125   offset  3e800000000   spacemap    184   free    12.1G
    metaslab    126   offset  3f000000000   spacemap    216   free    3.88G
    metaslab    127   offset  3f800000000   spacemap    217   free    28.6G
    metaslab    128   offset  40000000000   spacemap    149   free    26.5G
    metaslab    129   offset  40800000000   spacemap    219   free    17.5G
    metaslab    130   offset  41000000000   spacemap    227   free    19.4G
    metaslab    131   offset  41800000000   spacemap    229   free    20.1G
    metaslab    132   offset  42000000000   spacemap    222   free    21.7G
    metaslab    133   offset  42800000000   spacemap    218   free    5.73G
    metaslab    134   offset  43000000000   spacemap    225   free    4.36G
    metaslab    135   offset  43800000000   spacemap    228   free    1.82G
    metaslab    136   offset  44000000000   spacemap    234   free    3.15G
    metaslab    137   offset  44800000000   spacemap    236   free    1.79G
    metaslab    138   offset  45000000000   spacemap    237   free    1.77G
    metaslab    139   offset  45800000000   spacemap    238   free    1.96G
    metaslab    140   offset  46000000000   spacemap    193   free    6.64G
    metaslab    141   offset  46800000000   spacemap    239   free    14.3G
    metaslab    142   offset  47000000000   spacemap    226   free    28.6G
    metaslab    143   offset  47800000000   spacemap      0   free      32G
    metaslab    144   offset  48000000000   spacemap    162   free    32.0G
    metaslab    145   offset  48800000000   spacemap      0   free      32G
    metaslab    146   offset  49000000000   spacemap      0   free      32G
    metaslab    147   offset  49800000000   spacemap      0   free      32G
    metaslab    148   offset  4a000000000   spacemap      0   free      32G
    metaslab    149   offset  4a800000000   spacemap      0   free      32G
    metaslab    150   offset  4b000000000   spacemap      0   free      32G
    metaslab    151   offset  4b800000000   spacemap      0   free      32G
    metaslab    152   offset  4c000000000   spacemap      0   free      32G
    metaslab    153   offset  4c800000000   spacemap      0   free      32G
    metaslab    154   offset  4d000000000   spacemap      0   free      32G
    metaslab    155   offset  4d800000000   spacemap      0   free      32G
    metaslab    156   offset  4e000000000   spacemap    211   free    32.0G
    metaslab    157   offset  4e800000000   spacemap    175   free    32.0G
    metaslab    158   offset  4f000000000   spacemap      0   free      32G
    metaslab    159   offset  4f800000000   spacemap      0   free      32G
    metaslab    160   offset  50000000000   spacemap      0   free      32G
    metaslab    161   offset  50800000000   spacemap      0   free      32G
    metaslab    162   offset  51000000000   spacemap      0   free      32G
    metaslab    163   offset  51800000000   spacemap      0   free      32G
    metaslab    164   offset  52000000000   spacemap      0   free      32G
    metaslab    165   offset  52800000000   spacemap      0   free      32G
    metaslab    166   offset  53000000000   spacemap      0   free      32G
    metaslab    167   offset  53800000000   spacemap      0   free      32G
    metaslab    168   offset  54000000000   spacemap      0   free      32G
    metaslab    169   offset  54800000000   spacemap    231   free    32.0G
    metaslab    170   offset  55000000000   spacemap      0   free      32G
    metaslab    171   offset  55800000000   spacemap      0   free      32G
    metaslab    172   offset  56000000000   spacemap      0   free      32G
    metaslab    173   offset  56800000000   spacemap    192   free    32.0G

Dataset mos [META], ID 0, cr_txg 4, 56.7M, 239 objects
Dataset vol1/Timemachine/david-macbook-pro [ZPL], ID 54, cr_txg 324, 167G, 42852 objects
Dataset vol1/Timemachine/belinda-macbook-air [ZPL], ID 68, cr_txg 966, 89.5G, 22937 objects
Dataset vol1/Timemachine [ZPL], ID 48, cr_txg 315, 213K, 9 objects
Dataset vol1/Media [ZPL], ID 41, cr_txg 28, 1.77T, 9536 objects
Dataset vol1/Downloads [ZPL], ID 156, cr_txg 29960, 45.1G, 37969 objects
Dataset vol1/Music [ZPL], ID 60, cr_txg 423, 29.5G, 18481 objects
Dataset vol1/Pictures [ZPL], ID 130, cr_txg 21674, 79.2G, 63310 objects
Dataset vol1 [ZPL], ID 21, cr_txg 1, 98.2G, 38767 objects

Traversing all blocks to verify metadata checksums ...

44.6M completed (   0MB/s) estimated time remaining: 1629hr 39min 05sec        zdb_blkptr_cb: Got error 122 reading <0, 175, 0, 6>  -- skipping
48.2M completed (   0MB/s) estimated time remaining: 1650hr 52min 59sec        zdb_blkptr_cb: Got error 122 reading <0, 178, 0, 2a>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 178, 0, 31>  -- skipping
53.8M completed (   0MB/s) estimated time remaining: 1664hr 26min 04sec        zdb_blkptr_cb: Got error 122 reading <0, 188, 0, 1>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 188, 0, 4>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 188, 0, 7>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 188, 0, a>  -- skipping
54.4M completed (   0MB/s) estimated time remaining: 1661hr 42min 20sec        zdb_blkptr_cb: Got error 122 reading <0, 189, 0, e>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 189, 0, 15>  -- skipping
55.1M completed (   0MB/s) estimated time remaining: 1661hr 42min 19sec        zdb_blkptr_cb: Got error 122 reading <0, 190, 0, 1>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 190, 0, 4>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 190, 0, 7>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 190, 0, 9>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 190, 0, 10>  -- skipping
59.4M completed (   0MB/s) estimated time remaining: 1656hr 16min 22sec        zdb_blkptr_cb: Got error 122 reading <0, 197, 0, 19>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 197, 0, 1a>  -- skipping
62.4M completed (   0MB/s) estimated time remaining: 1672hr 40min 13sec        zdb_blkptr_cb: Got error 122 reading <0, 204, 0, d>  -- skipping
  63M completed (   0MB/s) estimated time remaining: 1672hr 40min 12sec        zdb_blkptr_cb: Got error 122 reading <0, 204, 0, 13>  -- skipping
65.5M completed (   0MB/s) estimated time remaining: 1669hr 54min 47sec        zdb_blkptr_cb: Got error 122 reading <0, 208, 1, 0>  -- skipping
77.2M completed (   0MB/s) estimated time remaining: 1697hr 52min 45sec        zdb_blkptr_cb: Got error 122 reading <0, 228, 0, 0>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 228, 0, 23>  -- skipping
78.4M completed (   0MB/s) estimated time remaining: 1689hr 23min 22sec        zdb_blkptr_cb: Got error 122 reading <0, 228, 0, 26>  -- skipping
zdb_blkptr_cb: Got error 122 reading <0, 228, 0, 2c>  -- skipping
83.7M completed (   0MB/s) estimated time remaining: 1675hr 25min 30sec       
Error counts:

    errno  count
      122  23
block traversal size 87859200 != alloc 3736741363712 (unreachable 3736653504512)

    bp count:            3550
    bp logical:      16754688      avg:   4719
    bp physical:     10549760      avg:   2971     compression:   1.59
    bp allocated:    87859200      avg:  24749     compression:   0.19
    bp deduped:             0    ref>1:      0   deduplication:   1.00
    SPA allocated: 3736741363712     used: 62.50%

[root@server] ~#

It's reporting back the Datasets I created so I'm still optimistically hopeful recovery is an option. At this point the commands I've seen and looked at all come with warnings about potential destruction, so I've pulled the pin until the new hardware arrives.

 

[cyberjock]

Umm.. you really shouldn't be doing this kind of thing if you want me to recover your data. Some command *will* write data to your pool and make recovery even harder. :P