raidz2 degraded. Unable to restore pool.

[SuXarik]

Hello all.
I would appreciate any help with my problem. I've got a machine with FreeNAS 9.1, having raidz2 on 6 disks (300GB each).
Some time ago someone using this server saw error concerning one of the disks in a pool, so he decided to unplug the disk and check it consistency, smart and so on on different machine. Of course no zpool unplag, replace has been done.
After this procedure (the drive was actually ok) the drive was put back in and server restarted, after that the pool can'be imported and no manipulations can be done.

ZDB:

Code:

storage1:
    version: 5000
    name: 'storage1'
    state: 0
    txg: 13368806
    pool_guid: 10371289679507810473
    hostid: 3838000927
    hostname: 'some-host'
    vdev_children: 2
    vdev_tree:
        type: 'root'
        id: 0
        guid: 10371289679507810473
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 9288871519253200646
            nparity: 2
            metaslab_array: 34
            metaslab_shift: 34
            ashift: 12
            asize: 1907523846144
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 11239903135
                path: '/dev/gptid/eb436305-5b5d-11e3-94b6-6805ca1d9039'
                phys_path: '/dev/gptid/eb436305-5b5d-11e3-94b6-6805ca1d9039'
                whole_disk: 1
                DTL: 188
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 6216757391228578512
                path: '/dev/gptid/ebac58a3-5b5d-11e3-94b6-6805ca1d9039'
                phys_path: '/dev/gptid/ebac58a3-5b5d-11e3-94b6-6805ca1d9039'
                whole_disk: 1
                DTL: 187
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 14369027937484422138
                path: '/dev/gptid/ec0c749c-5b5d-11e3-94b6-6805ca1d9039'
                guid: 1790071512574027942
                path: '/dev/gptid/ec6e0371-5b5d-11e3-94b6-6805ca1d9039'
                phys_path: '/dev/gptid/ec6e0371-5b5d-11e3-94b6-6805ca1d9039'
                whole_disk: 1
                DTL: 185
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 15937867517427237865
                path: '/dev/gptid/ecd5152e-5b5d-11e3-94b6-6805ca1d9039'
                phys_path: '/dev/gptid/ecd5152e-5b5d-11e3-94b6-6805ca1d9039'
                whole_disk: 1
                DTL: 182
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 132539782926748938
                path: '/dev/gptid/ed398ed9-5b5d-11e3-94b6-6805ca1d9039'
                phys_path: '/dev/gptid/ed398ed9-5b5d-11e3-94b6-6805ca1d9039'
                whole_disk: 1
                DTL: 181
                create_txg: 4
                offline: 1
        children[1]:
            type: 'disk'
            id: 1
            guid: 9851107490459590955
            path: '/dev/gptid/4f064914-bc9f-11e5-b703-6805ca1d9039'
            phys_path: '/dev/gptid/4f064914-bc9f-11e5-b703-6805ca1d9039'
            whole_disk: 1
            metaslab_array: 159
            metaslab_shift: 31
            ashift: 12
            asize: 317920641024
            is_log: 0
            create_txg: 13368804
    features_for_read:

camcontrol devlist

Code:

<ST3320418AS CC38>                 at scbus0 target 0 lun 0 (ada0,pass0)
<ST3320418AS CC38>                 at scbus1 target 0 lun 0 (ada1,pass1)
<ST3320418AS CC38>                 at scbus2 target 0 lun 0 (ada2,pass2)
<ST3320418AS CC38>                 at scbus3 target 0 lun 0 (ada3,pass3)
<ST3320418AS CC38>                 at scbus4 target 0 lun 0 (pass4,ada4)
<ST3320418AS CC38>                 at scbus5 target 0 lun 0 (pass5,ada5)
<JetFlash Transcend 8GB 1100>      at scbus7 target 0 lun 0 (pass6,da0)

zpool import -f

Code:

   pool: storage1
     id: 10371289679507810473
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

        storage1                                        UNAVAIL  missing device
          raidz2-0                                      DEGRADED
            gptid/eb436305-5b5d-11e3-94b6-6805ca1d9039  ONLINE
            gptid/ebac58a3-5b5d-11e3-94b6-6805ca1d9039  ONLINE
            gptid/ec0c749c-5b5d-11e3-94b6-6805ca1d9039  ONLINE
            gptid/ec6e0371-5b5d-11e3-94b6-6805ca1d9039  ONLINE
            gptid/ecd5152e-5b5d-11e3-94b6-6805ca1d9039  ONLINE
            132539782926748938                          OFFLINE

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.

glabel status

Code:

                                      Name  Status  Components
gptid/eb436305-5b5d-11e3-94b6-6805ca1d9039     N/A  ada0p2
gptid/ebac58a3-5b5d-11e3-94b6-6805ca1d9039     N/A  ada1p2
gptid/ec0c749c-5b5d-11e3-94b6-6805ca1d9039     N/A  ada2p2
gptid/ec6e0371-5b5d-11e3-94b6-6805ca1d9039     N/A  ada3p2
                             ufs/FreeNASs3     N/A  da0s3
                             ufs/FreeNASs4     N/A  da0s4
                            ufs/FreeNASs1a     N/A  da0s1a
gptid/b3d504dd-bc98-11e5-9da7-6805ca1d9039     N/A  ada4p1
gptid/b3dec865-bc98-11e5-9da7-6805ca1d9039     N/A  ada4p2
gptid/ecc8d4d1-5b5d-11e3-94b6-6805ca1d9039     N/A  ada5p1
gptid/ecd5152e-5b5d-11e3-94b6-6805ca1d9039     N/A  ada5p2

zpool status
no pools available

gpart list

Code:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: ebffc347-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: ec0c749c-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: eb9e6042-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: ebac58a3-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: eb385332-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: eb436305-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: ec62d171-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: ec6e0371-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: ecc8d4d1-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada4p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: ecd5152e-5b5d-11e3-94b6-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada4
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142414
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: b3d504dd-bc98-11e5-9da7-6805ca1d9039
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada5p2
   Mediasize: 317925367296 (296G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: b3dec865-bc98-11e5-9da7-6805ca1d9039
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 317925367296
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 625142414
   start: 4194432
Consumers:
1. Name: ada5
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15826943
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1
   attrib: active
   rawtype: 165
   length: 988291584
   offset: 32256
   type: freebsd
   index: 1
   end: 1930319
   start: 63
2. Name: da0s2
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988356096
   Mode: r0w0e0
   rawtype: 165
   length: 988291584
   offset: 988356096
   type: freebsd
   index: 2
   end: 3860639
   start: 1930383
3. Name: da0s3
   Mediasize: 1548288 (1.5M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1976647680
   Mode: r0w0e0
   rawtype: 165
   length: 1548288
   offset: 1976647680
   type: freebsd
   index: 3
   end: 3863663
   start: 3860640
4. Name: da0s4
   Mediasize: 21159936 (20M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1978195968
   Mode: r1w1e2
   rawtype: 165
   length: 21159936
   offset: 1978195968
   type: freebsd
   index: 4
   end: 3904991
   start: 3863664
Consumers:
1. Name: da0
   Mediasize: 8103395328 (7.6G)
   Sectorsize: 512
   Mode: r2w1e4

Geom name: da0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
   Mediasize: 988283392 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 40448
   Mode: r1w0e1
   rawtype: 0
   length: 988283392
   offset: 8192
   type: !0
   index: 1
   end: 1930256
   start: 16
Consumers:
1. Name: da0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1

The way I see it, the ada5 disk is still inside the server, but in pool configuration it has missing gptid. No drive has been wiped, so I hope I can restore the data.
Thanks in advance!

 

[DrKK]

What about this: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbvb.html

 

[SuXarik]

What about this: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbvb.html

no such pool

 

[DrKK]

@cyberjock

What say you, sir?

 

[Mirfster]

Confused... So the pool was originally on the FreeNas 9.1 system and all that occurred is a single drive was manually taken out and then put back in (after being tested on another machine)?

RaidZ2 can withstand two drive failures, so I am not seeing how this would be the case, even though the drive was improperly removed. That being said, why then would the same pool that already existed on the same system need to be "imported"?

Maybe I am missing something here... (which is more than likely the case...)

 

[DrKK]

Confused... So the pool was originally on the FreeNas 9.1 system and all that occurred is a single drive was manually taken out and then put back in (after being tested on another machine)?

RaidZ2 can withstand two drive failures, so I am not seeing how this would be the case, even though the drive was improperly removed. That being said, why then would the same pool that already existed on the same system need to be "imported"?

Maybe I am missing something here... (which is more than likely the case...)

Indeed, hence the punt to Cyberjock.

 

[SuXarik]

Confused... So the pool was originally on the FreeNas 9.1 system and all that occurred is a single drive was manually taken out and then put back in (after being tested on another machine)?

RaidZ2 can withstand two drive failures, so I am not seeing how this would be the case, even though the drive was improperly removed. That being said, why then would the same pool that already existed on the same system need to be "imported"?

Maybe I am missing something here... (which is more than likely the case...)

That's exactly what has been done. After reboot pool won't import automatically, but zpool.cache exists and looks real. What's also interesting, that if I take out this drive once more (exactly this, exactly at the same sata port) another disk became unavailable after I input zpool import, so it looks like it's working somehow. After plugging it back, it's OK, zpool import shows ONLINE for that disk.

 

[Mirfster]

So are you saying that the disk is now properly online and in the original pool ("storage1")? Or are you saying that the disk is online but in a new pool?

I am wondering if when the drive was originally pulled, it was added to a different pool on another system (when they were doing the testing) and if it retained configurations from that pool (mirrored?).

 

[SuXarik]

So are you saying that the disk is now properly online and in the original pool ("storage1")? Or are you saying that the disk is online but in a new pool?

I am wondering if when the drive was originally pulled, it was added to a different pool on another system (when they were doing the testing) and if it retained configurations from that pool (mirrored?).

All but one disk shows online as shown in the first post (but only 5 of them, then there are 6 plugged). Yes it was to another system, but not unix like, no pools there.

 

[Mirfster]

So is "storage1" online and simply showing a drive offline/missing? If so, could you simply use the "replace" or "offline" to re-add the same drive?

Can you post a screenshot of the Volume Status?

 

[SuXarik]

So is "storage1" online and simply showing a drive offline/missing? If so, could you simply use the "replace" or "offline" to re-add the same drive?

Can you post a screenshot of the Volume Status?

No it is not. In GUI it shows no volume available. Whe I try to import I says one device unavailable can't restore. I can't do anything with zpool command. Only zpool import shows I have storage1 but with one disk missing. All in first post. Thank you.

 

[jde]

Just a thought: the output from ZBD appears to show that there are two vdevs in the system (line 9). It appears to show that the two vdevs are a raidz2 and a single disk (lines 15-16 & 74-75). I'm unsure if that means the raidz2 vdev is striped with the single disk vdev. If so, I don't see the single disk (gptid/4f064914-bc9f-11e5-b703-6805ca1d9039) still present in system. Loss of a single disk vdev from a striped array would kill the pool. Might the drive which was removed and subsequently reintroduced to the system have been inadvertently reintroduced as a single disk vdev?

OP, was there any destructive testing (i.e. badblocks) performed on the disk when it was removed?

 

[jde]

Upon further review, I'm leaning more strongly that the pool was striped. The pool ID from ZBD line 6, which show two vdevs in the pool matches the pool ID from zpool import -f line 3. My guess is that if you can't find, or have overwritten gptid/4f064914-bc9f-11e5-b703-6805ca1d9039 your pool is toast.

 

[tvsjr]

Looking at the zdb output, this pool consisted of two vdevs (vdev_children = 2). One a raidz2, the other a single-disk vdev. Perhaps someone tried to extend the pool in the past by adding a 6th drive, without understanding the implications of doing so?

If the single-disk vdev can't be restored, the pool is toast...

 

[DrKK]

Looking at the zdb output, this pool consisted of two vdevs (vdev_children = 2). One a raidz2, the other a single-disk vdev. Perhaps someone tried to extend the pool in the past by adding a 6th drive, without understanding the implications of doing so?

If the single-disk vdev can't be restored, the pool is toast...

agreed.

If you had a single-disk vdev striped into the pool (probably #1 on the list things to never ever ever ever ever ever ever do with ZFS), then unless you can find that disk, and every bit on it is in pristine, virginal, condition, your data pool will be toasted.

 

[SuXarik]

agreed.

If you had a single-disk vdev striped into the pool (probably #1 on the list things to never ever ever ever ever ever ever do with ZFS), then unless you can find that disk, and every bit on it is in pristine, virginal, condition, your data pool will be toasted.

Ok. Not sure this had been done. I'll try to interrogate the one who did it. If I do find that mysterious disk what are my steps to recover data?

 

[jde]

If you can find the disk and it hasn't been adulteraded, I think you should be able to just plug it into your system and import the pool. Once the pool is imported, you should be able to resilver the raidz2 vdev.

 

[jde]

I'm pretty sure your pool was setup wrong from the start. I don't think it's a 6 disk raid2 vdev striped with another single disk vdev. I think it's a 5 disk raidz2 vdev striped with a single disk vdev. Moreover, the single disk vdev consists of a ~300 GB disk, the same size as you say we're initially installed. If I had to wager money at this point, I would say the disk that was removed was probably the one that comprised the single disk vdev. I would further speculate that whatever testing was done on the disk when it was removed included destructive testing. In short, I bet that gptid/b3dec865-bc98-11e5-9da7-6805ca1d9039, which is currently in your system, was formerly gptid/4f064914-bc9f-11e5-b703-6805ca1d9039, which is the "missing" disk. I'm not optimistic in your chances for recovery.

 

[DrKK]

I concur. I would bet 10 million rubles that your pool is toast, comrade.

 

[SuXarik]

I concur. I would bet 10 million rubles that your pool is toast, comrade.

Вот дерьмо! Which translates to "shit". But , anyhow , I've learned new things, so thank You all.