Recovery of zpool with "missing" disk that has a damaged partition table

Status
Not open for further replies.

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
Earlier today I noticed the alert blinking the web GUI of my FreeNAS and there were two errors:

CRITICAL: Device: /dev/ada2, 7 Currently unreadable (pending) sectors
CRITICAL: Device: /dev/ada2, 7 Offline uncorrectable sectors

I followed the steps in https://dekoder.wordpress.com/2014/10/08/fixing-freenas-currently-unreadable-pending-sectors-error/ but I was still not able to reimport the pool because that disk is showing as unavailable to the zpool even though I can access it with low-level tools. New smartctl tests show no errors.

I think it's not available because of damage to the partition table on that particular disk.

[root@twilightsparkle] /mnt# fdisk /dev/ada2
******* Working on device /dev/ada2 *******
parameters extracted from in-core disklabel are:
cylinders=620181 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=620181 heads=16 sectors/track=63 (1008 blks/cyl)

fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 625142385 (305245 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 660/ head 15/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

Is there enough info somewhere to recreate that partition table on that disk to see if that will restore my zpool?

[root@twilightsparkle] /mnt# gpart list
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7847246
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
Mediasize: 524288 (512k)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 17408
Mode: r0w0e0
rawuuid: d8c9e9b4-edda-11e4-891e-6805ca2736ae
rawtype: 21686148-6449-6e6f-744e-656564454649
label: 1
length: 524288
offset: 17408
type: bios-boot
index: 1
end: 1057
start: 34
2. Name: da1p2
Mediasize: 4017242112 (3.8G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 544768
Mode: r1w1e1
rawuuid: d8d22a6e-edda-11e4-891e-6805ca2736ae
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: 1
length: 4017242112
offset: 544768
type: freebsd-zfs
index: 2
end: 7847239
start: 1064
Consumers:
1. Name: da1
Mediasize: 4017807360 (3.8G)
Sectorsize: 512
Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: 2910e49f-8ba7-11e3-ab72-ac220b50e07c
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: 1
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada1p2
Mediasize: 3998639460352 (3.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: 292e8055-8ba7-11e3-ab72-ac220b50e07c
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: 1
length: 3998639460352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037127
start: 4194432
Consumers:
1. Name: ada1
Mediasize: 4000787030016 (3.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: 28744380-8ba7-11e3-ab72-ac220b50e07c
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: 1
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada0p2
Mediasize: 3998639460352 (3.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: 289378a4-8ba7-11e3-ab72-ac220b50e07c
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: 1
length: 3998639460352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037127
start: 4194432
Consumers:
1. Name: ada0
Mediasize: 4000787030016 (3.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0


[root@twilightsparkle] /mnt# zpool import -d /dev
pool: Archives
id: 3808770709780828448
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-6X
config:

Archives UNAVAIL missing device
gptid/289378a4-8ba7-11e3-ab72-ac220b50e07c ONLINE
gptid/292e8055-8ba7-11e3-ab72-ac220b50e07c ONLINE
ada3 ONLINE
ada4 ONLINE

Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.

[root@twilightsparkle] /boot/zfs# zdb -C -U ./zpool.cache
Archives:
version: 5000
name: 'Archives'
state: 0
txg: 14319549
pool_guid: 3808770709780828448
hostid: 1133646529
hostname: ''
vdev_children: 5
vdev_tree:
type: 'root'
id: 0
guid: 3808770709780828448
children[0]:
type: 'disk'
id: 0
guid: 14354937141893476491
path: '/dev/gptid/27fe8d78-8ba7-11e3-ab72-ac220b50e07c'
whole_disk: 1
metaslab_array: 39
metaslab_shift: 31
ashift: 12
asize: 317920641024
is_log: 0
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 3741186956870640445
path: '/dev/gptid/289378a4-8ba7-11e3-ab72-ac220b50e07c'
whole_disk: 1
metaslab_array: 37
metaslab_shift: 35
ashift: 12
asize: 3998634737664
is_log: 0
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 16287532080604082671
path: '/dev/gptid/292e8055-8ba7-11e3-ab72-ac220b50e07c'
whole_disk: 1
metaslab_array: 35
metaslab_shift: 35
ashift: 12
asize: 3998634737664
is_log: 0
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 9244735741447355287
path: '/dev/ada3'
whole_disk: 1
metaslab_array: 311
metaslab_shift: 35
ashift: 12
asize: 4000782221312
is_log: 0
create_txg: 1044801
children[4]:
type: 'disk'
id: 4
guid: 18282693746607454160
path: '/dev/ada4'
whole_disk: 1
metaslab_array: 310
metaslab_shift: 35
ashift: 12
asize: 4000782221312
is_log: 0
create_txg: 1044801
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data

[root@twilightsparkle] /boot/zfs# zpool import -d /dev
pool: Archives
id: 3808770709780828448
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-6X
config:

Archives UNAVAIL missing device
gptid/289378a4-8ba7-11e3-ab72-ac220b50e07c ONLINE
gptid/292e8055-8ba7-11e3-ab72-ac220b50e07c ONLINE
ada3 ONLINE
ada4 ONLINE

Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
please follow the forum rules for posting or you may not be offered much help

and please use code tags for such a lot of information
 
Last edited:

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
please follow the forum rules for posting or you may not be offered much help
Let me see:
- I've been googling for a few hours, including reading through this forum
- I've given my post a descriptive subject
- I'm posting in Help & Support
- I gave a clear description of my problem and asked a specific question
- I posted output from some diagnostics

What did I miss?
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
hardware and how it is configured - freenas version
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
is it a stripe - mirror - raidz1 - raidz2 - raidz3 ???????

what hardware are you using?
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
post the results of zpool status and camcontrol devlist
 

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
is it a stripe - mirror - raidz1 - raidz2 - raidz3 ???????

what hardware are you using?

As I said, the 5 disk are in a single pool. The output from zdb about the pool doesn't provide enough information?

It's an AMD motherboard from Asus, though I'm not sure how the mobo, CPU & RAM specs will help somebody tell me whether the info to let me rebuild a corrupted partition table exists somewhere in the system settings?
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
as you know better - help yourself - i was only trying to help you - and you didn't even provide the minimum information required
 

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
post the results of zpool status and camcontrol devlist

As I mentioned, that one disk is unavailable to the zpool (see the zpool import output above), so status shows nothing.

Code:
[root@twilightsparkle] ~# zpool status
  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Apr 21 03:45:55 2016
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da1p2     ONLINE       0     0     0

errors: No known data errors
[root@twilightsparkle] ~#  camcontrol devlist
<ST4000DM000-1F2168 CC52>          at scbus4 target 0 lun 0 (pass0,ada0)
<ST4000DM000-1F2168 CC52>          at scbus5 target 0 lun 0 (pass1,ada1)
<Optiarc DVD RW AD-7240S 1.03>     at scbus6 target 0 lun 0 (pass2,cd0)
<ST320LT012-9WS14C 0002SDM1>       at scbus7 target 0 lun 0 (pass3,ada2)
<ST4000DM000-1F2168 CC52>          at scbus8 target 0 lun 0 (pass4,ada3)
<ST4000DM000-1F2168 CC52>          at scbus9 target 0 lun 0 (pass5,ada4)
<Generic USB SD Reader 1.00>       at scbus11 target 0 lun 0 (pass6,da0)
<Generic USB CF Reader 1.01>       at scbus11 target 0 lun 1 (pass7,da1)
<Generic USB xD/SM Reader 1.02>    at scbus11 target 0 lun 2 (pass8,da2)
<Generic USB MS Reader 1.03>       at scbus11 target 0 lun 3 (pass9,da3)
<Generic Mini SD Reader 1.06>      at scbus11 target 0 lun 4 (pass10,da4)
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
smartctl –a /dev/drive (where ‘drive’ is the drive letter, typically ada0, or da0) for all your hdds

put the results in codes
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
i asked for the configuration of the pool because a properly configured pool shouldn't be lost with a single hdd loss.

i presume you don't know
 
Last edited:

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
My guess is you zero'd out your drive while following the "directions" on that blog Did you run something like this: "dd if=/dev/zero of=/dev/ada2 bs=892134344 count=1 seek= conv=noerror,sync"

Did you export the pool, because it's not showing up in zpool status.

At this point, I'd recover from backup and recreate that pool. You have clearly manually added/replaced drives, and given this recent issue, I fear things will get worse in the future.
 

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
My guess is you zero'd out your drive while following the "directions" on that blog Did you run something like this: "dd if=/dev/zero of=/dev/ada2 bs=892134344 count=1 seek= conv=noerror,sync"

No, I caught that error since bs is block size and realized that it should have been bs=512 seek=892134344, so I subbed in the output of the 'smartctl -a' after the 'smartctl -t long'.

Many moons ago in a previous millennium, I fixed similar errors on OpenBSD on much smaller drives by either by copying a backup partition table from another sector to the beginning, or by knowing what the partitions been ad recreating the table from scratch and not newfsing anything.

I was hoping there would be something stored in /boot/zfs/zpool.cache or somewhere else in the FreeNAS settings that I could use here.

Did you export the pool, because it's not showing up in zpool status.

No, I never tried to export it because 'zpool status' couldn't find it.

At this point, I'd recover from backup and recreate that pool. You have clearly manually added/replaced drives, and given this recent issue, I fear things will get worse in the future.

Yes. Two weeks after creating the pool when I realized I wouldn't have enough space, I added two more disks to it via the FreeNAS GUI. First it was a 300G disk (ada2) and two identical 4T disks (ada0, ada1), and I added two more 4T (ada3, ada4) disks identical to the first 2. Those first two 4Ts show up in 'gpart list' and can be queried with 'geom' and and each show a 2G partition (ada0p1, ada1p1) and a 3.7T partition (ada0p2, ada1p2) under the slice, whereas the later two 4Ts aren't lists by gpart/geom. So I am assuming that ada2 also has partitions under the whole disk slice. I just don't know what sizes they were.

The backups, unfortunately, are several hundred audio CDs and movie DVDs that I am trying to avoid having to rerip. Nothing lost, I'm just hoping to better understand FreeBSD disk stuff better by seeing if I can recreate that the slices/partitions on ada2 but not having much luck.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
This thread has a bunch of nonsense posts describing the problem but the op has no clue what they are doing but won't do what is asked to get help.

If you look at the facts it looks like there is a stripped pool and some disks have gone missing. This pool has had lots of modifications done outside of freenas. Unless the OP can describe everything they have done this thread is going nowhere. Even at that I suspect all the data is gone.
 

Bytor

Dabbler
Joined
Apr 15, 2015
Messages
12
This thread has a bunch of nonsense posts describing the problem but the op has no clue what they are doing but won't do what is asked to get help.

If you look at the facts it looks like there is a stripped pool and some disks have gone missing. This pool has had lots of modifications done outside of freenas. Unless the OP can describe everything they have done this thread is going nowhere. Even at that I suspect all the data is gone.

Actually no. Until NFS stopped working which prompted me to look at the GUI and notice the alert image blinking and I found the zpool to be offline, I had done absolutely no modifications outside FreeNAS, and the link I gave in my first post described exactly everything that I did after that discovery.

As it turns out, the fix was exactly what I suspected in my first post when I asked:

Is there enough info somewhere to recreate that partition table on that disk to see if that will restore my zpool?

Given that the data was essential gone at this point and I could not make it any worse, I made an educated guess last night. I used fdisk to remake the slice and 'zdb Archives' found the pool but with GPT warnings, so I used dd to copy the secondary GPT at the end of the disk to a file and then to the front of the disk to overwrite the primary. (I could have used 'geom recover', I guess, but force of habit always makes me choose olde-style MBR schemes rather than GPT and I am admittedly not that familiar with the geom and associated tools.) I then went back to the FreeNAS GUI and used it to reimport my pool and there appears to be no data loss. I've been playing music all day as I worked with no issues.

I get what you're saying about giving enough information when I make a request for help, and I shall certainly endeavour to do so if the future if I need more help from this forum. However, permit me to point out that this is a two way street. If only somebody had seen that above question and said "Why yes, Bytor, have you forgotten FreeBSD usually makes a whole disk slice and there's a backup copy of the GPT at the end of the disk. Perhaps you could use that to restore the partitioning and see if zpool can see your pool again?" While FreeNAS version might have helped somebody target specific commands, hardware specifics about my NAS box were not really important to the question at hand.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
Actually no. Until NFS stopped working which prompted me to look at the GUI and notice the alert image blinking
Do you not have your e-mail properly configured? FreeNas would have sent an e-mail if it is. Also, do you have SMART Tests (Short & Long) as well as Scrubs scheduled? These all will very helpful. ;)
 
Status
Not open for further replies.
Top