zpool lost after (update install) reboot

[Tekkie]

It happened :( after almost 7 years of running FreeNas, I lost my ZRAID2 pool after a reboot.

System details:
Build FreeNAS-11.1-RELEASE
Platform AMD Phenom(tm) II X4 B15e Processor
Memory 16088MB
Drives 12 x 2TB (WD/Seagate/Toshiba)
- 1 ZRAID2 pool (6 drives)

[root@shrek ~]# zpool status -v
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:08:09 with 0 errors on Fri Dec 15 03:53:09 2017
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/e23168e7-a0fb-11e5-bcd0-001b21ad7873 ONLINE 0 0 0

errors: No known data errors
[root@shrek ~]#

However when I go into the UI and look in the storage tab I can see the pool name there, and I get the option to detach it however that's not really helping me in recovering the pool. When I do a 'view disks' I can also see all the drives being present and accounted for, so the question is where do I start the recovery process?

 

[Tekkie]

Clearly all the gpart data is there, now what is the safest way to reconstruct this baby?

[root@shrek ~]# gpart show
=> 34 3907029101 da0 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da1 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da2 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 da4 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 125045357 da5 GPT (60G)
34 94 - free - (47K)
128 125045256 1 freebsd-zfs (60G)
125045384 7 - free - (3.5K)

=> 34 60062433 da6 GPT (29G)
34 1024 1 bios-boot (512K)
1058 6 - free - (3.0K)
1064 60061400 2 freebsd-zfs (29G)
60062464 3 - free - (1.5K)

=> 34 3907029101 ada0 GPT (1.8T)
34 94 - free - (47K)
128 3907029000 1 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 ada2 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

[root@shrek ~]# zpool import
pool: storage1
id: 10718062685504435789
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://illumos.org/msg/ZFS-8000-4J
config:

storage1 DEGRADED
raidz2-0 ONLINE
gptid/0c55ca54-f193-11e2-9b74-001b21ad7873 ONLINE
gptid/d87d2b8b-b42b-11e6-b994-001b21ad7873 ONLINE
gptid/f49ae910-a562-11e0-93da-001b21ad7873 ONLINE
gptid/14548422-9c71-11e5-af38-001b21ad7873 ONLINE
gptid/f64efe95-a562-11e0-93da-001b21ad7873 ONLINE
gptid/f76d9183-a562-11e0-93da-001b21ad7873 ONLINE
logs
9646051538988606649 UNAVAIL corrupted data

I tried the answer suggested in this thread https://forums.freebsd.org/threads/56606/ but that didn't work, I am still getting the error "
cannot import 'storage1': one or more devices is currently unavailable".

Additional information `da5` is the unavailable LOG drive, but how do I re-attach it?

 

[Ericloewe]

zpool import -m - you will lose the last few seconds of transactions. What are you using as an SLOG?

 

[Tekkie]

Ok I am a step further this command

Code:

zpool import -F -d /dev/gptid/ -m storage1

did the trick. However what is still mind boggling is that

root@shrek:/ # zpool status

pool: freenas-boot

state: ONLINE

scan: scrub repaired 0 in 0 days 00:08:09 with 0 errors on Fri Dec 15 03:53:09 2017

config:


NAME STATE READ WRITE CKSUM

freenas-boot ONLINE 0 0 0

gptid/e23168e7-a0fb-11e5-bcd0-001b21ad7873 ONLINE 0 0 0


errors: No known data errors


pool: storage1

state: DEGRADED

status: One or more devices could not be used because the label is missing or

invalid. Sufficient replicas exist for the pool to continue

functioning in a degraded state.

action: Replace the device using 'zpool replace'.

see: http://illumos.org/msg/ZFS-8000-4J

scan: scrub repaired 0 in 0 days 10:04:52 with 0 errors on Fri Dec 15 12:05:03 2017

config:


NAME STATE READ WRITE CKSUM

storage1 DEGRADED 0 0 0

raidz2-0 ONLINE 0 0 0

gptid/0c55ca54-f193-11e2-9b74-001b21ad7873 ONLINE 0 0 0

gptid/d87d2b8b-b42b-11e6-b994-001b21ad7873 ONLINE 0 0 0

gptid/f49ae910-a562-11e0-93da-001b21ad7873 ONLINE 0 0 0

gptid/14548422-9c71-11e5-af38-001b21ad7873 ONLINE 0 0 0

gptid/f64efe95-a562-11e0-93da-001b21ad7873 ONLINE 0 0 0

gptid/f76d9183-a562-11e0-93da-001b21ad7873 ONLINE 0 0 0

logs

9646051538988606649 UNAVAIL 0 0 0 was /dev/gptid/7965b700-a103-11e5-ad6f-001b21ad7873


errors: No known data errors

shows the missing drive as

Code:

/dev/gptid/7965b700-a103-11e5-ad6f-001b21ad7873

however that label is very well present here

root@shrek:/ # glabel status

Name Status Components

gptid/f3d00bc0-a562-11e0-93da-001b21ad7873 N/A da0p2

gptid/f64efe95-a562-11e0-93da-001b21ad7873 N/A da1p2

gptid/f76d9183-a562-11e0-93da-001b21ad7873 N/A da2p2

gptid/0c55ca54-f193-11e2-9b74-001b21ad7873 N/A da3p2

gptid/d87d2b8b-b42b-11e6-b994-001b21ad7873 N/A da4p2

gptid/7965b700-a103-11e5-ad6f-001b21ad7873 N/A da5p1

gptid/e2281c35-a0fb-11e5-bcd0-001b21ad7873 N/A da6p1

gptid/e23168e7-a0fb-11e5-bcd0-001b21ad7873 N/A da6p2

gptid/14548422-9c71-11e5-af38-001b21ad7873 N/A ada0p1

gptid/f49ae910-a562-11e0-93da-001b21ad7873 N/A ada2p2

 

[Tekkie]

zpool import -m - you will lose the last few seconds of transactions. What are you using as an SLOG?

"-m" did not work, I had to add "-F" to get the pool imported for some reason.

I am using an SSD drive for the SLOG, which I was planning on removing this evening as I was going to do some open heart surgery on my FreeNAS system as it has a couple of drives that are 6+ years old and I wanted to preemptively replace those. I guess the drives felt this and decided to gang up and kill the SLOG drive. :D

 

[Ericloewe]

I am using an SSD drive for the SLOG

The important question is "what SSD?" - an SLOG without power loss protection is an expensive "make my server slower" device, since it provides about as much protection as disabling sync.

 

[Tekkie]

The important question is "what SSD?" - an SLOG without power loss protection is an expensive "make my server slower" device, since it provides about as much protection as disabling sync.

I did realize that after adding it, that's why I was going to remove it this evening. ;)

I was able to add the pool and remove the missing logs drive and so far "zpool scrub" has not found any issues...

 

[Ericloewe]

It shouldn't, data at rest would not have been affected.

 

[Tekkie]

Code:

		storage1/movies:<0x0>

		storage1/backup/time-machine:<0x0>

Yes I imported the pool and scrubbed it, and no errors were found this morning however I am seeing the above, and I stomped because I can't really find much on the internet about error <0x0>.

 

[Ericloewe]

Okay, looks like you were unlucky and ended up with corrupt metadata. My suggestion is to try copying that stuff to new datasets, deleting the old ones, and scrubbing again.

 

[Tekkie]

Rebuilt my box with some new drives and copied all the data from the old dataset, somehow nothing was actually corrupted by the looks of things.