Zpool problems

Terence Faul · Jan 17, 2016

Hi

Please help, my Freenas box had a panic in boot. As a result I built a new boot drive and booted the system.

I am now trying to import the zpool volume to the new system, but the system is not picking it up.
I tried through the GUI but there are no volumes to import.

The pool is 5 drives 2 TB each (WD)

running on intel i3 with 16GB memory and the system has run great from over 3 years

pool status:

Code:

[root@freenas] ~# zpool status

  pool: freenas-boot

state: ONLINE

  scan: none requested

config:


    NAME        STATE     READ WRITE CKSUM

    freenas-boot  ONLINE       0     0     0

      da0p2     ONLINE       0     0     0


errors: No known data errors

gpart list:

[root@freenas] ~# gpart list

Geom name: da0

modified: false

state: OK

fwheads: 255

fwsectors: 63

last: 15728606

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: da0p1

  Mediasize: 524288 (512k)

  Sectorsize: 512

  Stripesize: 0

  Stripeoffset: 17408

  Mode: r0w0e0

  rawuuid: b14ae654-bcf0-11e5-81b5-0014d1fa056b

  rawtype: 21686148-6449-6e6f-744e-656564454649

  label: 1

  length: 524288

  offset: 17408

  type: bios-boot

  index: 1

  end: 1057

  start: 34

2. Name: da0p2

  Mediasize: 8052498432 (7.5G)

  Sectorsize: 512

  Stripesize: 0

  Stripeoffset: 544768

  Mode: r1w1e1

  rawuuid: b152e8e9-bcf0-11e5-81b5-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 8052498432

  offset: 544768

  type: freebsd-zfs

  index: 2

  end: 15728599

  start: 1064

Consumers:

1. Name: da0

  Mediasize: 8053063680 (7.5G)

  Sectorsize: 512

  Mode: r1w1e2


Geom name: ada0

modified: false

state: OK

fwheads: 16

fwsectors: 63

last: 3907029134

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: ada0p1

  Mediasize: 2147483648 (2.0G)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 74787420-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 2147483648

  offset: 65536

  type: freebsd-swap

  index: 1

  end: 4194431

  start: 128

2. Name: ada0p2

  Mediasize: 1998251364352 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 748a4d81-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 1998251364352

  offset: 2147549184

  type: freebsd-zfs

  index: 2

  end: 3907029127

  start: 4194432

Consumers:

1. Name: ada0

  Mediasize: 2000398934016 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0


Geom name: ada1

modified: false

state: OK

fwheads: 16

fwsectors: 63

last: 3907029134

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: ada1p1

  Mediasize: 2147483648 (2.0G)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 71e6f2f2-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 2147483648

  offset: 65536

  type: freebsd-swap

  index: 1

  end: 4194431

  start: 128

2. Name: ada1p2

  Mediasize: 1998251364352 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 71f5a9b2-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 1998251364352

  offset: 2147549184

  type: freebsd-zfs

  index: 2

  end: 3907029127

  start: 4194432

Consumers:

1. Name: ada1

  Mediasize: 2000398934016 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0


Geom name: ada2

modified: false

state: OK

fwheads: 16

fwsectors: 63

last: 3907029134

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: ada2p1

  Mediasize: 2147483648 (2.0G)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 72f6036b-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 2147483648

  offset: 65536

  type: freebsd-swap

  index: 1

  end: 4194431

  start: 128

2. Name: ada2p2

  Mediasize: 1998251364352 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 73083c48-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 1998251364352

  offset: 2147549184

  type: freebsd-zfs

  index: 2

  end: 3907029127

  start: 4194432

Consumers:

1. Name: ada2

  Mediasize: 2000398934016 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0


Geom name: ada3

modified: false

state: OK

fwheads: 16

fwsectors: 63

last: 3907029134

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: ada3p1

  Mediasize: 2147483648 (2.0G)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 73c76e47-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 2147483648

  offset: 65536

  type: freebsd-swap

  index: 1

  end: 4194431

  start: 128

2. Name: ada3p2

  Mediasize: 1998251364352 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 73de0e73-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 1998251364352

  offset: 2147549184

  type: freebsd-zfs

  index: 2

  end: 3907029127

  start: 4194432

Consumers:

1. Name: ada3

  Mediasize: 2000398934016 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0


Geom name: ada4

modified: false

state: OK

fwheads: 16

fwsectors: 63

last: 3907029134

first: 34

entries: 128

scheme: GPT

Providers:

1. Name: ada4p1

  Mediasize: 2147483648 (2.0G)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 727d4b14-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 2147483648

  offset: 65536

  type: freebsd-swap

  index: 1

  end: 4194431

  start: 128

2. Name: ada4p2

  Mediasize: 1998251364352 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0

  rawuuid: 728ef0e9-d841-11e3-b528-0014d1fa056b

  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b

  label: 1

  length: 1998251364352

  offset: 2147549184

  type: freebsd-zfs

  index: 2

  end: 3907029127

  start: 4194432

Consumers:

1. Name: ada4

  Mediasize: 2000398934016 (1.8T)

  Sectorsize: 512

  Stripesize: 4096

  Stripeoffset: 0

  Mode: r0w0e0


gpart show:

[root@freenas] ~# gpart show

=>      34  15728573  da0  GPT  (7.5G)

        34      1024    1  bios-boot  (512k)

      1058         6       - free -  (3.0k)

      1064  15727536    2  freebsd-zfs  (7.5G)

  15728600         7       - free -  (3.5k)


=>        34  3907029101  ada0  GPT  (1.8T)

          34          94        - free -  (47k)

        128     4194304     1  freebsd-swap  (2.0G)

    4194432  3902834696     2  freebsd-zfs  (1.8T)

  3907029128           7        - free -  (3.5k)


=>        34  3907029101  ada1  GPT  (1.8T)

          34          94        - free -  (47k)

        128     4194304     1  freebsd-swap  (2.0G)

    4194432  3902834696     2  freebsd-zfs  (1.8T)

  3907029128           7        - free -  (3.5k)


=>        34  3907029101  ada2  GPT  (1.8T)

          34          94        - free -  (47k)

        128     4194304     1  freebsd-swap  (2.0G)

    4194432  3902834696     2  freebsd-zfs  (1.8T)

  3907029128           7        - free -  (3.5k)


=>        34  3907029101  ada3  GPT  (1.8T)

          34          94        - free -  (47k)

        128     4194304     1  freebsd-swap  (2.0G)

    4194432  3902834696     2  freebsd-zfs  (1.8T)

  3907029128           7        - free -  (3.5k)


=>        34  3907029101  ada4  GPT  (1.8T)

          34          94        - free -  (47k)

        128     4194304     1  freebsd-swap  (2.0G)

    4194432  3902834696     2  freebsd-zfs  (1.8T)

  3907029128           7        - free -  (3.5k)

joeschmuck · Jan 17, 2016

To provide you with proper help you will need to provide us with the information requested by the forum rules. The FreeNAS version you are running, your hardware configuration, additional info like was your pool encrypted, how was it configured, etc... Be as descriptive as possible and definitely do not try something if you are not 100% certain about what you are doing. Also include exactly what you did to recover from the failure, including if you used a new boot device. Also, in the future, don't just post data the way you did, I put it into code brackets but the format was already screwed up. Again, forum rules please.

Terence Faul · Jan 17, 2016

Sorry. I thought I did that. How do I do code brackets.

I have not tried recovery yet all I have done is create a new freenas usb bootable and boot in. I then tried to import the volume, but the volume does not come up.

As mentioned zpool list does not show the pool.

Running freenas 9.3
Gigabyte motherboard.
Intel I3
16 GB ram
5x WD 2 TB hdd
Not encrypted
Zpool

gpsguy · Jan 17, 2016

Select the insert icon, circled in red below, and select code off the drop down menu.

You an also do it manually, like this: (CODE)write stuff in here(/CODE) If you do it manually, you need to use square brackets [ ] instead of parenthesis ( ). I only used ( )'s, to give you an idea what it looks like.

Terence Faul said:
How do I do code brackets.

joeschmuck · Jan 17, 2016

Some additional questions:
1) Before your system died, what version of FreeNAS were you running?
2) Did you restore the same version of FreeNAS that you were running?
3) Although I know you already said it, I'll ask just to be thorough, did you follow the steps outlined in the FreeNAS user guide to import your pool? (the link is here)
4) Did you try to restore you backup configuration file? (assuming you made a backup configuration file)
5) Do you recall how the pool was created (GUI or CLI)?

One good thing you have going for you is all the drives appear to be recognized. We really don't want to force mounting the pool unless it's really required. Also, someone else may have to take over here giving you advice, I'm an old man and I'm preparing for an old man operation tomorrow so starting in a few hours I will likely not be on the internet much until late tomorrow evening, plus I wouldn't trust any advice I give tomorrow as I could still be drugged up (ooh, Happy Drugs)

Terence Faul · Jan 17, 2016

joeschmuck said:
Some additional questions:
1) Before your system died, what version of FreeNAS were you running?
2) Did you restore the same version of FreeNAS that you were running?
3) Although I know you already said it, I'll ask just to be thorough, did you follow the steps outlined in the FreeNAS user guide to import your pool? (the link is here)
4) Did you try to restore you backup configuration file? (assuming you made a backup configuration file)
5) Do you recall how the pool was created (GUI or CLI)?

One good thing you have going for you is all the drives appear to be recognized. We really don't want to force mounting the pool unless it's really required. Also, someone else may have to take over here giving you advice, I'm an old man and I'm preparing for an old man operation tomorrow so starting in a few hours I will likely not be on the internet much until late tomorrow evening, plus I wouldn't trust any advice I give tomorrow as I could still be drugged up (ooh, Happy Drugs)

Version 9.3.
Yes restored the same
Yes tried import steps
No backup
And pool created from gui

Robert Trevellyan · Jan 17, 2016

Please post the output from running
zpool import
in the Shell or (preferably) via SSH.

Do not attempt to actually import the pool from the CLI unless guided by an experienced forum member.

Robert Trevellyan · Jan 17, 2016

Terence Faul said:
my Freenas box had a panic in boot

Please post as much detail as you can about the original panic that led to the current situation.

Terence Faul · Jan 17, 2016

Code:

[root@freenas] ~# zpool import
[root@freenas] ~#

This is the problem, there is no output.
I have a free USB key that I used to boot, all other hardware is the same. Is the pool info on the drives itself, or the boot disk?

Terence Faul · Jan 17, 2016

Here is a screenshot of the panic

Terence Faul · Jan 18, 2016

This is the result for all disks with:

Code:

zdb -l /dev/ada4

Code:

[root@freenas] /# zdb -l /dev/ada0
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
[root@freenas] /# zdb -l /dev/ada1
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
[root@freenas] /# zdb -l /dev/ada2
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
[root@freenas] /# zdb -l /dev/ada3
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
[root@freenas] /# zdb -l /dev/ada4
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

I am not getting a good feeling about this

joeschmuck · Jan 18, 2016

Well this is outside my comfort, I'd do a few things risking my data but not yours.

Until someone else can chime in how about checking your drives for mechanical errors by running a long SMART test on each one. The tests can be run all at the same time. The command to start the SMART test is: "smartctl -t long /cev/ada0" and then run that for each drive (ada1, ada2, etc...). Wait 5 hours for the tests to complete and then post the output of "smartctl -a /dev/ada0" and do that for all drives. ID's 5, 196, 197, and 198 are key indicators if there is a hard drive problem. Even if you have these errors, it doesn't mean your data is gone, it's just additional info to help someone figure this out for you.

Terence Faul · Jan 18, 2016

Thanks, I have started the tests, but in the mean time the current "RAW" values for all on all drives is 0

joeschmuck · Jan 18, 2016

Hopefully after the test it remains at zero. I didn't really think this was your issue but it rules out a possible problem for the next person helping you out. 2 hours before the happy drugs!

Terence Faul · Jan 18, 2016

good luck, is there any other info that you or anyone else would need, I am bumping my head.

Regards

joeschmuck · Jan 18, 2016

After the SMART tests complete you could try something else and this is grasping at straws for me, move your hard drives into another computer, any computer that can handle all the drives and boot off the USB flash drive. See if you can import the pool. I don't see this fixing the issue however it's just another way to rule something out.

Let me ask you this, what model motherboard and are you using a RAID controller card?
Have you tried to use a different USB flash drive and build the boot device again?
And you are definitely using the same version of FreeNAS that you were previously using? 9.3 changes every month or so.

Have you read this thread? https://forums.freenas.org/index.php?threads/controller-ate-my-zfs-pool.21611/

Terence Faul · Jan 18, 2016

Gigabyte Z77-DS3H
No Raid, Sata
yes to new USB, as the other could not even boot (Incidentally, does the USB boot drive have the ZFS info on it? Could that be why ifs list does not shoe it?)
I am sure it is the same version as I just updated it recently.
Reading now

jgreco · Jan 18, 2016

Terence Faul said:
Running freenas 9.3
Gigabyte motherboard.
Intel I3
16 GB ram
5x WD 2 TB hdd
Not encrypted
Zpool

More information. Motherboard model. CPU model. Drive models.

Terence Faul · Jan 18, 2016

intel HD 4000 graphics
motherboard Rev 1.1 (Model is GA-Z77-DS3H)
Ivy Bridge i3 processor
Drives are WD

WDC WD20EARS-00J99B0:

Capacity: 2 TB (2,000,398,934,016 bytes)

Model: WDC WD20EARS-00J99B0

Revision: 80.00A80

Serial Number: WD-WCAWZ1695717

Native Command Queuing: Yes

Queue Depth: 32

Removable Media: Yes

Detachable Drive: No

BSD Name: disk4

Medium Type: Rotational

Partition Map Type: GPT (GUID Partition Table)

S.M.A.R.T. status: Verified

Terence Faul · Jan 18, 2016

Here is the cam control devlist, incase that helps

Code:

[root@freenas] ~# camcontrol devlist
<WDC WD20EFRX-68EUZN0 80.00A80>    at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD20EARS-00J99B0 80.00A80>    at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus2 target 0 lun 0 (ada2,pass2)
<WDC WD20EZRX-00D8PB0 80.00A80>    at scbus3 target 0 lun 0 (ada3,pass3)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus4 target 0 lun 0 (ada4,pass4)
<ADATA USB Flash Drive 1100>       at scbus6 target 0 lun 0 (pass5,da0)

Important Announcement for the TrueNAS Community.

Zpool problems

Dabbler

Old Man

Dabbler

Active Member

Old Man

Dabbler

Pony Wrangler

Pony Wrangler

Dabbler

Dabbler

Attachments

Dabbler

Old Man

Dabbler

Old Man

Dabbler

Old Man

Dabbler

Resident Grinch

Dabbler

Dabbler

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Zpool problems"

Similar threads