Zpool problems

[Terence Faul]

Are any of these destructive commands?
Should they be done to the data disks, or the OS?

 

[titan_rw]

No, of course not. They're all read only. The "if=" is input file. The output is always piped into other commands for processing, or let output to the screen.

They can even be ran on a live system (mounted pool), as I did to show my examples.

EDIT: The data disks. They are what we're trying to find the pool on.

 

[joeschmuck]

So thanks to the others who posted why the drive was a failure and as far as I can tell it is the only drive which is failing right now. Since you say you had a RAIDZ1 pool, your pool may be able to be restored but if it were just a striped pool then your data is gone. As for if your pool is recoverable, well the guys here are trying to help to recover it, assuming it can be recovered.

 

[Terence Faul]

Ok I am running the first command on my first drive, is I'm just looking at a curser on the very next line, I assume that it is running?

 

[Terence Faul]

If I pop any of these drives into a external enclosure and connect to a linux box by usb is there any way to mount the drive and recover some of the data?

 

[joeschmuck]

Ok I am running the first command on my first drive, is I'm just looking at a curser on the very next line, I assume that it is running?

Please be specific in what you are running. I may assume you mean one thing but you are actually doing something else. I'm not trying to be mean or anything, it's just the best way to convery what you are doing and to remove any confusion.

 

[joeschmuck]

If I pop any of these drives into a external enclosure and connect to a linux box by usb is there any way to mount the drive and recover some of the data?

No, you cannot recover the data that way, if you have a RAIDZ1 then only one drive can be missing, the failed one of course. The rest must be present.

 

[Terence Faul]

I am currently running

Code:

dd if=/dev/da0p2 | strings | grep pool_guid

 

[titan_rw]

So thanks to the others who posted why the drive was a failure and as far as I can tell it is the only drive which is failing right now. Since you say you had a RAIDZ1 pool, your pool may be able to be restored but if it were just a striped pool then your data is gone. As for if your pool is recoverable, well the guys here are trying to help to recover it, assuming it can be recovered.

Even if it was a striped pool, the other good disks should show the zfs labels via the zdb -l command. And I'd expect 'zpool import' to complain about not enough devices or something. The fact that no zfs labels are showing up at all on any of the disks is strange.

 

[titan_rw]

I am currently running

Code:

dd if=/dev/da0p2 | strings | grep pool_guid

That should show the "pool_guid" fairly quickly:

Code:

root@nas ~ # dd if=/dev/da0p2 | strings | grep pool_guid
  pool_guid
  pool_guid
^C33459+0 records in
33459+0 records out
17131008 bytes transferred in 1.700069 secs (10076655 bytes/sec)

 

[Terence Faul]

Nope I am still waiting, each drive is 2TB. (See attached)

I only get a quick response if I run it on the boot device (The New fresh install USB I made)

 

[titan_rw]

Stop it then. It's just wasting time.

Try this:

Code:

dd if=/dev/da2p2 bs=1k count=1k | hd | grep "0c b1 ba 00"

That searches the first megabyte (1024*1024) for uberblocks. Should only take a few seconds.

I've got 64, from offset 20000 up to 7f000.

Change the drive of course.

Running it against one of my boot drives I get 256 uberblocks (lines).

 

[Terence Faul]

Ok this is the result

Code:

[root@freenas] ~# dd if=/dev/ada0p2 bs=1k count=1k | hd | grep "0c b1 ba 00"
1024+0 records in
1024+0 records out
1048576 bytes transferred in 0.195593 secs (5361013 bytes/sec)

EDIT: in fact the result is the same on all 5 drives

 

[titan_rw]

Try something like this:

Code:

dd if=/dev/da2 bs=1m count=4k | hd | grep "0c b1 ba 00"

Not the lack of the p2, so it'll search starting at the very start of the disk. And the 1 meg block size, and 4k count so it'll search the first 4 gigs of the disk regardless of partitions.

It can be cpu bound as hexdump does it's thing. I still get my uberblocks within 20 seconds or so (cut for brevity):

Code:

root@nas ~ # dd if=/dev/da2 bs=1m count=4k | hd | grep "0c b1 ba 00"
40030000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40031000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40032000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40033000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40034000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40035000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40036000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40037000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40038000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|
40039000  0c b1 ba 00 00 00 00 00  88 13 00 00 00 00 00 00  |................|

Which makes sense, as 40030000 hex is 1073938432 bytes, which is 1024 megabytes (1 gig) + .1875 megabytes (192k).

If you haven't gathered, I have 1 gig swap partitions:

Code:

root@nas ~ # gpart show da2
=>  34  5860533101  da2  GPT  (2.7T)
  34  94  - free -  (47k)
  128  2097152  1  freebsd-swap  (1.0G)
  2097280  5858435848  2  freebsd-zfs  (2.7T)
  5860533128  7  - free -  (3.5k)

 

[Terence Faul]

That is also taking a long time, again the same command on my boot drive quickly returns results

 

[titan_rw]

Let it finish. You can press Control-t to see how many blocks (out of 4,096) it has done:

Code:

load: 0.75  cmd: hd 41651 [running] 71.64r 68.18u 0.42s 100% 1628k
1262+0 records in
1258+8 records out
1323302912 bytes transferred in 71.677958 secs (18461783 bytes/sec)

 

[Terence Faul]

Ok done, but still no result, I am thinking this is bad.

Code:

[root@freenas] ~# dd if=/dev/ada0 bs=1m count=4k | hd | grep "0c b1 ba 00"
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 786.348227 secs (5461915 bytes/sec)

 

[titan_rw]

Yea, this means it couldn't find any uberblocks in the first 4 gigs.

You can always look at the raw data itself:

Code:

dd if=/dev/da2p2 bs=1k count=1k | hd | less

Then scroll around (page up / down, or <space> and b). I can see host name, vdev metaslab stuff, etc.

 

[titan_rw]

If you see something like this:

Code:

root@nas test # dd if=/dev/da2p2 bs=1k count=1k | hd | less
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000

That means that particular section of disk is blank (all zeros). Hexdump by default omits identical lines for readability. You can tell it not to: (-v)

Code:

dd if=test bs=1k count=1k | hd -v | less

 

[Terence Faul]

Code:

~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~

I think that I am ready to throw the towel in, I will have to say goodbye to my data!
Thank you to everyone who has attempted to help me.