Help putting Zpool backon line

[norskman]

I am running V9.2.1.5 - I have a slowly failing drive. I offlined it. Powered down. Replaced it with a new drive. Powered backup - I thought I would be able to do a 'replace' and be done.

Now my Zpool has totally dissapeared.

So I put back my old drive.
Here you see the output of 'zpool status'

[root@freenas ~]# zpool status pool:
store state: DEGRADED
status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: resilvered
63.9G in 0h58m with 0 errors on Mon Jun 16 03:03:36 2014
config:
NAME STATE READ WRITE CKSUM
store DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
gptid/16f6ea24-de85-11e3-93d4-e840f2ed2010 ONLINE 0 0 0
gptid/6fe95451-2f51-11e2-83bb-e840f2ed2010 ONLINE 0 0 0
6023222925197702943 OFFLINE 0 0 0 was /dev/gptid/705ffb46-2f51-11e2-83bb-e840f2ed2 010
gptid/70dc6782-2f51-11e2-83bb-e840f2ed2010 ONLINE 0 0 0 logs
gptid/7128306b-2f51-11e2-83bb-e840f2ed2010 ONLINE 0 0 0

errors: No known data errors


I tried the command:
dd if=/dev/zero of=6023222925197702943 bs=1m count=2 seek=2048

The response from Freenas is dd: 6023222925197702943: read-only file system

Can any one help me to get this drive back on-line?

Or maybe understand why the new drive causes the whole zpool not to mount at all?? even though it has been replaced in exactly the same position as the failed drive?

If this does not work - I suppose I could copy off the data to another drive and then start again. I currently CAN access the pool - only 3.4Tb to copy.....could be worse!

Any clues to fixing this would be gratefully received.

 

[krikboh]

Sounds like you pulled the wrong drive making the pool unmountable. Did you verify by the serial number you pulled the bad drive?

 

[cyberjock]

First, post the output of the following in pastebin. Pastebin preserves the formatting and is critical for problems like this.

camcontrol devlist
zpool status
zpool import

Second, don't try to do anything until you hear back from me. That dd command was a clear sign you are a noobie and I'd rather you not lose your data because you tried something that backfired. ;)

 

[norskman]

I will post back.

I am copying off the data to another external drive overnight.
I did not pull the wrong drive. The zpool is degraded but accessible.

I am a noobie....especially for non gui access.

I will complete the copy and process the commands as requested.

 

[gpsguy]

Please don't start any more threads on this issue.

This is the 3rd message for the same issue, in the past couple of days.

On Friday, the OP started this one: http://forums.freenas.org/index.php?threads/hdd-error.21582/#post-126495

We all thought he was going to replace the drive. Then, out of the blue, we learn that he reinstalled the OS in this thread: http://forums.freenas.org/index.php...p-net-inet-tcp-delayed_ack.21606/#post-126236 and he was posting in both threads.

Now, this message. With three threads on the topic, one can't easily follow the trail of what's been done.

 

[norskman]

Apologies for the confusion. This is the history.
My initial problem was my one of my HDD's - Advice - change it.
So I did.
I thought that was the end of this matter - I knew how to do that.
So thread closed.

Next message - separately was my machine kept hanging at startup - advice - boot up and install FreeNAS from scratch and then auto import.
did that. It worked. Thought that was the end of this matter.
so thread closed.

Next message - replace failing drive ( actually my second drive in a ZFS of 5 drives - all SATA). took it offline - as per manual .fine. Shutdown. Put in new drive. No ZFS pool.
Panic....
ok replace drive. ZFS pool returns - degraded and old drive - shows up - but will not go online.

I am only following one thread now.
This one:http://forums.freenas.org/index.php?threads/help-putting-zpool-backon-line.21637/#post-126575

This is the only open thread on this matter right now.

I am backing-up the data - onto anther SATA drive and then will be a position to work on this. I want my data secured firstly.

Once that process is complete. We can attempt to fix this.

 

[norskman]

see here

http://pastebin.com/zMgHajd0

For an answer to Cyberjocks request for data from me.

 

[norskman]

I have posted the output to Cyberjocks requests in tyhe Pastepin under ZFS Pool error

You should see it here:

http://pastebin.com/zMgHajd0

It is public

 

[cyberjock]

Ok, so when you ran those commands were all of your disks(even the bad one) installed in the system?

At the time you ran those commands you were already missing a disk in your pool. If you lose a second disk your pool won't mount. So either you offlined a good disk and then physically removed a bad disk, or you have 2 bad disks and don't know about it(yet).

In any case, your pool is down a single disk right now. Since you made the choice to go with RAIDZ1 if you lose another disk you will lose all of your data. I'm going to presume that the reason your pool disappeared is because for whatever reason you lost 2 disks at the same time.

 

[norskman]

I have 4 x 3TB - physical in the Zpool. Yes a RAIDZ1. With only one drive spare.

Initially, the Zpool was totally ok. I was getting the errors only on a single drive - ( actually the second one).

I have only ever removed that drive and replaced.

The output of pastebin - is of the 4 drives.
I offlined the drive - powered down. Replaced with a brand new drive. Thats when the pool was TOTALLY offline.
Thats why I had to put the old drive back. But I am also sure that FreeNAs recognises this as an old drive and will not allow its re-use.

So what you see is now, that old drive in physical slot 2. Not online - with the ZPOOL Degraded.
Right now I am copying off data from the Zpool.

But I don't know how to get round this.

One way, once I am satisfied that I have a backup is literally to start again. I am sure there is nothing wrong with the drives I have - Except the faulty one, which I have a new replacement for.

 

[cyberjock]

Well, your camcontrol devlist shows only 3 drives that are Seagate 3TB drives and an OCZ drive plus a USB stick. So I'd say one of your 3TB drives is failing and you don't know it or you didn't have a drive properly connected to power and SATA causing your problem.

In any case, your pool didn't mount because insufficient replicas existed for the pool to function. That problem is obviously resolved right now. Your guess is as good as mine as to why the pool didn't mount before. :/

 

[norskman]

I went into the BIOS. You were 100% correct. The drive was not showing.

I realised why. I pulled the wrong drive.

So I did offline the correct drive. Replacing the wrong drive with a new one - killed the ZFS pool - that is 100% correct for a Z1 ZFS system.
Now I replaced the correct totally dead - not even appearing in the BIOS with the new drive. BINGO - all four drives are there.

Booted up.

ZFS is there but degraded.
Did a 'REPLACE'. Drive was initialised and now is in 'zpool status' is showing as 'replacing-2'.

So lessons to be learned:
1. Check BIOS and cables for physical attachments first.
2. Double check failed drive is the correct one.
3. Better to have a Z2 array than a Z1. Cost is not really an issue any more.
4. Don;t panic
5. Look for the obvious - not the esoteric.


I know there is a command that shows the progress of the resilvering process -

FreeNAS is a great system - very robust indeed

This forum has great help on it. Many thanks to those who pointed me in the right direction

 

[cyberjock]

zpool status will show you the status of the resilvering.

The biggest thing to learn from this thread... "don't panic". That right there would save many people if they'd not panic. ;)