totally hosed for now

[Steve Simpson]

I shutdown, then added a drive, then rebooted.
On the boot screen it says it can't find the pool "Neo".
When I click storage it "error getting available space". The Alert icon is blinking and when I click it it says the status is "". The view disks page shows the disks are there and still part of the pool "Neo".
So I shutdown, removed the additional drive, and rebooted.
I still have the exact same situation. I restored the saved configuration. No effect. During the boot it says it can't find the pool, "error getting available space", Alert icon with a status of blank, the GUI says the disks are still supposedly part of the pool Neo.
Any ideas on how to recover?

 

[Steve Simpson]

I should have said - this server is running 8.0.4.

 

[Steve Simpson]

here are some snapshots of the GUI

 

[cyberjock]

If you look at the view disks, you'll see that one disk appears to be a stripe. You added individual disks to the zpool after it was originally created. If that disk has any problems you will be unable to access your data. My first guess is that you have another disk that is offline making your pool unmountable. If you've removed any disks from your server that were bad or you otherwise didn't want, don't do anything with them yet. If you read my presentation(link in my sig) slides 29 and 30 show what you have. You do not have protection from disk failure for all disks in your pool.

First, upgrade to 8.3.2 so you can use some newer ZFS code. Or install 8.3.2 onto a USB temporarily for troubleshooting.

Second, post the output of zpool status and zpool import. That will tell us what is going on with your pool.

 

[Steve Simpson]

zpool status said there were no pools.

I upgraded to 8.3.2 and it has been hung at this point for an hour or more:
apply database schema changes
(ada4:ata0:0:0:0): lost device

it just sits there and hasn't progressed after an hour or so.

 

[Steve Simpson]

oh cool. It finally came back to the command prompt. zpool status still says "no pools available".

here is the zpool import:

 

[cyberjock]

To be honest, I think you are done for. Here's why I think that:

1. zpool import is warning you that additional devices are missing.
2. The ada4 lost device pretty much means that one disk is bad.
3. If you look at the view_disks.jpg you can see that ada4 was a stripe. Because it was a stripe, if that disk fails the pool is unmountable and all data in the pool is inaccessible. Even if the disk was physically moved elsewhere or otherwise changed and wasn't ada4, you still have the issue with #1; disks are missing and it knows it isn't mountable.

You have 2 options:

1. Destroy your pool and recreate it from scratch, then recover your data from backups, or
2. You didn't make backups in which case you have lost your data unless you contact a hard drive recovery company to have them do a recovery of that one disk, then when you get the new disk back try to do an import and pray to your diety it works.

It sure does look like you added a disk to the pool after you created the initial vdev, than wrongly assumed it would fall within the RAIDZ1 vdev. It doesn't. Then that one disk failed and you are left with a zpool that is unmountable.

Expect to pay something around $1000 for disk recovery if your data is that important.

 

[Steve Simpson]

Thanks for your input and help.

The stripe was unintentional. I don't know how it got there.

I made two identical freenas servers each with 5 3TB drives using raidz (or so I thought). The second server is a backup of the first in case something went wrong (which now it did). Hopefully the backup server will stay alive long enough for me to rebuild the first server and copy everything back to it. I may have lost a few files but not many. I sync the two servers together every few days to make sure the backup server is up to date. Recently the sync started to fail partway through the sync and that is when I noticed the problems and the striped drive. I hadn't even gotten to to point of fixing the stripe issue before everything went downhill.

Thanks for your help.

 

[cyberjock]

You put a smile on my face. Not because you lost data, but because you did backups. Such a small number of people have backups that I really thought when I was writing my last response you were going to say something like "guess I'll be getting a divorce when my wife finds out" since that seems to be the norm. :P