ZFS disk lost horror

[Mike]

Hi all. My FreeNAS 8.0.1b2 install crapped out on me tonight and i have been trying to recover from it since already half a day ago. I tried, i really did but i guess i need some help before it really turns ugly (if everything is not already lost).

Ok, so; I'm running 2*2tb disks in a mirror zfs pool, or well, i used to. All of a sudden samba stopped working while nfs didnt. After a reboot nothing worked, on inspection of the console i found out rather quick that there were no zpools whatsoever. status, import, it wouldnt budge. I made a quick install on a seperate disk and tried to import the mirror or just 1 disk and it wouldn't show up there either.

I'm at a standstill and took both of the disks to a spare box with a fresh install of freenas and i'm trying to find the pools again and so far no luck.
ZFS list, zpool status, zpool import. It all tells me it's empty or nothing is found. With 1 disk dead i guess i would be able to atleast find the other disk.

My inexperience with ZFS is really showing off and i have no clue on what to do. If somebody could shine some light on this case i would really appreciate it. I have partial backups but some of the files were in the process of getting copied and i would like to get those back.

Thanks for the help. Mike

 

[ProtoSD]

Mike,

Try not to panic just yet. Your data should still be there. Hopefully @matthewowen01 is around here cause he is the master and hopefully can shine some light on things.

Just to verify, your drives are showing up during POST, and does FreeNAS recognize them / can you see them if you do 'dmesg' from the command line?

 

[Mike]

Thanks for the help and trust Protosd! I'm pretty confident that the data is indeed there but I haven't practiced nearly enough (read; barely) to do some troubleshooting on ZFS. The irony is that it has only been running for about 3 weeks now and i was just backing some files up when this happened. First Cifs/SMB got a time out, but NFS was still working. I completed the transfer over NFS and tried to login on the webgui later. At that time the GUI and SMB were not working anymore while NFS strangely was. Don't know if this ended up in /dev/null right away but it was accessible for the moment that i checked while SMB was out already.

This is probably just a side effect. To clear it all up a bit; I cannot find the pools with zpool import and the usual parameters. The drives do show up in dmesg, /dev and i can do a SMART scan without problems. Neither of the disks show up errors or excessive values although SMART is ofcourse only so SMART. zpool status shows nothing, it seems my FreeNAS is having amnesia of some sort. I'm sure that it had a ZFS mirror configured as it did show up in the past.

Thanks in advance.

 

[matthewowen01]

ok, what's the output of 'gpart show'

 

[Mike]

The output of gpart show only seems to show the primary disk of Freenas and not either of the data disks.

FreeNAS# gpart show
=> 63 16777152 da0 MBR (8.0G)
63 953505 1 freebsd [active] (466M)
953568 63 - free - (32K)
953631 953505 2 freebsd (466M)
1907136 3024 3 freebsd (1.5M)
1910160 41328 4 freebsd (20M)
1951488 14825727 - free - (7.1G)

=> 0 953505 da0s1 BSD (466M)
0 16 - free - (8.0K)
16 953489 1 !0 (466M)

FreeNAS#

In the /dev/ folder da1 and da2 show up but no subdevices of those and i'm not sure if that's right. I thought there weren't supposed to be partitions with zfs so subdevices wouldn't show up like with ufs or other filesystems.

 

[ProtoSD]

do a gpart show for both of those disks, like 'gpart show /dev/da1' and then da2, and post. I'm off to bed, but maybe @matt is still around and he can help.

 

[Mike]

A gpart on either da1 or da2 comes up with; no such geom: da*
From that it looks like the GPT tables on both disks are corrupt or completely wiped and I'm really starting to wonder why my karma turned this bad all of a sudden. Hahaha.

I guess i'm out of luck on getting this repaired. After all there were no critical files on there yet so lets say this was more of an eyeopener to keep a folder with important files rsync'ed with an external system.

 

[Tekkie]

You are right... RAID is not a replacement for backups ;)

 

[Mike]

Indeed, it isn't. I did mirror the disks though to guarantee a continue service availability, or to be able to go on crippled on a single disk but something bad happened to both disks from either a failure or a bug. Unforseen, oh well.
I did run on a pretty low value of RAM and this happened with a big file transfer. Maybe it decided to crap out on me because of that.

Either way, i'm formatting the disks now and will be testing the disks afterwards. I'm still somewhat sceptical about the WD EARS made in Thailand.
Thanks for the help everyone.

 

[matthewowen01]

it looks like somehow your original format and partitioning was blown away.

did you accidentally format the disks, or mess around with dd, maybe click 'destroy volume' in the gui?

 

[Mike]

I was not touching the box in any other way than a SMB transfer from the htpc.

I'm back online again and will be testing for the coming weeks to see if a similar case pops up or if this is a one time error. Either way, rsync and cron jobs are mandatory from now on.

Thanks.

 

[matthewowen01]

I was not touching the box in any other way than a SMB transfer from the htpc.

I'm back online again and will be testing for the coming weeks to see if a similar case pops up or if this is a one time error. Either way, rsync and cron jobs are mandatory from now on.

Thanks.

now just for kicks, what's the output of 'gpart show' with your working setup.

 

[bluesteel]

Help, Same thing happened to me

I have been running the freenas setup for about 7 months with no issues. Randomly this morning my iSCSI volumes went offline and I rebooted freenas. Things did not come back online. After some research it looks like the zpool is gone. Both of my disks are still there and they look healthy. Zpool import shows nothing and if I do gpart show on the disks I get the geom error.

Any ideas besides formatting? I have some backups but they are about a month old. :(

Thanks

 

[bluesteel]

I have been running the freenas setup for about 7 months with no issues. Randomly this morning my iSCSI volumes went offline and I rebooted freenas. Things did not come back online. After some research it looks like the zpool is gone. Both of my disks are still there and they look healthy. Zpool import shows nothing and if I do gpart show on the disks I get the geom error.

Any ideas besides formatting? I have some backups but they are about a month old. :(

Thanks

I fixed my zpool. I am wondering if vmware didn't do something to my partitions.

My GPT partitions on both drives of the mirror became corrupted. Since they were both corrupted zfs could not see the volume with the zpool import command.


WARNING: The steps below are extremely dangerous. You could lose all your data. You have been warned.

To fix this I had to mount the root file system as writable ( mount -uw / )
Then install gdisk ( pkg_add -r gdisk )

Then I followed the instructions on this site:
http://www.rodsbooks.com/gdisk/repairing.html

For me I had to do the following on both my drives: This my vary based on your issues. See the website above.

gdisk /dev/da1
Choose 1 for keep current partition.
and the w to write the changes.

Then I rebooted and the zpool came back online on its own.


My current Setup:
VMware ESXi 4.1 on a Proliant ML150
FreeNas 8 installed on a vmkd
Software ZFS mirror on 2 - 1tb drives mapped to Freenas using Raw device mappings.

 

[area51tazz]

Good post bluesteel...That may come in handy someday :)