ZFS damaged due to user error - need help (hopefully) recovering what is left.

[jason.rohm]

I have a serious issue. I hope someone can help.

I am in the process of retiring my Drobo 800i and building out a new FreeNAS environment.

During this process I built a secondary FreeNAS device with older hardware to temporarily hold my data so I can move disks to the new FreeNAS.

In my haste, I accidently told my secondary device's Volume1 to "Detach and mark new".

I almost immediately saw my error and powered off the machine in the hope that the changes would not be committed.

Unfortunately, the system boots and I still see the Volume configured but it is listed as "Unknown state".

I've SSH'd to the box and see the following:

"zpool list" reports "no pools available"
"zpool import -a" reports nothing and does nothing

The original disk appears to be online and is the same physical array as the boot (it was a CD install disk)

I'm obviously Linux/FreeBSD literate enough to do a lot of damage, but I'm unsure what to do next and am a little afraid of doing more damage.

Most of my truly irreplacable data (taxes, legal documents, etc) are available somewhere else (dropbox, Gmail, etc), but I have lots of personal photos and about 2TB of stuff I'd really rather not write off at this point.

Any serious help is welcome and I'll put up a $50 Amazon or BestBuy gift card for the person who successfully helps me recover this volume.

Thanks much!!

 

[cyberjock]

Yeah, typically the second you click that "accept" button, its too late. The writes that erase all the important data take a small fraction of a second to execute. :(

 

[titan_rw]

I think the 'mark disks as new' while exporting does the equivalent of a 'quick erase' in the gui. Which I think erases a portion of the beginning and end of the drive. This overwrites both locations that store the GPT. It may overwrite more than just the gpt areas though.

Maybe you could try recreating the partitions exactly as they would have been before the wipe? If I understand how freenas uses disks, the first part of the drive is used for swap. You'll need to know how big of a partition it was. You can tell under "settings - advanced - swap size on each drive". Defaults to 2 gig. So the zfs data partition should start after the swap partition. This should mean the start of the zfs partition should stil be intact? The end of the zfs partition might have suffered damage from the quick wipe though.

 

[jason.rohm]

I'm willing to give that a try before I go out and cry into a pint. Unfortunately, I'm not confident enough in the process to ensure I won't further damage the data.

Here's some more info, if someone can give me an idiot's checklist so I don't f-it up. I'd appreciate it.... I've already had one "Oh F$@#!" incident today. Don't need another.

Hardware array with 5x 400GB SATA disks.
Configured as one RAID 0 array (2TB)
FreeNAS (latest) x86 build.
System sees RAID disk as /dev/twed1
Old volume was "Volume1" and used full disk
Was mounted at /mnt/Volume1
Previously contained a bunch of iSCSI file extents and a direct CIFS volume. Each in their own folders.

I'm willing to give direct SSH/HTTPS access to the box as long as you are willing to explain what you are doing so next time I can help someone else (or atleast not have to beg).

Thanks.

 

[titan_rw]

I just did a test in a vm, and it failed. I wasn't able to reimport the pool after recreating the gpt.

Just to clarify, you have a zpool on a hardware raid array of 5x 400gb disks? Was the zpool created in the freenas gui? If so, what is the swap setting in the gui set for?

Going to try a few more things on my end.

 

[jason.rohm]

Yes, this was a fresh 8.3 build. Spun the thing up from scratch about four or five days ago. Everything was created through the GUI with more or less default settings.

I'm not sure about the swap settings, but if you want to look, I'll hook you up with access. Let me know how to reach you off-board.

 

[cyberjock]

Yeah, just recreating the GPT won't help. It's really a game over unless you are willing to spend 4-5 figure dollar amounts to data recovery companies.

 

[titan_rw]

I tested this in a VM, and I got it working. Not that a vm with test data, and bare-metal on top of hardware raid is the same thing, but this worked:

Ran a vm with a 6 gig .vmdk for data. Created a zpool on the test disk. Created a test file on the test pool. Exported the pool from the GUI while CHECKING 'mark disks as new'. Recreating the GPT from the command line, I was able to reimport the old pool. It shows my test file on the pool, so it's not just a 'new' pool created. Haven't verified the datas ok, as my test file was just a zero-filled file created with dd from /dev/zero.

I make no guarantees of course, but I'll take the 4-5 figures if I can get the data back.

jason.rohm: Can you get on irc? Would be easier done on irc with quicker back and forth times. Other than that, I could PM you the info to a teamspeak server I use.

 

[jason.rohm]

Don't have an IRC client loaded. I have XMPP. Check your private messages for info. If you don't have, let me know and I'll get myself on the IRC channel asap.

 

[cyberjock]

Interesting.. someone else had tested it and it didn't work. I wonder what changed, or if your VM's configuration was slightly different from the other person that had tried it and that made all the difference.

 

[jason.rohm]

titan_rw got it working. I'm looking forward to his synopsis posting. Thanks man!

 

[ProtoSD]

I make no guarantees of course, but I'll take the 4-5 figures if I can get the data back.

It's certainly worth more than $50 IMHO.

 

[titan_rw]

Success.

Got his pool back online. He's in the process of pulling data off of it now.

In this case, because the pool was on hardware raid, there's only one /dev/something device. If this happened with individual drives, you'd have to repeat the steps. Here's what I did.

Assumptions: /dev/da1 is the device in question.


First, I checked to see what swap size was set to, as I need this later. Swap creating confirmed at 2g.

Check various things:

Code:

"zpool status" = no pools.

"zpool import" = no results.

"gpart show da1" = "no such geom"

(re)Create gpt on da1:

Code:

[root@freenas] ~# gpart create -s GPT da1
da1 created
[root@freenas] ~# gpart add -b 128 -s 2g -t freebsd-swap da1
da1p1 added
[root@freenas] ~# gpart add -t freebsd-zfs da1
da1p2 added
[root@freenas] ~# gpart show da1
=>      34  12582845  da1  GPT  (6.0G)
        34        94       - free -  (47k)
       128   4194304    1  freebsd-swap  (2.0G)
   4194432   8388447    2  freebsd-zfs  (4G)

Note, this is the virtual disk on my VM, not the actual drive in question.

Code:

[root@freenas] ~# zdb -l /dev/da1p2

Label 0 - 4 LISTED, along with guid's, hostnames, etc.

Code:

[root@freenas] ~# zpool import
no pools available to import

Hmm, lets try:

Code:

[root@freenas] ~# zpool import -D
   pool: test
     id: 3708910445546102099
  state: ONLINE (DESTROYED)
 action: The pool can be imported using its name or numeric identifier.
 config:

        test                                          ONLINE
          gptid/542a2482-bc4d-11e2-bb40-000c2935fb99  ONLINE

Ha! There it is.


Conservative approach:

Code:

[root@freenas] ~# zpool import -D -R /mnt -o rdonly=on 3708910445546102099
[root@freenas] ~# df -h
Filesystem             Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs1a    926M    381M    470M    45%    /
devfs                  1.0k    1.0k      0B   100%    /dev
/dev/md0               4.6M    3.2M      1M    76%    /etc
/dev/md1               823k    2.0k    756k     0%    /mnt
/dev/md2               149M    7.7M    129M     6%    /var
/dev/ufs/FreeNASs4      19M    540k     17M     3%    /data
test                   3.9G    1.0G    2.9G    26%    /mnt/test
[root@freenas] ~# ls -la /mnt/test
total 1049106
drwxr-xr-x  2 root  wheel           3 May 13 18:36 ./
drwxrwxr-x  4 root  wheel         512 May 13 21:22 ../
-rw-r--r--  1 root  wheel  1073741824 May 13 18:36 testfile

From there, either recover data, or go for it and mount it read write for the freenas GUI.

Code:

[root@freenas] ~# zpool export test
[root@freenas] ~# zpool import -D -R /mnt 3708910445546102099
[root@freenas] ~# zpool export test
[root@freenas] ~# zpool import
   pool: test
     id: 3708910445546102099
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        test                                          ONLINE
          gptid/542a2482-bc4d-11e2-bb40-000c2935fb99  ONLINE

Now it's ready for an auto-import in the gui.

I wouldn't trust the pool at this point though. It's possible it's been damaged. I'd back everything up and recreate. A scrub may help. In this case, the pool existed on a single /dev device, it being a hardware raid array. So backup / recreate on non hardware raid would be suggested.

Moral of the story? Don't click "mark disks as new, destroys data!" unless you mean it.

 

[cyberjock]

Pretty impressive. Still, makes me wonder why the other thread didn't work.. hmm!

Glad the OP got his data back though.

 

[paleoN]

I think the 'mark disks as new' while exporting does the equivalent of a 'quick erase' in the gui. Which I think erases a portion of the beginning and end of the drive. This overwrites both locations that store the GPT. It may overwrite more than just the gpt areas though.

Quick erase does. For ZFS drives 'Mark the disks as new (destroy data)' does a zpool destroy and wipes only the GPT info. Which I guess you could consider a bug or the GUI being misleading at least. Still I prefer the current behavior.

Label 0 - 4 LISTED, along with guid's, hostnames, etc.

Nit, Label 0-3.

Conservative approach:

Code:

[root@freenas] ~# zpool import -D -R /mnt -o rdonly=on 3708910445546102099

You would need the -D to see destroyed pools. I believe a destroy is essentially an export with the addition of the labels updated as a destroyed pool. As long as you haven't started scribbling on the drive(s) with something else I don't see a particular need to import -o rdonly or recreate the pool.

Anyway, nice work titan_rw.

Pretty impressive. Still, makes me wonder why the other thread didn't work.. hmm!

The one thread I remember, the person recreated a new pool with the same name. Which overwrote all the ZFS labels and destroyed their original pool.

 

[paleoN]

Hardware array with 5x 400GB SATA disks.
Configured as one RAID 0 array (2TB)

Now that you have your data back it's an excellent time to backup and destroy this rather foolish pool configuration. One that has 5x greater chance of failure than a single drive pool. At least do hardware RAID 5 or better yet raidz1 & gain ZFS self-healing.

 

[titan_rw]

Nit, Label 0-3.

Whoops. Brain fart there. I knew there was 4 labels, and forgot to apply the base 0 offset to the total.