~75% of data disappeared

[ji-]

Hi. I've had a machine running FreeNAS 8.0 off of a USB stick for about a year. It has one 6 TB volume consisting of two 3 TB Seagate Barracuda drives.

Recently (~3-4 weeks), some data started dis- and reappearing when I browsed the volume over AFP from my (2) Macs. Data would reappear when I restarted FreeNAS. Then ~2 weeks ago, a huge chunk of data (~4.5 TB) disappeared. In this case, restarting hasn't helped. It went from having only ~0.1 to ~4.5 TB free.

The FreeNAS WebGUI lists ~4.5 TB as available. I guess that narrows it down to being a FreeNAS or hardware problem. There's also the possibility of user deletion (sudo rm -rf on the entire volume).
Sadly there are no ZFS snapshots to restore from.

I'm planning to make bit-identical images of the drives before I do anything else using BackTrack and ddrescue.
Then attempt to assess the damage and restore critical files from the images using whatever tools work.
I'll keep the original drives safe in case a cheaper solution appears, or send them to Kroll Ontrack as a last resort.

This is the first time I've had data loss on this scale, so I would appreciate any insight. Can anyone think of anything I'm overlooking, or recommend any software/utilities for the various steps?

 

[ProtoSD]

That sucks, and it definitely sounds like you have a good plan lined up. "rm -rf" has bitten most of us a some point or another.

As for ZFS recovery tools, there aren't really any official ones, and only a post 2 years old from someone that claims to have written something and hasn't been heard of since.

ddrescue is a great tool, but if you consider that your data is striped (unless you had a mirror), then I'm not sure any scraping tool will be able to get anything.

How often did you run scrubs?

Snapshots probably would have helped, they don't have to be done frequently like most people tend to do, just often enough to capture significant changes. Backups are a good idea too ;)

Good luck and post back what the outcome was so others can hopefully learn from your misfortune... :(

 

[ben]

What is the output of 'zpool status'? Do you run any other services between clients and the FreeNAS box? How are the drives connected?