How to lose data with ZFS

[garm]

Step 1. Buy a house
Step 2. Start 2 or 3 major projects on the house with multiple contractors
Step 3. Have lots of work related stress
Step 4. Lose a drive in a mirror during a weekend
Step 5. Decide to burn in a drive on your production server as everything is getting ready for the big move
Step 6. Disconnect all data drives from the system so not to lose data by typo
Step 7. Assume you have disconnected the data drives
Step 8. Run badblocks on the good mirror drive

This is my first loss of data in over a decade sins starting to use ZFS.
Heed the gumption traps of Pirsig.. and keep backups.. mine are not complete.. so there are some painfull data loss..

 

[sretalla]

Sorry to hear that.

Thanks for sharing the story to help others to not repeat it.

 

[sretalla]

Perhaps there's another lesson in there... do we really need to run badblocks in destructive mode? (maybe it's faster like that, but is it worth the risk?)

 

[garm]

Perhaps there's another lesson in there... do we really need to run badblocks in destructive mode? (maybe it's faster like that, but is it worth the risk?)

Well, putting in a new drive in a degraded pool and finding it has its own issues is a horror in itself.
I always trippel check drive serial numbers and other clues to make sure I do stuff to the right drive, and usually I run a dedicated Linux boot env on my workstation with nothing but the drive when burning in.

In this case I rushed it between trips to the new house, I didn't check that I had the right drive after unplugging all other drives and I assumed that the one remaining drive on the systems was the new drive.

I would say that my biggest sin is the lack of proper backups.. I don't know yet what the damage is.. photos and Nextcloud storage is backed up and some of the other data might still be on the primary drives I backed up from, but most of those are old and laying in a box waiting to be destroyed.. and some stuff where one of a kind and only on this server.. those are now lost to entropy

 

[garm]

Perhaps there's another lesson in there... do we really need to run badblocks in destructive mode? (maybe it's faster like that, but is it worth the risk?)

ordered some drives to take this opportunity to migrate away from 2+2 mirror pool to a Z2 pool, while waiting Im properly burning in my original replacement drive and tried non-destructive.. that is painfully slow, some 90 hours to do a 4 TB drive, while the destructive run takes 15

 

[sretalla]

non-destructive.. that is painfully slow, some 90 hours to do a 4 TB drive, while the destructive run takes 15

OK, good to know... I expected maybe 2x, but not 6x. Still safer, but painfully slow.