Recover data from failed drive in RAID

Agret

Cadet
Joined
Aug 31, 2021
Messages
5
I have a FreeNAS setup I made for a client about 5yrs ago. Has been running great until a drive of his failed, he said when he boots up the system he can see the network share and the contents in explorer but when he tries to access files it will play a part of a video then the system freezes so he would do a hard power off and start it back up and it would run fine for awhile and then have the same freezing issue. He told me it has gotten worse and worse over time and now freezes basically the first file he tries to access.

I booted it up into Windows PE and checked the drives with Hard Disk Sentinel and one of them has failed and the other is reporting around 70% health. There was 4 disks in this RAID however one of them the SATA cable was bumped loose out of the board (did not have the locking clip) so i'm not sure how long that 4th disk has been offline. So in this setup we currently have 1 failed disk, 1 offline disk, 1 partially degraded disk and 1 that is operating normally.

After I unplugged the one that has failed and was causing the system to keep freezing I booted it up into FreeNAS and i'm not sure how to see the volume, it doesn't come up as a Windows share and I can't seem to see any file browser in the webui. The webui was not functional in chrome so I had to access it with IE. He was using an old build that was released in 2016 so I updated his USB to the latest 9.10 branch of FreeNAS using the webui updater which went through no problem.

I think before doing the recovery I will have to clone the degraded disk to a fresh drive using ddrescue. Should I connect up the offline disk as well? How do I get read-only mount of the system for file recovery and how to access each drive individually and copy just the files on each to a USB for backup before I wipe the array and rebuild it with new drives. I understand the advantage of ZFS is that you can access any files stored on the individual drive as it stripes full files rather than logical sectors like legacy RAID setups right?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I understand the advantage of ZFS is that you can access any files stored on the individual drive as it stripes full files rather than logical sectors like legacy RAID setups right?
It seems you don't understand ZFS (at least not that part of it).

I hope you didn't allow Windows to mess with the disks when you ran it with those drives connected.

How do I get read-only mount of the system for file recovery and how to access each drive individually and copy just the files on each to a USB for backup
You don't. That isn't possible.

I think before doing the recovery I will have to clone the degraded disk to a fresh drive using ddrescue. Should I connect up the offline disk as well?
Those are good options (although using dd from the FreeNAS shell would be best if you can).

You may be able to have the pool mount normally if the disks are OK. If you let Windows mess with them, you'll need a solution like Klennet to get the files back (if any will be recoverable).
 

Agret

Cadet
Joined
Aug 31, 2021
Messages
5
Windows doesn't mess with them, when it has unknown filesystem it leaves it alone. What's my process for trying to mount the pool?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
I understand the advantage of ZFS is that you can access any files stored on the individual drive as it stripes full files rather than logical sectors like legacy RAID setups right?
You badly misunderstand ZFS.
i'm not sure how to see the volume,
Log into the shell and run zpool status and zpool import; show the output of both.
 

Agret

Cadet
Joined
Aug 31, 2021
Messages
5
I have the failed drive disconnected from the system, have left the full health and degraded drive in there and also plugged in the drive that has an old data set that I don't know how long it has been uplugged so now have 3/4 drives in there. As expected it has now come up as a fault since one of the drives has old data it would be inconsistent and "corrupt", what's the command to resync this?

1630466260573.png
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
what's the command to resync this?
There is no command to resync this; unless you can bring the dead disk back to life, the pool is toast. RAIDZ1 can tolerate the loss of one disk without data loss. You've been running the pool with one disk disconnected, so there's been no redundancy. When the second disk failed, it was all over.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
What I see there on the screenshot is that your server is called Nas4Free...

If that's the case, maybe you'll be able to get more help on the Xigmanas forums (Nas4Free was a fork from FreeNAS a long time ago and it eventually renamed to Xigmanas).
 

Agret

Cadet
Joined
Aug 31, 2021
Messages
5
When the system was first installed it was running nas4free but that stopped working at one point so I reinstalled it with FreeNAS and imported the zpool, just kept the server name the same for his windows shares to keep working.
 
Joined
Jun 2, 2019
Messages
591
I'm guessing no backups. :oops:
 

Agret

Cadet
Joined
Aug 31, 2021
Messages
5
Yes unfortunately no backups have been made by the client, he is adamant that whatever path we take we do not lose any of his data (realistically I just want to get off what I can that exists across these 3 drives). What commands can I run to get the system to mount the pool in a read-only way? (should be some way to explore the available snapshots?) How do you identify which files are 100% available on the raid (matching the ZFS checksum) and which are missing due to stripes on the failed disk? I have never tried to recover from ZFS before and my usual RAID recovery tools do not recognize the filesystem as it is not EXT4. I am not very experienced with ZFS recovery and would be appreciated if anyone can send me to a guide on where to begin thanks.
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
What part of "the pool is toast" are you not understanding? There are no commands that will import a RAIDZ1 pool with two failed disks (and if a disk has been unplugged for any significant length of time, it's effectively a failed disk), whether read-only or not. I've heard that there are companies that do ZFS data recovery, though I don't know any names--that's likely your best bet.

There isn't much available in the way of ZFS data recovery tools, as ZFS isn't supposed to fail--but persistent neglect combined with failure to back up critical data is pretty near guaranteed to result in data loss.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I am not very experienced with ZFS recovery and would be appreciated if anyone can send me to a guide on where to begin thanks.
you'll need a solution like Klennet to get the files back (if any will be recoverable).

Your client will discover the price of not having a backup... it isn't free. You may get some, but certainly not all of his data back based on what you said so far, but Klennet can show you what is recoverable before asking you to pay to get it back.
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,110
A flat US$399 for any amount of recovered data isn't that bad.
What's going to be painful is explaining to the client that ZFS is no magic bullet, that operating his NAS in degraded state with the "reboot until it freezes again" method was a Very Bad Idea™, and that he will likely not get all of his data back, even after parting with the $399.
 
Joined
Jun 2, 2019
Messages
591
The "client" failed to implement a backup strategy and/or was ill advised, nor had a good ITSEC policy.

RAID is not a backup, RAID is resiliency.
 
Last edited:
Top