How to Recover Data from NAS with TrueNAS Core?

[Welsh_Tony]

Though RAID-Z employs a data distribution scheme similar to the traditional RAID 5, it uses dynamic stripes instead of stripes with fixed size. The information about the width of each stripe is written in metadata. Given the dynamic nature of the system, data recovery from it is only possible when the metadata is intact. It must be analyzed properly to determine the correct geometry of RAID-Z.
Сan any of data recovery tools handle with this specific RAID type and capable of interpreting the metadata, reconstructing RAID-Z configurations and provides effective means for regaining of files lost from them?
I know that I can use Google and find a list of relevant programs in the search. But this is not what is needed, I want to get feedback from those who already had relevant experience.

 

[Etorix]

Only ZFS recovery tool I know of:

Klennet ZFS Recovery

Klennet ZFS Recovery software - recover data from damaged or deleted ZFS pools. Fully automatic pool layout, RAID level, and disk order detection.

www.klennet.com

Search 'Klennet' on this forum to find actual reports from its use. As far as I remember the few cases are positive, but one should do one's best to NOT be in the position of ever trying recovery.

 

[Constantin]

ZFS obviously has built in tools for said recovery should a drive fail and need replacement. It also has some tools to aid recovery if drive losses exceed parity limits. However, known-good backups are a much better option at that point than trying to extract what may be left.

I my limited experience re: alleged data loss, I found TrueNAS to be less than helpful re error messages in the GUI and the ZFS CLI was not much better.

For example, telling me that my pool was dead and needed to be destroyed (while omitting *why* TrueNAS/ZFS considered the pool dead) is a basic troubleshooting / diagnostic step that should be incorporated into the software. In my case, it was a loose electrical connection to multiple drives that made TrueNAS / ZFS freak out. Imagine all the time lost, had I followed the advice of the OS / Appliance!

No, you really have to rely on the knowledge of the folk here and the many helpful postings folk have left behind to help newbies like me to administer the TrueNAS through thick and thin.

 

[Arwen]

ZFS does something somewhat unique in the RAID world, redundant metadata. By default;

1. Data has 1 copy
2. Standard Metadata, (aka directory entries), has 2 copies
3. Critical Metadata has 3 copies

Now this is in addition to any RAID level that is applied. For example;

• A single disk would have Data=1, Metadata=2 and Critical Metadata=3
• A simple 2 way Mirror would have the above line on EACH disk. Thus, Data=2 copies, Metadata=4 copies and Critical Metadata=6 copies
• RAID-Zx would have the copies spread out over multiple disks. So, each vDev would have 1 Data copy, Metadata would have 2 copies, but not on the same disk, and Critical Metadata would have 3 copies, also not on the same disk.

I don't quite know how ZFS handles Metadata and Critical Metadata when their are multiple data vDevs. But, never the less, their are lots of Critical Metadata copies.


Some of this can be controlled by these ZFS dataset attributes:
copies redundant_metadata
See manual page for details if interested.

I have personally experienced some of this. I have a non-redundant ZFS pool for my miniature media server, (because I have good backups and the PC is too tiny for redundant storage). Every now and then I lost a media file, generally video because they were much larger than music MP3s / M4As.

Once I saw an error but no file was listed and ZFS said their were no permanent errors. I scratched my head for weeks over this. Then I realized that it was redundant Metadata that was in error for one copy. And ZFS was able to read the second copy, fix the bad copy and supply the application the good copy.

 

[Welsh_Tony]

So, as far as I understand from your answers, data recovery from TrueNAS Core and/or RAID-z is actually the same as data recovery from ZFS? Are the features the same?
And accordingly, should I look at the appropriate data recovery tools? (I mean, precisely the tools for data recovery from the ZFS.)

 

[Arwen]

Yes, TrueNAS, (Core or SCALE), only uses ZFS for data volumes.

In general, with proper hardware, (aka not hardware RAID), ZFS is incredibly robust. ZFS was designed to be always consistent on disk, due to COW, Copy On Write. Meaning nothing is over-written, new data & metadata is written to free space. Whence that is complete, a final table entry in the Uber blocks is updated activating the new writes. Until that new Uber block is written, the new data or metadata does not "exist". Thus, their is no "fsck", file system check utility, (other than a ZFS scrub which is done on-line).

ZFS writes like this, in this SPECIFIC order:

1. Data to free space
2. Metadata to free space
3. Critical Metadata to the oldest Uber block, in the Uber block tables

Uber blocks, if I understand them correctly, are final pointers to the state of the pool. Only the most current is more or less guaranteed to be "perfect". But, most of the time you can roll back a transaction or 2 without too much worry.

As for data recovery tools for ZFS, they are few and far between. ZFS pool structure is complex enough and under quite active development, any data recovery tool would have be kept up to date for new features added. Even if it was something simple like a new compression algorithm.

So the only ZFS recovery tool I know of, is the previously mentioned Klennet.

 

[Welsh_Tony]

Understand. Thanks for the detailed answer.
However, many developers of data recovery software offer their own tools for restoring the ZFS. Or, at least, emphasize the support of this file system (for example: Top Tools to Recover Data from ZFS Drives).
Are they inferior to the Klennet?

 

[Constantin]

Again, why put yourself in the position to need said services if you have a viable backup strategy that includes off-site backup?

Snapshots, scrubs, SMART tests, etc exist for a reason and should be used in addition to ZFS to minimize the chances of data loss. if something catastrophic happens, then there are always multiple backups to go back to.

Data recovery tools should always be an absolute last resort if data integrity is important (which is one main reason to choose ZFS in the first place).

 

[Redcoat]

However, many developers of data recovery software offer their own tools for restoring the ZFS. Or, at least, emphasize the support of this file system (for example: Top Tools to Recover Data from ZFS Drives).
Are they inferior to the Klennet?

I do not know the answwer to your question. Neither can I meet the test of " I want to get feedback from those who already had relevant experience." as you stated in your first post.

What I do know is that I have read posts speaking of positive recovery with Klennet, one relatively recently: https://www.truenas.com/community/threads/data-recovery-in-failed-drive-scenario.113936/

 

[Welsh_Tony]

Understand. Thank you all for your feedback