Help me recover my data - I cannot import my pool

[zegaroid]

Hello to all,

Please help! I want to recover my data.

My Device:
OS Version:TrueNAS-SCALE-23.10.0.1
Model:Intel(R) Pentium(R) CPU G870 @ 3.10GHz
Memory:12 GiB

pool: pool
id: 6104381456958275430
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

pool DEGRADED
raidz1-0 DEGRADED
2db3de83-aa9a-4ca4-b841-f091acebed9b UNAVAIL
a6a75426-21ba-4618-9f87-6f9030b00a60 ONLINE
199669b7-44b5-433c-95b0-398a61647368 ONLINE

Let me tell you first the history of my problem, This is my first truenas setup and i was very excited until i run into a problem.

My setup been running for months until one day, i heard this clicking sound so i have checked the console and see a degraded disk, Im planning to buy a replacement but after 1 day, i noticed the network started to run slow when im copying some file. Then its started to freeze. So i turned it off till get some replacement.

After 2 days, i open and suddenly i cannot access my data, im not sure what's the real message or error there but it says, There are new disk available to add to pool. I searched the forum and i tried to export disconnect (without checking the delete data). I remove the faulty disk because it freezes and it's not readable anyway.

So i tried to import back my pool without the faulty disk:

zpool import -f pool

cannot import 'pool': insufficient replicas
Destroy and re-create the pool from
a backup source.

I can import in read-only but there are some errors

zpool import -o readonly=on pool
cannot mount 'pool/plex': Input/output error
cannot mount 'pool/dev': Input/output error
cannot mount 'pool/backup': Input/output error
cannot mount 'pool/ix-applications/catalogs': Input/output error
cannot mount 'pool/ix-applications/k3s': Input/output error
cannot mount 'pool/ix-applications/releases/plex/volumes/ix_volumes/ix-plex_transcode': Input/output error
Import was successful, but unable to mount some datasets#

Is there a way i can recover my data? i only have around 700GB of data. Its ok i if i loose my setup as long as i get my data back.

Thank you in advance for helping me.

 

[zegaroid]

With the above situation, is there other way to mount in read mode only?

Is it possible to replace the faulty disk even the pool is not imported due to some issue i am encountering?

I do have also snapshots backup but i cannot restore them because the pool is not imported yet.

 

[Arwen]

With the above situation, is there other way to mount in read mode only?
...

It appears you have problems on at least 2 disks, which for a RAID-Z1 is generally fatal.

Are you sure you removed the faulty disk?
And not one of the working disks?

...
Is it possible to replace the faulty disk even the pool is not imported due to some issue i am encountering?
...

No. ZFS does not allow off-line actions, like disk replacements. ZFS was specifically designed for all actions to be on-line so as to increase up time.

If you can't import the pool properly, then their is not enough redundancy to re-create missing disks in a vDev.

I do have also snapshots backup but i cannot restore them because the pool is not imported yet.

Snapshots are not backups. They exist in the pool, so any problems in the pool, affect snapshots just as much.

That said, snapshots can be used to recover deleted files, or create backups on other pools.

 

[zegaroid]

It appears you have problems on at least 2 disks, which for a RAID-Z1 is generally fatal.

Im sure that my setup are only with 3 disks. From the info above using zpool import there is only 1 unavailable and 2 are online. so im still safe from data disasters if only 1 fail right? May i know how did you spot that there are 2 disks failed?

Are you sure you removed the faulty disk?
And not one of the working disks?

I think im sure, because when i plug back in the other disk it wont show or not detectable, so i assume that's the faulty one.

Thank you for your response. Appreciated.

 

[zegaroid]

On the other hand, i do have another options to try out, but im not sure if this would work.

Before all this happened, I did have replaced 1 disk because of bad sector or degraded notice. It was a success replacement and i still keep that faulty disk which i think its still readable.

I do have a config back up before the replacement of the faulty disk. So my question, if i put back the old disk using the old config backup will it still work?

 

[Arwen]

The following indicates that something is seriously wrong with the pool;

Code:

# zpool import -f pool
    cannot import 'pool': insufficient replicas
    Destroy and re-create the pool from
    a backup source. 

Normally you can import a pool with bad disk(s) as long as the level of redundancy is available. Meaning for RAID-Z1, if you loose just one disk, the pool should still be importable. It is possible that you have bad block(s) on another disk causing the problem. If that is the case, it is why we don't recommend RAID-Z1 with larger disks. Too much risk.

Try this;
zpool import -fF pool
But, based on the errors you got when you imported read/only, I don't have much hope.

No, an old replaced disk won't help. A ZFS pool always moves forward in it's internal data structures. ZFS can roll back a few changes, but almost certainly not to a replaced disk.

Configuration backups do not contain the data from the pool. Configuration backups contain the information about the shares, the network configuration, schedules for snapshots, SMART tests, scrubs, and plenty of other thing. Basically, a configuration backup is solely for loss of the BOOT pool. With a configuration backup, you can re-create the boot pool exactly, assuming your data pool(s) are intact.

 

[zegaroid]

zpool import -fF pool

Still the same result.

But, I did a little experiment.

Just for FYI, my disks capacity are as follows:
2x of 1TB and 1x of 2TB.

i tried to plug in back the faulty disk (1TB) alone without the other 2 disks and surprisingly can read without problem.

Then i insert the other disk one at a time, first is the other 1TB which is working fine both of them are ONLINE.

When trying to insert the other disk which is the 2TB, i got stuck at boot error:

This error happens only when inserting the 2TB disk.

I also tried to plugged 2 TB disk alone without the other disk, which is working fine.

So, i wonder if what is happening here, i thought its was the 1TB disk that is causing some trouble because it's not readable by the time i have checked.

With 2x 1TB setup, i tried to run import pool again and i get different error now,

cannot import 'pool': I/O error
Destroy and re-create the pool from
a backup source.

Any recommendation or guide?. Somehow, until now i still have a strong feeling my data are still intact. Ha ha

 

[Arwen]

Sorry, I have no further suggestions. Perhaps someone else in the forum will.

 

[zegaroid]

Sorry, I have no further suggestions. Perhaps someone else in the forum will.

I see, Thank you for your help. Appreciated much.

 

[MrGuvernment]

Another tough lesson, this is why you ALWAYS have backups of your data, proper 3-2-1 rule, or at minimum your important data stored on a separate device / system / medium.

People need to stop jumping into installing TrueNAS with out understanding what it is for and the risks if things go wrong....

 

[zegaroid]

lsblk

I tried to swapping drives and use only 2 disks 2 at a time and tried to force import using

zpool import -fF pool

It's trying to import but failed in the end after a reboot, probably detecting a lot of errors.

grep "error" /var/log/messages

I wonder, if there is a way to skip blocks that has errors on disk.

 

[zegaroid]

Another tough lesson, this is why you ALWAYS have backups of your data, proper 3-2-1 rule, or at minimum your important data stored on a separate device / system / medium.

People need to stop jumping into installing TrueNAS with out understanding what it is for and the risks if things go wrong....

Yes i get that, Will do next time.

Im already here, so im doing it to know more and explore truenas!