ZFS-8000-8A after failed disk replace procedure

[warlord9]

Hi everybody! This is short sad story: About five years ago i have built zfs pool type strip(named DataBackup) with 4 different hdd disks (mixed WD red and Seagate, 1TB and 3x 3TB ) on my home Truenas server (gagabyte GA-78lmt-s2+amd athlon x2, 16GB RAM) and recently i have gotten mistake messages about crc check summs. Also one of WD red showed mistakes on write operations. I updated my system up to Truenas 13 U3 and started replace procedure immediately (oh if someone could tell me that wd red can't take it well), in all system rebooted in 52% of replacing. Now my pool shows status online but with mistake ZFS-8000-8A and it is writting : "One or more devices has experienced an error resulting in data corruption. Applications may be affected." (sometimes status switching to unhealthy). Unfortunately later i have formatted that wd red from which replacing was started. And ofcource there are no backups.
So my question: is it possible to read and copy any data which was hosted on three other disks?

 

[warlord9]

upd: i have tried recommendations from this and that topics but unsuccessful.
my zpool status:

Show : output

root@us:/home/sasha9# zpool status -v DataBackup
pool: DataBackup
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub repaired 0B in 0h1m with 136 errors on Tue Dec 13 16:23:51 2022
config:

NAME STATE READ WRITE CKSUM
DataBackup ONLINE 0 0 2
sdf ONLINE 0 0 0
sdb2 ONLINE 0 0 8
sdc2 ONLINE 0 0 0
sda2 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

<metadata>:<0xa2>
DataBackup:<0x0>
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/irssi.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/mosquitto.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/netdata.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/info/exclude
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/pull_request_template.md
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/quasselcore.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/hooks/pre-receive.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/sabnzbd.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/hooks/pre-rebase.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/sickchill.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/hooks/update.sample
/DataBackup/iocage/defaults_backup.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/unificontroller.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/hooks/pre-push.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/hooks/pre-applypatch.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/xmrig.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/zrepl.json
/DataBackup/iocage/defaults.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/config
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/2a/ab7dd87efd36db286ef76183ca3c361f6b59dd
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/b2/898d9b5c5a8f15d941d8b051f9646adb1ef0d5
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/04/740a8d4b5ec51a20a0e27d62093f4b2215b55f
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/pack/pack-4875273d8bffc3c48dcea1bdd3ae072bd1fc87e9.pack
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/info/exclude
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/a6/fa87916b4438cc309e3d4448770dbe06218def
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/pre-receive.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/post-update.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/logs/refs/remotes/origin/13.1-RELEASE
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/prepare-commit-msg.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/pre-rebase.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/update.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/commit-msg.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/logs/HEAD
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/applypatch-msg.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/pre-push.sample
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/logs/refs/heads/master
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/pre-applypatch.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/fsmonitor-watchman.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/hooks/pre-commit.sample
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/objects/pack/pack-4b7002dd563d510658af970946c45ebe64e207e2.pack
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/objects/pack/pack-4b7002dd563d510658af970946c45ebe64e207e2.idx
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/packed-refs
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/logs/HEAD
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/logs/refs/heads/master
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/asigra.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/calibre-web.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/channels-dvr.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/bacula-server.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/bru-server.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/dnsmasq.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/crashplan.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/deluge.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/emby-server-png-128.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/gitlab.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/homebridge.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/gitlab.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/madsonic.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/nextcloud.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/openvpn.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/heimdall-dashboard.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/redmine.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/rtorrent-flood.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/i2p.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/subsonic.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/unificontroller.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/mosquitto.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/netdata.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/template/settings.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/quasselcore.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/README.md
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/rtorrent-flood.png
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/sonarr.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/couchpotato.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/clamav.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/iconik.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/mineos.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/qbittorrent.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/tautulli.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/icons/zoneminder.png
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/jenkins-lts.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/jenkins.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/mineos.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/plexmediaserver-plexpass.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/plexmediaserver.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/qbittorrent.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/tautulli.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/transmission.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/zoneminder.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/logs/refs/heads/11.3-RELEASE
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/config
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/a7/1bb40fe7821685b8b232b8e02e0416560efab7
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/c3/99d067bad04d64e213341d76709d52e5f93c1a
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/89/245fdec730c8b2032da40f5520f7ff640ed068
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/26/43016e23032a5f0631358432d0fb9cd95d4140
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/68/c0e4b2397b3e99f1d3598d5b10bbfdedb336a1
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/0e/970a7a153cb89ee22fcc9ec8b9c6b80e970de1
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/e7/e14b97876e4a731f78572833cdb60bda3db093
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/dc/e756f02f811d610d9a01ade71de1918f239af1
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/template/overlay/usr/local/bin/quasselget
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/91/7f2742d2a2fd183ce6fe63cb17d7e3536d5c4e
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/template/overlay/usr/local/bin/quasselset
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/4a/e7b2ce84fdb8563d9cc561eb01072d67e6f138
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/e3/b8e2e1f9873956bf2950f8d1c463a8931518da
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/5f/de13d695f76b5b38ad136fb8f7854cb8a2b4ac
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/03/6b8fd87b166b59275fd2b38a16135ba5c9a02a
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/b2/a34e8f5ce0f58e122fe18e50b844f5da53455e
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/f4/8873737746b612f2928e239dc97501c4edf096
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/8e/140f66bd72e6d5d5db4b2f335da030350be56f
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/54/3a34e0fe8e81d27b3625c2d55fff8b60df2b0b
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/6d/0fb220321761e1b73fe75fe84a9bf1d72e2e5c
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/0c/4889812ad7da680e4684371c87892ab4701f13
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.github/workflows/validateindex.yml
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/INDEX
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/calibre-web.json
/DataBackup/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/FETCH_HEAD
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/deluge-pip.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/dnsmasq.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/emby-server-stable.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/famp.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/gogs.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/grafana.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/hoobs.json
/DataBackup/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/i2p.json

 

[NugentS]

Hope you have a backup, as you have (and had) no parity

 

[warlord9]

Hope you have a backup, as you have (and had) no parity

actually there are no chances?

 

[jgreco]

actually there are no chances?

No, because lots of your data is still sitting there on the failed hard drive.

 

[NugentS]

Given the "actually there are no chances?" - I guess no seperate backup

As @jgreco says there is no chance - for those files.
Make a backup of anything important on the pool to a seperate disk.
Then trash that pool, removing sdb2 and replacing it then recreate the pool with at a minimum a Z1 Pool so you have some parity

 

[warlord9]

Given the "actually there are no chances?" - I guess no seperate backup

As @jgreco says there is no chance - for those files.
Make a backup of anything important on the pool to a seperate disk.
Then trash that pool, removing sdb2 and replacing it then recreate the pool with at a minimum a Z1 Pool so you have some parity

Sorry, seems i didn't get it. You have written: "Make a backup of anything important on the pool to a seperate disk". How can i do this? I don't see my files.

 

[jgreco]

I think it would be unusual if there was anything salvageable. Sorry.

There's a reason for the scary warning TrueNAS gives you if you try to build a pool without redundancy. When you build something with what is effectively similar to RAID 0, not only do you NOT get the "R" (Redundant) in "RAID", but you also increase the chance of catastrophic failure taking your data from you.

 

[warlord9]

I think it would be unusual if there was anything salvageable. Sorry.

There's a reason for the scary warning TrueNAS gives you if you try to build a pool without redundancy. When you build something with what is effectively similar to RAID 0, not only do you NOT get the "R" (Redundant) in "RAID", but you also increase the chance of catastrophic failure taking your data from you.

One of disks was add later then three first. So my hope, may be part of data saved on them.

 

[warlord9]

Maybe we have any command which makes pool available force?

 

[jgreco]

One of disks was add later then three first. So my hope, may be part of data saved on them.

That might work if entire files were saved to individual disks. Unfortunately, that's not how it works. The blocks in your files are spread amongst the various vdevs in your pool.

 

[Etorix]

Unfortunately later i have formatted that wd red from which replacing was started. And ofcource there are no backups.

So you've witnessed someone having a heart attack, thoroughly dismembered the corpse and are now asking for medical help?

You may download the evaluation version of Klennet ZFS Recovery and see if it can recover anything, but in all likelihood there was a bit of every large file on every disk in the stripe and the data is lost.

 

[AlexGG]

That might work if entire files were saved to individual disks. Unfortunately, that's not how it works. The blocks in your files are spread amongst the various vdevs in your pool.

Actually, there is some merit in the argument.
If there were three drives, A, B, and C, the system was in use for a while, and then drive D was added, which (D) subsequently failed, I expect most of the data which was put on A, B, and C before D was added (and was never modified), most of that should be recoverable if there is a usable copy of the object set metadata. As the default is two copies of metadata, there is a pretty good chance some metadata survives. There is no automatic rebalancing, as far as I know, so the data stays where it was originally put.

How much practical utility is in it depends on how long the period was when the drive D was in use and how valuable is the old data.

As far as dismembering the corpse goes, that depends on the exact definition of "formatted that wd red".

If this means, like, "quick format to NTFS and left it at that", it is actually no big deal. There will be some damage, and maybe for some very bad luck, it is precisely on a pretty limited spot where the required part of metadata goes, but the chances of that are not that big.

If this means, "I formatted it and then filled it to the brim with whatever new files", then yes, that's a problem.

 

[jgreco]

How much practical utility is in it depends on how long the period was when the drive D was in use and how valuable is the old data.

This is still close to zero practical utility; it would need to be sent off to a ZFS data recovery specialist. Which also happens to be the same answer for the array with its failed disk, as most "failed" disks can be recovered too. The problem is that this is very costly, but if you've got money to spare and don't mind shipping off your disks, then, yes, there might be a path to recovery.