SOLVED ZPOOL IMPORT FAIL

[loh]

Thank you for your reply.
Yes that's what I did with the tgx 28114827

Code:

zpool import -d /dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26 -T 28114827 DATA

But I got the result

Code:

 cannot import 'DATA': one or more devices is currently unavailable 

I made the result of 'zdb -e DATA' available in a file called log.txt.<-link
It is from this result that I deduced the tgx to choose, but perhaps I was wrong?!
If anyone would like to look at the file and tell me if I chose the correct 'tgx'. What if my order has the right flags
Thanks for your help

 

[HoneyBadger]

Hi @loh

Can you provide the (smaller) output of zdb -eul -d /dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26 - this should dump a list of the most recent 32 uberblocks and their associated timestamps & txg's - you can try the simulated import with the zpool import -nT txg command (this may take a long time) and if it succeeds, remove the n to do it for real and "rewind" your pool to that point.

 

[Davvo]

I suggest to do what @HoneyBadger sugested in a tmux, or equivalent, session.

 

[loh]

Thank you for your reply.
I tried several TGX starting from the oldest to the most recent, but always with results depending on the option

zpool import -F -nT 28113080 3527424950673752672 (no result so the order is successfully !?)

zpool import -T 28113080 3527424950673752672
cannot import 'DATA': one or more devices is currently unavailable

zpool import -F -T 28113080 3527424950673752672
cannot import 'DATA': I/O error
Destroy and re-create the pool from
a backup source.

I noticed that the TGXs are not the same on the different disks. except on the first and fourt (I'm attaching the results of zdb -eul -d for all 4 disks.)

I will try to continue by going back to the older TGXs.

 

[HoneyBadger]

Your pool appears to have a spare disk in it, so one disk being "out of sync" with the others isn't entirely unexpected as it may have been sitting idle due to either being the spare or being replaced by the spare.

There are three txg's that are common across three disks (d1/d3/d4) and those are 28113905, 28114071, and 28113080 - since you've tried the last one above, try the other two with -FXT

 

[loh]

Thanks for your help.
I always get the same type of response from my pool with all the TGX and the different option.

root@freenas[~]# zpool import -FXT 28114071 DATA
cannot import 'DATA': one or more devices is currently unavailable

I found an interesting article: link
I ran this command which shows me my Dataset and which seems to patch my disks, but I'm not sure...

zdb -dep /dev -G DATA
Dataset mos [META], ID 0, cr_txg 4, 109M, 258 objects
Dataset DATA/.bhyve_containers [ZPL], ID 304, cr_txg 20277, 70.4M, 10 objects
Dataset DATA/DATA [ZPL], ID 661, cr_txg 27462, 4.69T, 2227037 objects
Dataset DATA/BACPROSN [ZPL], ID 97, cr_txg 1048639, 11.5G, 93268 objects
Dataset DATA [ZPL], ID 21, cr_txg 1, 117K, 10 objects
MOS object 265 (DSL dir clones) leaked
MOS object 288 (DSL dir clones) leaked
MOS object 327 (DSL dir clones) leaked
MOS object 331 (DSL dir clones) leaked
MOS object 365 (DSL dir clones) leaked
MOS object 425 (DSL dir clones) leaked
MOS object 429 (DSL dir clones) leaked
Verified large_blocks feature refcount of 0 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 4 is correct
Verified encryption feature refcount of 0 is correct
Verified project_quota feature refcount of 4 is correct
Verified redaction_bookmarks feature refcount of 0 is correct
Verified redacted_datasets feature refcount of 0 is correct
Verified bookmark_written feature refcount of 0 is correct
Verified livelist feature refcount of 0 is correct
Verified zstd_compress feature refcount of 0 is correct

ZFS_DBGMSG(zdb) START:
spa.c:6109:spa_import(): spa_import: importing DATA
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/ada4p2': best uberblock found for spa DATA. txg 28114827
spa_misc.c:419:spa_load_note(): spa_load(DATA, config untrusted): using uberblock with txg=28114827
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 6111600488197449739: path changed from '/dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26' to '/dev/ada4p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 5923385502359816919: path changed from '/dev/gptid/e26c5bd0-19a3-11e9-aa72-7085c2b0131c' to '/dev/ada3p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 12673765311600314262: path changed from '/dev/gptid/e37f8645-19a3-11e9-aa72-7085c2b0131c' to '/dev/ada2p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 11057893291964413131: path changed from '/dev/gptid/172f542b-2149-11ed-b0a9-08606e691d26' to '/dev/ada5p2'
spa.c:8382:spa_async_request(): spa=DATA async request task=2048
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADED
spa.c:8382:spa_async_request(): spa=DATA async request task=32
ZFS_DBGMSG(zdb) END

The problem is that the TGX tells me that it is good and that on disk 1 and the results are therefore the same.

zpool import -f -T 28114827 DATA
cannot import 'DATA': one or more devices is currently unavailable

If you can help me understand this report.
Thanks in advance.

 

[loh]

Your pool appears to have a spare disk in it, so one disk being "out of sync" with the others isn't entirely unexpected as it may have been sitting idle due to either being the spare or being replaced by the spare.

There are three txg's that are common across three disks (d1/d3/d4) and those are 28113905, 28114071, and 28113080 - since you've tried the last one above, try the other two with -FXT

I tested the three TGX and I still have the same result, do you have another idea?

zpool import -FXT 28113905 DATA
cannot import 'DATA': one or more devices is currently unavailable
zpool import -FXT 28114071 DATA
cannot import 'DATA': one or more devices is currently unavailable
zpool import -FXT 28113080 DATA
cannot import 'DATA': one or more devices is currently unavailable

If not, do you have any idea about the report?

zdb -dep /dev -G DATA
Dataset mos [META], ID 0, cr_txg 4, 109M, 258 objects
Dataset DATA/.bhyve_containers [ZPL], ID 304, cr_txg 20277, 70.4M, 10 objects
Dataset DATA/DATA [ZPL], ID 661, cr_txg 27462, 4.69T, 2227037 objects
Dataset DATA/BACPROSN [ZPL], ID 97, cr_txg 1048639, 11.5G, 93268 objects
Dataset DATA [ZPL], ID 21, cr_txg 1, 117K, 10 objects
MOS object 265 (DSL dir clones) leaked
MOS object 288 (DSL dir clones) leaked
MOS object 327 (DSL dir clones) leaked
MOS object 331 (DSL dir clones) leaked
MOS object 365 (DSL dir clones) leaked
MOS object 425 (DSL dir clones) leaked
MOS object 429 (DSL dir clones) leaked
Verified large_blocks feature refcount of 0 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 4 is correct
Verified encryption feature refcount of 0 is correct
Verified project_quota feature refcount of 4 is correct
Verified redaction_bookmarks feature refcount of 0 is correct
Verified redacted_datasets feature refcount of 0 is correct
Verified bookmark_written feature refcount of 0 is correct
Verified livelist feature refcount of 0 is correct
Verified zstd_compress feature refcount of 0 is correct

ZFS_DBGMSG(zdb) START:
spa.c:6109:spa_import(): spa_import: importing DATA
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/ada4p2': best uberblock found for spa DATA. txg 28114827
spa_misc.c:419:spa_load_note(): spa_load(DATA, config untrusted): using uberblock with txg=28114827
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 6111600488197449739: path changed from '/dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26' to '/dev/ada4p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 5923385502359816919: path changed from '/dev/gptid/e26c5bd0-19a3-11e9-aa72-7085c2b0131c' to '/dev/ada3p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 12673765311600314262: path changed from '/dev/gptid/e37f8645-19a3-11e9-aa72-7085c2b0131c' to '/dev/ada2p2'
vdev.c:2412:vdev_copy_path_impl(): vdev_copy_path: vdev 11057893291964413131: path changed from '/dev/gptid/172f542b-2149-11ed-b0a9-08606e691d26' to '/dev/ada5p2'
spa.c:8382:spa_async_request(): spa=DATA async request task=2048
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADED
spa.c:8382:spa_async_request(): spa=DATA async request task=32
ZFS_DBGMSG(zdb) END

Thank you in advance for any help that will allow me

 

[Davvo]

Consider letting the poor thing die in peace.

 

[loh]

Consider letting the poor thing die in peace.

I can't let him rest in peace. I want to recover as much data as possible :'(

 

[loh]

It's a holiday.
I finally managed to recover my data.
After a lot of experimentation looking for the right TXG and a stint on recovery software.
klenet recovery zfs is quite effective at scanning pools.

But ...
finally with 3 modifications of the import behavior it happened:

sysctl vfs.zfs.max_missing_tvds=1
sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0

zpool import -f -o readonly = on DATA

here I am with my imported pool, my data accessible.
It seems to me that I have lost nothing, or only marginally so.
I copy all my data and I will upgrade to truenas scale to have LXC.
Thank you all for your help.

 

[Davvo]

I copy all my data and I will upgrade to truenas scale to have LXC.

I don't think SCALE support those, but I might be wrong.

Anyway, congrats!

 

[loh]

Good morning. This will be my last post on this thread.
Just to say thank you once again for the time you spent on my problem.
I recovered everything without loss of data and everything and 100% operational again.
I will end with: never give up if it's important!! :)

 

[ukash]

I don't think SCALE support those, but I might be wrong.

Anyway, congrats!

I have a similar issue with my corrupted pool after a power cut. Can anyone advise how to use Linux equivelents of:
sysctl vfs.zfs.max_missing_tvds=1
sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0

On Scale? Or perhaps there is another way to skip the metadata verification? Importing a pool with TXG takes ages, as whole disks are scanned.