SOLVED ZPOOL IMPORT FAIL

[loh]

Good morning, This is the first time I've been in this situation with my pool and I can't find a solution. I am on Truenas-13.0-U5.3. My pool has gone offline and since then I can't import it back into Truenas. I tried with the GUI and I got I/O and python errors. Here is the information I can give. Thanks for your help.

zpool import
pool: DATA
id: 3527424950673752672
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

DATA ONLINE
raidz1-0 ONLINE
gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26 ONLINE
gptid/e26c5bd0-19a3-11e9-aa72-7085c2b0131c ONLINE
spare-2 ONLINE
gptid/e37f8645-19a3-11e9-aa72-7085c2b0131c ONLINE
gptid/172f542b-2149-11ed-b0a9-08606e691d26 ONLINE
spares
gptid/172f542b-2149-11ed-b0a9-08606e691d26

zpool import DATA
cannot import 'DATA': I/O error
Destroy and re-create the pool from
a backup source.

 

[loh]

I tried this command, but it doesn't work and I don't know how to find the disk(s) that are unavailable.

zpool import DATA -f -F -m -X
cannot import 'DATA': one or more devices is currently unavailable

 

[loh]

I tried with ZDB to find the latest TGX and import with it but I always have the same result.
Please help me.
How can I find out which disk(s) is causing the problem?

ZFS_DBGMSG(zdb) START:
spa.c:6109:spa_import(): spa_import: importing DATA
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26': best uberblock found for spa DATA. txg
spa_misc.c:419:spa_load_note(): spa_load(DATA, config untrusted): using uberblock with txg=28114827
spa.c:8382:spa_async_request(): spa=DATA async request task=2048
spa_misc.c:419:spa_load_note(): spa_load(DATA, config trusted): LOADED
spa.c:8382:spa_async_request(): spa=DATA async request task=32
ZFS_DBGMSG(zdb) END

zpool import -d /dev/gptid/6a7380b0-0d8c-11ec-aa1f-08606e691d26 -T 28114827 DATA
cannot import 'DATA': one or more devices is currently unavailable

 

[NugentS]

Please read Forum Rules - conveniently placed at the top of every page

 

[Whattteva]

Post your hardware, how the disks are connected and the output of geom disk list in code tags.

 

[loh]

Thank you for your reply.
My hardware is:
Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
Motherboard ASUS P8z77-V
16 Go DDR3 no ECC
ada0 -> zpool JAIL vdev JAIL (1disk)
ada1 -> Truenas
ada2,3,4,5 -> zpool DATA vdev DATA (4disk)

geom disk list
Geom name: ada1
Providers:
1. Name: ada1
Mediasize: 64022175232 (60G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
descr: C300-CTFDDAC064MAG
lunid: 500a075102fbfbce
ident: 00000000102602FBFBCE
rotationrate: 0
fwsectors: 63
fwheads: 16

Geom name: ada4
Providers:
1. Name: ada4
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: WDC WD40EFZX-68AWUN0
lunid: 50014ee268f1e044
ident: WD-WXC2D90LRK2C
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada5
Providers:
1. Name: ada5
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: WDC WD40EFZX-68AWUN0
lunid: 50014ee268f0b15b
ident: WD-WXC2D90ADU5Z
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada0
Providers:
1. Name: ada0
Mediasize: 128035676160 (119G)
Sectorsize: 512
Mode: r2w2e5
descr: TS128GSSD370S
ident: C392365620
rotationrate: 0
fwsectors: 63
fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: ST4000DM004-2CV104
lunid: 5000c500be0321c2
ident: WFN0NCZJ
rotationrate: 5425
fwsectors: 63
fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: ST4000DM004-2CV104
lunid: 5000c500be032a17
ident: WFN0ND1N
rotationrate: 5425
fwsectors: 63
fwheads: 16

 

[Davvo]

Can you post the output of camcontrol devlist as well?
There seem to be missing drives.

 

[loh]

Thank you for your reply.

camcontrol devlist
<TS128GSSD370S N1114H> at scbus2 target 0 lun 0 (ada0,pass0)
<C300-CTFDDAC064MAG 0007> at scbus3 target 0 lun 0 (ada1,pass1)
<ST4000DM004-2CV104 0001> at scbus4 target 0 lun 0 (ada2,pass2)
<ST4000DM004-2CV104 0001> at scbus5 target 0 lun 0 (ada3,pass3)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus6 target 0 lun 0 (ada4,pass4)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus7 target 0 lun 0 (ada5,pass5)
<AHCI SGPIO Enclosure 2.00 0001> at scbus8 target 0 lun 0 (ses0,pass6)

 

[loh]

I attach the result of 'zdb -e DATA'.
Thanks for your help
zdb -e DATA

 

[Davvo]

Code:

<TS128GSSD370S N1114H> at scbus2 target 0 lun 0 (ada0,pass0)
<C300-CTFDDAC064MAG 0007> at scbus3 target 0 lun 0 (ada1,pass1)
<ST4000DM004-2CV104 0001> at scbus4 target 0 lun 0 (ada2,pass2)
<ST4000DM004-2CV104 0001> at scbus5 target 0 lun 0 (ada3,pass3)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus6 target 0 lun 0 (ada4,pass4)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus7 target 0 lun 0 (ada5,pass5)
<AHCI SGPIO Enclosure 2.00 0001> at scbus8 target 0 lun 0 (ses0,pass6) 

Those drives are SMR, known to cause this kind of issues with ZFS. If you aren't able to mount the pool at least in read-only mode I'd say your data is lost.

 

[loh]

Code:

<TS128GSSD370S N1114H> at scbus2 target 0 lun 0 (ada0,pass0)
<C300-CTFDDAC064MAG 0007> at scbus3 target 0 lun 0 (ada1,pass1)
<ST4000DM004-2CV104 0001> at scbus4 target 0 lun 0 (ada2,pass2)
<ST4000DM004-2CV104 0001> at scbus5 target 0 lun 0 (ada3,pass3)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus6 target 0 lun 0 (ada4,pass4)
<WDC WD40EFZX-68AWUN0 81.00A81> at scbus7 target 0 lun 0 (ada5,pass5)
<AHCI SGPIO Enclosure 2.00 0001> at scbus8 target 0 lun 0 (ses0,pass6) 

Those drives are SMR, known to cause this kind of issues with ZFS. If you aren't able to mount the pool at least in read-only mode I'd say your data is lost.

Thank you for your reply. Can you give me the read-only mount options you recommend please.
- Otherwise, if I clone the disk will that change anything?
- Otherwise i'm in RAIDZ1 with spare I can't do anything with it?

 

[Davvo]

Before any of that, I just noticed that the state of the pool in the first post is online.
Can you please post the output of zpool status DATA between [CODE][/CODE] tags?

 

[loh]

Before any of that, I just noticed that the state of the pool in the first post is online.
Can you please post the output of zpool status DATA between [CODE][/CODE] tags?

Code:

zpool status DATA
cannot open 'DATA': no such pool

 

[Davvo]

Running zpool import -F -n DATA will tell you if it's possible to forcefully (with potential data loss) recover the pool; zpool import -F -X DATA is the more extreme way of recovery in case the simple zpool import -F DATA doesn't work.

In order to import the pool in read-only mode, run zpool import -F -o readonly=on DATA; you can use the -X here as well.

Cloning drives won't help, because the system expects drive number (examples) aa1, aa2, and aa3 to be in the pool: swapping them will only result in the pool yelling at you.
You can't manually do anything with that spare before you succesfully import the pool.

If anyone else has a better understanding of the situation, please chime in; I would prepare to restore from backups.

P.S.: Did you try power cyclying the system, didn't you?

 

[loh]

Thank you for your comprehensive response.
Yes i have power cyclying the system.

Code:

zpool import -F -n DATA

returns nothing. Which would mean that the command worked. but, always returns ->.

Code:

zpool status DATA
cannot open 'DATA': no such pool

zpool import -F DATA
cannot import 'DATA': I/O error
Destroy and re-create the pool from a backup source.

And the same

zpool import -F -o readonly=on DATA
cannot import 'DATA': I/O error
Destroy and re-create the pool from

with -X option

cannot import 'DATA': one or more devices is currently unavailable

thank you for your investment. the data on this pool is very important for me

 

[Davvo]

Well... I don't have anything else to give you. The pool is missing one or more devices, so unless moved one or more I don't think this is recoverable. If you don't have a backup you could try a recovery service, but they are bloody expensive for ZFS.

Maybe more experienced users can help you more.

 

[loh]

Thank you very much for your time in responding to me. Do you have a name for the recovery service recommended to me? If anyone else has an idea, thanks in advance.

 

[Davvo]

The only thing that comes to my mind is the following:

Klennet ZFS Recovery

Klennet ZFS Recovery software - recover data from damaged or deleted ZFS pools. Fully automatic pool layout, RAID level, and disk order detection.

www.klennet.com

 

[loh]

Thanks again.
Indeed this application is expensive.
If anyone has other ideas, thanks in advance.

 

[HoneyBadger]

Thanks again.
Indeed this application is expensive.
If anyone has other ideas, thanks in advance.

Have you attempted to import the pool at an older transaction group using the -T flag, as you mentioned in your third post? The latest txg is not valid, so working your way backwards may result in an older uberblock being usable.

Did you experience a power outage or component failure? SMR drives are known to cause problems, but the Seagate ones have more commonly shown up as slow performance versus the WD Red IDNF issue - but in extreme cases SMR drives can be marked as faulted due to long I/O times, and in a RAIDZ1 two drives faulting would take the pool offline.