RaidZ pool offline after usb failure

[ryandasilva]

So my usb boot drives failed. I bought a Sata ssd to rebuild /restore and after restoring from backup file my super important 3 disk raidz pool will not come back online and I have 2 other pools which are encrypted won’t unlock And are giving the error message “This geli-encrypted pool failed to decrypt.”

i’m assuming that’s because the restore process doesn’t recover the key files . I looked in the data directory and there’s no GELI files or even a directory for that matter. I think I have the GELI files Backed up I’m just not sure exactly how to restore them. But quite honestly those encrypted pools are less important than my non-encrypted three disk raidz pool. That unencrtoted 3 disk pool has very important personal data on it.

At first I actually attempted installing scale And to my surprise the unencrypted pool actually came back online no problem!

I was quite surprised and slightly amused that my super critical data came back online so easily in an entirely different OS, but for some stupid reason instead of backing it up immediately I decided to be smart and blow scale away.... then i installed core again because I couldn’t figure out why my encrypted pools were not coming online.

Since then I’ve learned a bit more about the encryption pools and I’m pretty certain the reason they’re not coming back online as those key files are not part of the main config backup and I hadn’t manually restored them. I think I know what to do with them somewhere… I believe they need to be uploaded to the data/GELI directory with the file name of (uuid).key . The issue I have is I don’t know what that UUid is. If anyone give me somewhere to look for that information it would be appreciated.

my main problem though is like I said the three disk raidz pool. It’s unencrypted. And it won’t come online this time around when restoring the config. The disk names may have changed in the system like the numbering of them like DA1 might be DA3 now. But other than that the discs are all there and they’re showing up in the OS so I don’t know why the pool is not coming online.

I even tried deleting the pool and re-importing it but it won’t find it. I did the non destructive delete.. or is it called disconnect? Anyways. The discs seem to be OK in health. It would be lovely if I can do a nice clean restore of the pool and bring it back up with minimal effort but at this point I’m ready do use any form of crude data Recovery necessary to get that data back. Even if it means mounting each of the three desks one at a time in recovery to it I’ll do that. I just don’t know where to start with this. And I would like to believe that with all the very intelligent people in this forum and all the hard work in this amazing product but I wouldn’t need to resort to some thing as crude as a recovery tool for disks that come online without any errors.

i’m going to try to restore the config again but assuming that nothing changes what do I do guys? I don’t know which log files or anything to even look in. Please let me know what I can do to help you help me here please. This data is very important to me. The moment it becomes available it will be backed up to two other locations. Lesson learned.

Thanks in advance . Stay safe. Cheers.

 

[sretalla]

what do you see if you do zpool import in the shell?

 

[ryandasilva]

pool: POOL_RAIDZ_12TB_HDD
id: 10731945887487471177
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

POOL_RAIDZ_12TB_HDD UNAVAIL missing device
raidz1-0 ONLINE
gptid/de6a2933-319d-11ea-bb72-001018efc630 ONLINE
gptid/e677d4c8-319d-11ea-bb72-001018efc630 ONLINE
gptid/ed9037e3-319d-11ea-bb72-001018efc630 ONLINE
logs
disk/by-partuuid/ac3c5127-21cf-11eb-b30c-001018efc630 UNAVAIL cannot open

 

[ryandasilva]

sorry if that's the incorrect way to post a log output.. if there's a better way to do it please tell me how.

so this is interesting.. the output of that command only references the very important pool... it doesn't even mention the 3 encrypted pools . which is fine.. we can talk about those after or on the side. this 12TB pool is right at my finger tips.. and I had actually suspected it had something to do with the log drive being removed. since my server was off-line so long I had the need for a 512GB SSD that was sitting in the machine idle.. I took it and it looks like ZFS doesn't like me very much for that. How can I force the pool to mount without the log disk? I question if I even needed that log disk. How would I go about bench marking if I need a log disk or not? IO performance benchmarking has never been my forte. I'm old school and still go by how fast in Mbps or Gpbs a file copies and consider that the performance metrics. I understand what IOPs are .. I just don't get them. Im a networking guy.. Mbps makes sense to me. IOPS does make sense... It just doesn't mean anything though . I Feel like I'm missing a nugget of info that will connect all that together and make me understand better.

 

[ryandasilva]

update: i went to the URL provided by the ZFS error message and read what it had to say. then I ran the import command with the -f and then the -m commands and finally got a zero response from my command which was a good thing in this case. here's the output of my command session:

root@echo[~]# zpool import
pool: POOL_RAIDZ_12TB_HDD
id: 10731945887487471177
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

POOL_RAIDZ_12TB_HDD UNAVAIL missing device
raidz1-0 ONLINE
gptid/de6a2933-319d-11ea-bb72-001018efc630 ONLINE
gptid/e677d4c8-319d-11ea-bb72-001018efc630 ONLINE
gptid/ed9037e3-319d-11ea-bb72-001018efc630 ONLINE
logs
disk/by-partuuid/ac3c5127-21cf-11eb-b30c-001018efc630 UNAVAIL cannot open
root@echo[~]# zpool import POOL_RAIDZ_12TB_HDD
cannot import 'POOL_RAIDZ_12TB_HDD': pool was previously in use from another system.
Last accessed by echo.cloudniner.solutions (hostid=409ba6c8) at Wed Jun 30 08:24:00 2021
The pool can be imported, use 'zpool import -f' to import the pool.
root@echo[~]# zpool import POOL_RAIDZ_12TB_HDD -f
The devices below are missing or corrupted, use '-m' to import the pool anyway:
disk/by-partuuid/ac3c5127-21cf-11eb-b30c-001018efc630 [log]

cannot import 'POOL_RAIDZ_12TB_HDD': one or more devices is currently unavailable
root@echo[~]# zpool import POOL_RAIDZ_12TB_HDD -f -m
root@echo[~]#

 

[sretalla]

OK, so your syntax is a little off:

Try it this way...

zpool import -f -m POOL_RAIDZ_12TB_HDD

You may or may not also need to add -F too.

 

[ryandasilva]

IVE NEVER BEEN SO HAPPY TO SEE THAT POOL SHOWING UP AS DEGRADED!!!! Yahoo! That pool has pictures and videos of my late wife and my kids early years that I don't have properly, fully backed up. i was in the process of organizing and ultimately having them backed up to the cloud when my pools shit the bed. I am going to make sure that they are backed up right now. Im almost afraid to spend another moment not backing that shit up to the cloud and ask about the the encrypted pools. I think I have the 3 key files .. i uploaded them to /data/geli after creating the geli folder but the filenames are similar to ENCRYPTEDPOOL_MIRROR_4TB_SSHD_BACKUP-GELI.KEY . i read somewhere that the filename needs to be UUID.KEY but I have no clue what UUID or where to get the uuid. Any help u can provide is greatly appreciated.

Ugh!! Im so relieved.

 

[ryandasilva]

OK, so your syntax is a little off:

Try it this way...

zpool import -f -m POOL_RAIDZ_12TB_HDD

You may or may not also need to add -F too.

my syntax was off but she understood me! :P

 

[sretalla]

I question if I even needed that log disk. How would I go about bench marking if I need a log disk or not?

If you're asking, you probably don't need it.

 

[ryandasilva]

If you're asking, you probably don't need it.

i , like many rookie storage admins, thought that we NEED ssd caching. I mean, in theory it sounds great. But it never worked out. RAM is better investment as the data still needs to go thru ram before it hits the cache disk is my understanding. i was under the impression though that it was smart and knew what to keep on the cache disk and if say a VMs boot files were on that pool and it kept being accessed the same, multiple times a day, every day, that it would operate of the cache disk . It doesnt seem to work that way. I Really dont know how it helps to be honest. Thats why id like to know how to properly bencnh mark it.

Any clues how to get my encrypted pools back online?