SOLVED GUI upgrade failed -> updated via CD -> encrypted pool unavailable

[tak21]

My problem: my encrypted RAID-Z2 pool is not found/accessable

My server: X9SCM-F, IBM ServeRAID M1015 in IT mode, 8x WD red 3TB

What I did:
- tried to upgrade to 9.3 beta from 9.2.1.8 via GUI which failed (reboot cycle)
- rebooted with 9.3 CD and did an upgrade install
- accounts / groups / passwords were like before - I did not check the pool right away because I
- realized an error: SCSI driver (v16) did not match SCSI bios version (v19)
- reflased SCSI bios to v16
- rebooted
- and now I realized that the pool is gone
I do have a lot error message in the console like:
Nov 11 17:07:58 mainsrv manage.py: [middleware.notifier:1381] Failed to geli detach gptid/e9b9082b-fea5-11e3-a471-002590d6593b: geli: No such device: gptid/e9b9082b-fea5-11e3-a471-002590d6593b.

Is there any chance to get the pool online again?

(And call me stupid - I reorganized my pool 3 month ago and did not refresh my GELI key backups for this mashine...)

 

[tak21]

O.k. the needed key was luckely converted to /data/geli and with that I can kind of import the volume.
It will actually not work entirely via the GUI as the GUI claims that tank is already there. But it is creating the correct geli devices
e.g.:
Nov 11 19:47:51 mainsrv GEOM_ELI: Device gptid/e9b9082b-fea5-11e3-a471-002590d6593b.eli created.
Nov 11 19:47:51 mainsrv GEOM_ELI: Encryption: AES-XTS 128
Nov 11 19:47:51 mainsrv GEOM_ELI: Crypto: hardware

and I can bring the pool alive via CLI by

# zpool import -fR /mnt tank

(And yes, I am running a backup right now :))

The only thing I am missing is an option to reset the jail system as the jail storage is on the encrypted pool aswell and the jail system is not recognizing the manual pool mount.
If I would have that up and running I could backup one mysql database and that I would basically recreate the mashine.

 

[jgkolt]

I also lost my encrypted pool when i upgraded from 9.2.1.9 to 9.3 beta. I didn't see any way in the gui to bring it back (and i didnt see this thread until it was too late).

 

[cyberjock]

Count your blessings! Bet the fear of God was in you that your data was lost forever huh?

Anyway, you are extremely lucky to have your data. Mounting the pool from the CLI isn't going to work well. What you *might* be able to do is keep your config file. Later when you get your data backed up you can destroy the pool and recreate it from the WebGUI and assuming you use the same name, after restoring your jails you could import your config file again and it would all "just work".

 

[tak21]

@cyberjock: actually you are the reason, why a total loss would have been not that bad ... your clear words in your guide "forced" me to rethink my backup strategy and actually lead to a mirrored system which is always 2 weeks behind the main system - which I found sufficient in my home environment. And due to this accident a new system from older components was born which I now will permanently assign as testing system (no ECC, with an i3 cpu, with 4 500gb drives). And yes, I am verry happy now to have all data save again :)

• To your recommendation to save the configuration: do I do a backup via GUI or directly copy config files?
• Is there any possibility to prevent the error on reboot (manage.py: [middleware.notifier:xxxx] Failed to geli detach gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: geli: No such device: gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)? As the system is capable of creating everything if I manually import the volume my thought is, that if I could overcome this error I am fully up and running again(?/!)
  
• Actually I do not understand this error. Could someone give me a hint what my system is actually telling me?

 

[cyberjock]

You'll probably have to copy the files yourself. The WebGUI is going to be very confused because a pool that it doesn't think is mounted actually is. I'd do ZFS replication from the CLI (it's easy to do if you do a little googling).
As for the error, what it means, and such I don't know. I haven't tested 9.3 with an encrypted pool so I'm not sure what the errors mean.

 

[mark_]

I had the same problem. After a fresh install of 9.3 beta, I was able to import the volume without problems.

 

[tak21]

@Mark: is your volume still there after a reboot? I am able to import o.k. but the volume is gone after a reboot...

 

[mark_]

Hmm, I was sure that I rebooted after the installation. But after rebooting now, the volume is gone again and if I try to import it, I get the "You already have a volume with same name" message like before. The console then shows the gptid/.... devices created and a zpool import -fR /mnt <volume> brings it back to the webinterface.
I hope that this can be fixed without creating a new volume and copying back the data because that takes a very long time in my case.
I found out that locking the volume removes it from the webinterface and I have to webiface-import/cancel/shell-import again to get it back, so a full reboot isn't needed to cause the problem.

 

[tak21]

O.k. same problem here. Now I am looking forward to your solution ;)

 

[clinta]

If you go through the import steps to decrypt the disks, then after getting the error that "a volume of this name already exists", you will see an alert in the gui that the volume is in an unknown state. At this point the disks are decrypted but the pool isn't imporetd. You can run zpool import on the command line at which point the errors are cleared and the volume shows up in GUI just fine.

Check https://bugs.freenas.org/issues/6591 for more info. Basically, after the initial import the volume still exists in the freenas database, but it is't being found by the front end once it's detached.

 

[tak21]

Just to inform you, a bug is now open for this topic: https://bugs.freenas.org/issues/6649

 

[mark_]

The bug has been closed and after I updated today, my volume showed up automatically. So far it works normally. Thanks!

 

[hugovsky]

Just upgraded and my volume also showed fine. Seems resolved.