SOLVED GUI crashed during resilvering-cannot get it back

[brhersh]

Running 9.2.1.6. Replaced a hard drive for a storage upgrade in vol1. I was stupidly messing with services and suddenly everything in the GUI started to crash and inform me it couldn't display anything. I've restarted nginx and django via the command line and it hasn't allowed the GUI to come back up.

Here's what I know:
1. My zvols are encrypted. And I was able to mount/decrypt to start the resilver/re-mirror process.
2. The resilver process is still taking place according to zpool status.
3. Restarting of nginx+django haven't fixed the web interface.

Here's what I need answered:
1. The documentation warns about losing access to the data if the pool is not re-keyed prior to the next reboot.
What are the chances of this happening if I wait for the resiliver to complete and reboot?

2. If I can't get the GUI back up is there a way to perform the re-key of the pool on the command line?

3. Are there other things I can try to get the GUI back?

 

[cyberjock]

Oh boy...

1. The pool is rekeyed when you add a disk, so unless you can get the keyfile I think you're already on borrowed time. (Edit: There's a chance it's not rekeyed until you rekey it. But I'm not sure if/when the change was made, but you should obviously assume the worst.)
2. I have no doubt there is since the WebGUI simply does CLI stuff behind the scenes to get work done, but I don't know how. You'd have to dig into the code to figure out how it does it and reproduce it yourself.
3. Typically the best option is to reboot, but that's not a good option.

If I were you I'd *seriously* consider getting a backup of your data if you don't have a backup already. There's a good chance that things are going to go badly for you and I think you know that (hence the question on the forum). I just don't know how to fix it. Generally once things have started crashing that's a sign of bigger problems and the single best and first answer is to try a reboot.

Sorry, wish I had better advice.

 

[brhersh]

Thanks for the reply. It's even refusing new ssh connections. I may just have to reboot and bite the bullet. At this point it may not even be worth waiting for the resilver to complete.

 

[cyberjock]

One of two things will happen when you reboot and try to mount your pool..

1. The pool will remount with the key you have, but the drive that was resilvering won't be included in the mount and you'll have to resilver (definitely not a bad thing).
2. The pool will not remount at all and you'll be locked out forever.

In my case, when I was experimenting in 8.3.0 it was #1. But I thought it changed to #2 (hence the stern warnings in the manual... I wrote that section on replacing disks on encrypted pools). I can't remember everything perfectly though because it was more than a year ago and it was one of those "you should have a backup if you are doing encryption anyway" type of things.

 

[brhersh]

Well it rebooted and I was able to unlock it. I don't think it's mounted properly because the new drive is larger than the old. The resilver process continued where it left off (49%) and has finished. Rebooted after I did the re-key and re-passphrase, unlocked, same 'Error getting available space' message. I'll replace the other disk tomorrow and see what happens.

 

[brhersh]

Looks like all went well. Functioning on new drives and the system has survived a couple reboots. The only strange thing is that if I unlock vol1 (the group that I had to reboot on in the middle of resilvering) prior to unlocking vol2 I get a pretty lengthy error. Something about expecting a .system file on vol2. I'll have to look in to that later. For now marking this as resolved. Thanks for the help!

 

[danb35]

One of the tabs on the settings page in the Web GUI is for the system dataset--you can choose which pool you want it to live on. That's where all your logs get stored, so it would be best if it's accessible before you try to do anything else with the server.