SOLVED Importing and unlocking encrypted pool on fresh FreeNAS 11.1 installation fails because it's too full??

[devnullius]

In follow-up of: https://www.ixsystems.com/community...lewareerror-at-storage-volume-2-unlock.75032/

It looked like decrypting my old pool was finally working... It took a long time to unlock instead of the sudden error message. That said, I ended up with the same a similar error now we fixed the Volume not Found problem and I'm still unable to unlock my 8x 3TB Z2 Volume… :(

PS: chmod a+r made the key permissions per request. The folder /data/geli was empty before copying and renaming my geli.key to /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key

My volume is completely full (I ignored it until I had some time to spare to configure a completely new backup strategy for my computers). It looks like that is giving a problem now?? ;-( See end of log please.

Code:

CRITICAL: March 25, 2019, 10:50 p.m. - The capacity for the volume 'ZFS_8x_3TB_RAIDz2_pool' is currently at 96%, while the recommended value is below 80%.

Code:

Environment:

Software Version: FreeNAS-11.1-U7 (b45bfcf29)
Request Method: POST
Request URL: http://192.168.0.197/storage/volume/2/unlock/?X-Progress-ID=3164769a-9aaf-4d57-87b3-311bebe92e0b


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volume_unlock
  1036.             form.done(volume=volume)
File "./freenasUI/storage/forms.py" in done
  2884.         _notifier.restart("system_datasets")
File "./freenasUI/middleware/notifier.py" in restart
  238.                 **kwargs,
File "./freenasUI/middleware/notifier.py" in restart
  238.                 **kwargs,
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  434.             raise ClientException(c.error, c.errno, c.trace, c.extra)

Exception Type: ClientException at /storage/volume/2/unlock/
Exception Value: [Errno 28] No space left on device: '/var/db/system/cores'

At least there is SOME progress... No? That said, it now feels like you guys are gonna tell me that a full encrypted volume cannot be decrypted and imported and all is lost ;-(

Post-edit: the sudden silence is killing me :) I'll check back in tomorrow morning :/ I hope there's a solution to the catch 22 here. I'll also investigate what I found on Google.

 

[PhiloEpisteme]

At least there is SOME progress... No? That said, it now feels like you guys are gonna tell me that a full encrypted volume cannot be decrypted and imported and all is lost ;-(

Possibly, possibly not. It sounds like you've got the correct geli key, which is great.

I assume then that the UI does not show your pool as decrypted under storage->pools?

If the UI won't import your pool you can try to do it manually, delete some files to get more space in your pool, and then try again using the UI.

What you'll want to do try is

1. If you have a backup rebuild your pool with more or larger drives and restore from backup. If you do not have a backup proceed to the next steps.
2. Decrypt all of your drives. This can be accomplished with the command I mentioned in the prior post against each drive in the pool. Use geom disk list to get the name of each of your 3TB disks, the one we want is associated with the Name: lable.
3. Run geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key on each of your 3TB disks.
4. Try to import the pool. I think based on your prior post that your pool's name is ZFS_8x_3TB_RAIDz2_pool. You can then try zpool import ZFS_8x_3TB_RAIDz2_pool[/IOCODE]. If that fails you might try to identify the pool's ID with [ICODE]zpool import. Hopefully you'll see the name and ID of the pool you're trying to import. You can use either the name of the id and try zpool import <name OR id>
5. Once you've imported the pool you'll want to identify and delete large files that you don't need. If it isn't too full to perform a delete you can just delete or move them to another disk.
6. If the above fails you're going to need to try to be a bit more crafty. From this post it looks like you can use dd to delete files when a pool is too full. You'll want to identify large files and then dd if=/dev/null of=/path/to/file and then rm /path/to/file. Unfortunately, this will permanently delete those files. To save them prior to that command you can try dd if=/path/to/file of=/path/to/location/on/another/pool/or/device. If you really care about not losing the file you're trying to remove and that last dd command fails stop. Once you use /dev/null you'll lose that file.
7. Once you get the pool's usage to below 90% or so you can try rebooting the computer and unlocking it from the UI.

I will say that I am somewhat less knowledgable about the disk full issue than I am about the encryption issue you were having. You may want to see if someone more knowledgeable has a better suggestion or research my suggestions on your own to decide if you think it is safe. I believe my steps above are safe but I encourage you to do some research to understand the commands before you you enter them in case I am wrong and you lost your data or your pool.

 

[devnullius]

:( This article says I'm shit out of luck and ALL DATA IS LOST?? For real? :(

https://www.ixsystems.com/community/threads/errno-28-no-space-left-on-device.39476/post-244327

 

[devnullius]

Great writeup, I wish I was confident enough to start doing it :/

True, I cannot get the pool decrypted (see error in initial post)

I don't have a backup of the NAS. The NAS is supposed to be my safe backup place. Latest backup maxed the disks out and I had not yet made time to clean up.

2... "Decrypt all your drives" one by one? From the 8-disk Z2 pool? Can you even have a partially encrypted and decrypted volume? It sounds like I'll break the whole dataset if I start decrypting disks?

3... is this only AFTER I did step 2, decrypting each disk one by one?

4.... I'll try that next and edit the results back in here.
EDIT: no such luck (see command outputs below)

Code:

root@Freenas:~ # cd mnt
mnt: No such file or directory.
root@Freenas:~ # cd /mnt
root@Freenas:/mnt # ls
md_size         VOLU10TB
root@Freenas:/mnt # zpool import ZFS_8x_3TB_RAIDz2_pool
cannot import 'ZFS_8x_3TB_RAIDz2_pool': no such pool available
root@Freenas:/mnt # zpool import
root@Freenas:/mnt # zpool import 3164769a-9aaf-4d57-87b3-311bebe92e0b
cannot import '3164769a-9aaf-4d57-87b3-311bebe92e0b': no such pool available
root@Freenas:/mnt # zpool import fcb9048a-8caa-474f-bec1-ce0717623cc3
cannot import 'fcb9048a-8caa-474f-bec1-ce0717623cc3': no such pool available

5... Once I have access that will be no problem. I have plenty of space left on the 10TB disk to move some files to.

6.... How do I get access to a file list if the disk is still locked (and encrypted)?

Possibly, possibly not. It sounds like you've got the correct geli key, which is great.

I assume then that the UI does not show your pool as decrypted under storage->pools?

If the UI won't import your pool you can try to do it manually, delete some files to get more space in your pool, and then try again using the UI.

What you'll want to do try is

1. If you have a backup rebuild your pool with more or larger drives and restore from backup. If you do not have a backup proceed to the next steps.
2. Decrypt all of your drives. This can be accomplished with the command I mentioned in the prior post against each drive in the pool. Use geom disk list to get the name of each of your 3TB disks, the one we want is associated with the Name: lable.
3. Run geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key on each of your 3TB disks.
4. Try to import the pool. I think based on your prior post that your pool's name is ZFS_8x_3TB_RAIDz2_pool. You can then try zpool import ZFS_8x_3TB_RAIDz2_pool. If that fails you might try to identify the pool's ID with zpool import. Hopefully you'll see the name and ID of the pool you're trying to import. You can use either the name of the id and try zpool import <name OR id>
5. Once you've imported the pool you'll want to identify and delete large files that you don't need. If it isn't too full to perform a delete you can just delete or move them to another disk.
6. If the above fails you're going to need to try to be a bit more crafty. From this post it looks like you can use dd to delete files when a pool is too full. You'll want to identify large files and then dd if=/dev/null of=/path/to/file and then rm /path/to/file. Unfortunately, this will permanently delete those files. To save them prior to that command you can try dd if=/path/to/file of=/path/to/location/on/another/pool/or/device. If you really care about not losing the file you're trying to remove and that last dd command fails stop. Once you use /dev/null you'll lose that file.
7. Once you get the pool's usage to below 90% or so you can try rebooting the computer and unlocking it from the UI.

I will say that I am somewhat less knowledgable about the disk full issue than I am about the encryption issue you were having. You may want to see if someone more knowledgeable has a better suggestion or research my suggestions on your own to decide if you think it is safe. I believe my steps above are safe but I encourage you to do some research to understand the commands before you you enter them in case I am wrong and you lost your data or your pool.

I think you're awesome writing this all up for us!

 

[devnullius]

PS: I thought about buying an extra disk to expand the pool (like proposed), but that does not seem to be a valid option either. When I go to Storage - Volumes, select my locked & encrypted ZFS_8x_3TB_RAIDz2_pool, and chose Volume Manager, in the follow up screen I cannot select my 8-disk pool. Only my 10TB disk shows up to expand further.

I took a look at your article, but I don't think I can do that if I cannot unlock the volume and get an 'ls' file listing to pick a file to nuke with /dev/null?

 

[PhiloEpisteme]

I don't have a backup of the NAS. The NAS is supposed to be my safe backup place. Latest backup maxed the disks out and I had not yet made time to clean up.

That is unfortunate. You may be in trouble here but its worth a shot to try to get the data.

2... "Decrypt all your drives" one by one? From the 8-disk Z2 pool? Can you even have a partially encrypted and decrypted volume? It sounds like I'll break the whole dataset if I start decrypting disks?

Geli encrypts entire disks. The zpool operates on decrypted disks. Encrypting and decrypting the disk will not harm your data or break it. Until you decrypt your drives you cannot import the pool; this is because the data on the drives to import the pool is encrypted.

3... is this only AFTER I did step 2, decrypting each disk one by one?

You need to decrypt every drive in the pool before you try to import it.

4.... I'll try that next and edit the results back in here.
EDIT: no such luck (see command outputs below)

Did you decrypt the drives first?

5... Once I have access that will be no problem. I have plenty of space left on the 10TB disk to move some files to.

In zfs when a pool is 100% full it will behave oddly. You may not be able to delete things from the pool. First things first, decrypt the pool, then import it, and then see if you can access the data. All 3 of those things have to happen in order to recover your data.

When I go to Storage - Volumes, select my locked & encrypted ZFS_8x_3TB_RAIDz2_pool, and chose Volume Manager, in the follow up screen I cannot select my 8-disk pool. Only my 10TB disk shows up to expand further.

This is expected since your pool is still locked. Geli encryption works on entire disks. Your 8-disk pool is encrypted and thus your pool is unavailable until they are decrypted and the pool imported.


The steps 1-4 above should be safe. Where things get uncertain is step 5. The reason is that it really depends on whether you can access the data at all or not. Try steps 1-4 and report back?

 

[Chris Moore]

My volume is completely full (I ignored it until I had some time to spare to configure a completely new backup strategy for my computers). It looks like that is giving a problem now?? ;-(

If I recall correctly, you were dismissive of the advice you received the last time you were here asking for help.

 

[Redcoat]

Does OP have an alternative besides getting 8 larger drives and doing a one-by-one replace in order to create more room? Is there any reason that won't likely work?

 

[PhiloEpisteme]

Does OP have an alternative besides getting 8 larger drives and doing a one-by-one replace in order to create more room? Is there any reason that won't likely work?

If OP can import the pool and free up a bit of space as described above perhaps not. First thing's first; gotta decrypt and import that pool.

 

[Redcoat]

Ah - from the posts I am under the impression that OP has tested that idea and hasn't been able to decrypt, hence no import.

 

[devnullius]

If I recall correctly, you were dismissive of the advice you received the last time you were here asking for help.

I probably just didn't like your comments, because they are harsh, unfair, unfriendly and impatient. I'm always open for help. Just not the all the bullshit that seems to follow once in a while. Not everybody is rigid to the core. The last I remember though is that the site owner had to step in and apologize on your behalf, giving the FreeNAS community a bad name... Just saying ;p

Thank you.

 

[PhiloEpisteme]

Ah - from the posts I am under the impression that OP has tested that idea and hasn't been able to decrypt, hence no import.

@devnullius Can you confirm whether you tried to decrypt all of the disks and import the disks? If so, it would be helpful if you pasted here the exact commands and responses you received similar to what you did above when you tried importing the pool.

 

[devnullius]

Ah - from the posts I am under the impression that OP has tested that idea and hasn't been able to decrypt, hence no import.

No, I'm still stuck at the unlocking part. We get the decryption partly working (see other thread). Now it's time to decrypt the 8 disks, which should open up the filesystem so I can choose a file to wipe with /dev/null and then finally I should be able to import the pool again - or maybe even before that. First things first: step 1 to 4 :)

Thanks.

 

[devnullius]

@devnullius Can you confirm whether you tried to decrypt all of the disks and import the disks? If so, it would be helpful if you pasted here the exact commands and responses you received similar to what you did above when you tried importing the pool.

Will do! I have not yet tried it - too afraid :)

xx

 

[PhiloEpisteme]

Will do! I have not yet tried it - too afraid :)

You'll have to decrypt the drives some time. If you don't, you'll never have access to your data :) When you go through the steps outlined above (1-4) please copy the exact commands you entered and the exact output it gave you. It will help us debug if something does not work. :)

 

[devnullius]

I guess I have my evening planned then ;p This looks like a helpful article to help me along here :)

https://www.openattic.org/posts/unlock-geli-ecrypted-zfs-volume-freenas/

Gonna test this first (I'll be editing here for a while...) --> geli attach -k [geli_key_file] [dev_to_unlock] (with my disks listed here: https://www.ixsystems.com/community...-at-storage-volume-2-unlock.75032/post-521429)

If I understand you and that article correct, the decrypting is failing :(

Code:

root@Freenas:/mnt # ls
md_size         VOLU10TB
root@Freenas:/mnt # cd /data/geli
root@Freenas:/data/geli # ls
fcb9048a-8caa-474f-bec1-ce0717623cc3.key
root@Freenas:/data/geli # geli attach -k fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/da0
geli: Cannot read metadata from /dev/da0: Invalid argument.
root@Freenas:/data/geli # geli attach -k fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/da1
geli: Cannot read metadata from /dev/da1: Invalid argument.
root@Freenas:/data/geli #

FROM: http://zfsguru.com/forum/zfsgurusupport/1074

FreeNAS does not use GPT partition labels, so you have to guess the device name. But probably FreeNAS uses GELI encryption nodes.

If the above command geli attach /dev/ada2p2 asks you for a passphrase, you should be good! If it says the following, then you have got the wrong device:

geli: Cannot read metadata from /dev/ada2s1: Invalid argument.

So I'm not using the proper device name...? I'll check if I have some other device names… Till edit :)

 

[PhiloEpisteme]

If I understand you and that article correct, the decrypting is failing :(

Ah, sorry, I may have overlooked a step. Do you see anything in /dev/gptid? If so, what? Also, what is the output of /dev/da*?

 

[devnullius]

Ah, sorry, I may have overlooked a step. Do you see anything in /dev/gptid? If so, what? Also, what is the output of /dev/da*?

Hello Philo - you are what still gives me hope here :)

Code:

root@Freenas:/dev # cd gptid
root@Freenas:/dev/gptid # ls
6a228fbc-98b8-40df-84ec-60d9afd8d0ac    7a2fc1c9-08ef-11e8-ab60-0025901159d4    888ff6c8-08ef-11e8-ab60-0025901159d4
6a228fbc-98b8-40df-84ed-60d9afd8d0ac    7b297d44-08ef-11e8-ab60-0025901159d4    8bb7c249-08ef-11e8-ab60-0025901159d4
6a228fbc-98b8-40df-84ef-60d9afd8d0ac    7c382ed2-08ef-11e8-ab60-0025901159d4    8ec07bd1-08ef-11e8-ab60-0025901159d4
6baca59b-0553-11e8-a1db-0025901159d4    7eade0a6-08ef-11e8-ab60-0025901159d4    ada6a7ed-4c9c-11e9-aec6-0025901159d4
7a1e9bcd-08ef-11e8-ab60-0025901159d4    85693c1a-08ef-11e8-ab60-0025901159d4

Code:

root@Freenas:/dev/gptid # ls /dev/da*
/dev/da0        /dev/da1p1      /dev/da2p2      /dev/da4        /dev/da5p1      /dev/da6p2      /dev/da8        /dev/da9p1
/dev/da0p1      /dev/da1p2      /dev/da3        /dev/da4p1      /dev/da5p2      /dev/da7        /dev/da8p1      /dev/da9p2
/dev/da0p2      /dev/da2        /dev/da3p1      /dev/da4p2      /dev/da6        /dev/da7p1      /dev/da8p2      /dev/da9p3
/dev/da1        /dev/da2p1      /dev/da3p2      /dev/da5        /dev/da6p1      /dev/da7p2      /dev/da9

So I need to either use either /dev/da0p1 or /dev/da0p2 or both? It's still safe to just test, right? :)

 

[PhiloEpisteme]

So I need to either use either /dev/da0p1 or /dev/da0p2 or both? It's still safe to just test, right? :)

Sorry, my memory failed me earlier. I believe you actually want to use the gptid devices.

If you want you can use gpart list to map the gptid ID to the device name and then smartctl -i /dev/<device_name> to figure out which gptid devices map to your 3TB drives. OR, you can be lazy and just try each of the /dev/gptid devices like geli attach -k /path/to/geli.key /dev/gptid/<id>. You won't hurt anything this way because only the correct devices will decrypt and the incorrect devices will fail to decrypt without hurting anything on the device.

If that fails, what is the error message? If that works, what is the output of zpool import?

 

[devnullius]

You know you are forgiven, right? :)

Progress! After a disappointing start... 8 disks requested the password!

Code:

root@Freenas:/dev/gptid # ls
6a228fbc-98b8-40df-84ec-60d9afd8d0ac    7a2fc1c9-08ef-11e8-ab60-0025901159d4    888ff6c8-08ef-11e8-ab60-0025901159d4
6a228fbc-98b8-40df-84ed-60d9afd8d0ac    7b297d44-08ef-11e8-ab60-0025901159d4    8bb7c249-08ef-11e8-ab60-0025901159d4
6a228fbc-98b8-40df-84ef-60d9afd8d0ac    7c382ed2-08ef-11e8-ab60-0025901159d4    8ec07bd1-08ef-11e8-ab60-0025901159d4
6baca59b-0553-11e8-a1db-0025901159d4    7eade0a6-08ef-11e8-ab60-0025901159d4    ada6a7ed-4c9c-11e9-aec6-0025901159d4
7a1e9bcd-08ef-11e8-ab60-0025901159d4    85693c1a-08ef-11e8-ab60-0025901159d4
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/
geli: Cannot read metadata from /dev/gptid/: Inappropriate file type or format.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/6a228fbc-98b8-40df-84ec-60d9afd8d0ac
geli: Cannot read metadata from /dev/gptid/6a228fbc-98b8-40df-84ec-60d9afd8d0ac: Invalid argument.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/6a228fbc-98b8-40df-84ed-60d9afd8d0ac
geli: Cannot read metadata from /dev/gptid/6a228fbc-98b8-40df-84ed-60d9afd8d0ac: Invalid argument.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/6a228fbc-98b8-40df-84ef-60d9afd8d0ac
geli: Cannot read metadata from /dev/gptid/6a228fbc-98b8-40df-84ef-60d9afd8d0ac: Invalid argument.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/6baca59b-0553-11e8-a1db-0025901159d4
geli: Cannot read metadata from /dev/gptid/6baca59b-0553-11e8-a1db-0025901159d4: Invalid argument.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/7a1e9bcd-08ef-11e8-ab60-0025901159d4
geli: Cannot read metadata from /dev/gptid/7a1e9bcd-08ef-11e8-ab60-0025901159d4: Invalid argument.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/7a2fc1c9-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/7b297d44-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/7c382ed2-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/7eade0a6-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/85693c1a-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/                             geli: Cannot read metadata from /dev/gptid/: Inappropriate file type or format.
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/888ff6c8-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/8bb7c249-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/8ec07bd1-08ef-11e8-ab60-0025901159d4
Enter passphrase:
root@Freenas:/dev/gptid # geli attach -k /data/geli/fcb9048a-8caa-474f-bec1-ce0717623cc3.key /dev/gptid/ada6a7ed-4c9c-11e9-aec6-0025901159d4
geli: Cannot read metadata from /dev/gptid/ada6a7ed-4c9c-11e9-aec6-0025901159d4: Invalid argument.

Code:

root@Freenas:/dev/gptid # zpool import
   pool: ZFS_8x_3TB_RAIDz2_pool
     id: 4057754529204707565
  state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        ZFS_8x_3TB_RAIDz2_pool                              ONLINE
          raidz1-0                                          ONLINE
            gptid/7a2fc1c9-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/7b297d44-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/7c382ed2-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/7eade0a6-08ef-11e8-ab60-0025901159d4.eli  ONLINE
          raidz1-1                                          ONLINE
            gptid/85693c1a-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/888ff6c8-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/8bb7c249-08ef-11e8-ab60-0025901159d4.eli  ONLINE
            gptid/8ec07bd1-08ef-11e8-ab60-0025901159d4.eli  ONLINE

Out of curiosity… What we did manually just now, that's the first step the GUI would do if I press to unlock a pool? This decrypts the pool, then it needs to get "mounted" or something? Investigating next but so far it looks excellent like everything is up and running already!

xx