Encrypted array lost after upgrade to FreeNAS-11.0-RELEASE

Status
Not open for further replies.
B

BeWe

Cadet
Joined
Dec 17, 2017
Messages
7
Hi guys,

i have a really bad problem. I recently upgraded a FreeNAS 9.10 server to FreeNAS 11.
The server has 4 arrays. One, of a little crude construction, did not make the upgrade. The other three suvived.

One guess is that it is because of the construction of the array:
It once was a normal RAID-Z1 of 5 x 3TB HDDs. Encrypted.
Then, it should get expanded, a single 3 TB HDD was added. As a JBOD with the Z1 Pool...
JBOD
|-HDD
|-Z1
|-HDD
|-HDD
|-HDD
|-HDD
|-HDD

I downgraded at a blink, but it remains broken.

Error after trying to decrypt the pool:
Code:
Environment: Software Version: FreeNAS-11.0-RELEASE (a2dc21583) Request Method: POST Request URL: https://IP/storage/volume/1/unlock/?X-Progress-ID=2f89133e-f212-48d7-82ab-6a558cb2822a Traceback: File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner 39.			 response = get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response 249.			 response = self._get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response 178.			 response = middleware_method(request, callback, callback_args, callback_kwargs) File "./freenasUI/freeadmin/middleware.py" in process_view 162.		 return login_required(view_func)(request, *view_args, **view_kwargs) File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view 23.				 return view_func(request, *args, **kwargs) File "./freenasUI/storage/views.py" in volume_unlock 1190.			 form.done(volume=volume) File "./freenasUI/storage/forms.py" in done 2613.			 raise MiddlewareError(msg) Exception Type: MiddlewareError at /storage/volume/1/unlock/ Exception Value: [MiddlewareError: b'Volume could not be imported: 1 devices failed to decrypt']


Any ideas?

Thanks in advance,
BeWe
 
Last edited by a moderator:
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Do you have backups of the GELI metadata? If not, it is highly likely that your data is gone.

Even with it, your pool layout is a ticking time bomb...
 
B

BeWe

Cadet
Joined
Dec 17, 2017
Messages
7
Yes, you mean the geli.key one can export, right? I have those!.

As for the ticking time bomb - i know :( But i cannot afford the money for a new server/disk array at the moment to savely copy the data :(

Thanks for the reply!
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
BeWe said:
Yes, you mean the geli.key one can export, right? I have those!.
Click to expand...
No, you need that, too, but you may also need the GELI metadata, which may have been wiped from your drives.

See if you can import the pool in 11.1.

@rs225, do you remember off the top of your head how to quickly check if the metadata is still intact? If not, it can be directly read from adaXp2's last sector. You'd need to grab the relevant sizes using gpart list and then you can use something like dd if=/dev/adaXp2 bs=512 iseek=second-to-last-sector-number.
 
B

BeWe

Cadet
Joined
Dec 17, 2017
Messages
7
Ok, did the upgrade.

Situation unchanged.

"New" Error after trying to decrypt the pool:
Code:
Environment:

Software Version: FreeNAS-11.1-RELEASE (dc7d195f4)
Request Method: POST
Request URL: https://IP/storage/volume/1/unlock/?X-Progress-ID=c0da2515-e8d3-40a5-8c70-ddb2f9488a7b


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volume_unlock
  1005.			 form.done(volume=volume)
File "./freenasUI/storage/forms.py" in done
  2792.			 raise MiddlewareError(msg)

Exception Type: MiddlewareError at /storage/volume/1/unlock/
Exception Value: [MiddlewareError: Volume could not be imported: 1 devices failed to decrypt]


In /var/log/messages i can see:
Code:
Dec 17 18:07:31 fn1 savecore: /dev/ada9p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada8p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada7p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada6p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada5p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada4p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada3p1: Operation not permitted
Dec 17 18:07:31 fn1 savecore: /dev/ada2p1: Operation not permitted
Dec 17 18:07:32 fn1 savecore: /dev/ada1p1: Operation not permitted
Dec 17 18:07:32 fn1 savecore: /dev/ada0p1: Operation not permitted
Dec 17 18:07:56 fn1 GEOM_ELI: Device gptid/4d708ba9-c07d-11e5-936b-0015177cf92a.eli created.
Dec 17 18:07:56 fn1 GEOM_ELI: Encryption: AES-XTS 128
Dec 17 18:07:56 fn1 GEOM_ELI:	 Crypto: hardware
Dec 17 18:07:58 fn1 GEOM_ELI: Device gptid/53ba17dd-c07d-11e5-936b-0015177cf92a.eli created.
Dec 17 18:07:58 fn1 GEOM_ELI: Encryption: AES-XTS 128
Dec 17 18:07:58 fn1 GEOM_ELI:	 Crypto: hardware
Dec 17 18:08:00 fn1 GEOM_ELI: Device gptid/58aff366-c07d-11e5-936b-0015177cf92a.eli created.
Dec 17 18:08:00 fn1 GEOM_ELI: Encryption: AES-XTS 128
Dec 17 18:08:00 fn1 GEOM_ELI:	 Crypto: hardware
Dec 17 18:08:01 fn1 GEOM_ELI: Device gptid/5d7dd3a1-c07d-11e5-936b-0015177cf92a.eli created.
Dec 17 18:08:01 fn1 GEOM_ELI: Encryption: AES-XTS 128
Dec 17 18:08:01 fn1 GEOM_ELI:	 Crypto: hardware
Dec 17 18:08:04 fn1 uwsgi: [middleware.exceptions:36] [MiddlewareError: Unable to geli attach gptid/326cd7c2-adf1-11e7-8220-0015177cf92a: geli: Missing -p flag.
]
Dec 17 18:08:04 fn1 uwsgi: [middleware.notifier:603] [MiddlewareError: Unable to geli attach gptid/326cd7c2-adf1-11e7-8220-0015177cf92a: geli: Missing -p flag.
]
Dec 17 18:08:04 fn1 GEOM_ELI: Device gptid/e3ec98be-5834-11e7-b3d2-0015177cf92a.eli created.
Dec 17 18:08:04 fn1 GEOM_ELI: Encryption: AES-XTS 256
Dec 17 18:08:04 fn1 GEOM_ELI:	 Crypto: hardware
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=15845852502573375739
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=2071332394144773343
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=14440408338309020379
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=6939963922824899172
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=662214023928919137
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=15845852502573375739
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=2071332394144773343
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=14440408338309020379
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=6939963922824899172
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=662214023928919137
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:08 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=15845852502573375739
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=2071332394144773343
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=14440408338309020379
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=6939963922824899172
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=662214023928919137
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:09 fn1 ZFS: vdev state changed, pool_guid=3181903306352025378 vdev_guid=9618196201370477904
Dec 17 18:08:09 fn1 uwsgi: [middleware.notifier:2524] Importing StoragePool1 [3181903306352025378] failed with: cannot import 'StoragePool1': one or more devices is currently unavailable
Dec 17 18:08:09 fn1 uwsgi: [middleware.exceptions:36] [MiddlewareError: Volume could not be imported: 1 devices failed to decrypt]


I've especially noticed the:
Dec 17 18:07:31 fn1 savecore: /dev/ada3p1: Operation not permitted
message. But besides the one failing decryption, the all other disks are fine - no failures/inaccessibility.

So much thanks guys!
 
Last edited by a moderator:
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
geli dump /dev/adaXp2 will either give an error, or show that it is there. Do not paste the output of geli dump to the forums unless it is an error.
 
Last edited:
B

BeWe

Cadet
Joined
Dec 17, 2017
Messages
7
i did geli dump /dev/adaXp2 for every disk and got no error from none.
ada2 shows -1 iterations though.

Is this the metadata and should i save it?

The output of zpool import

Code:
   pool: StoragePool1
	 id: 3181903306352025378
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

		StoragePool1										UNAVAIL  missing device
		  raidz1-0										  ONLINE
			gptid/4d708ba9-c07d-11e5-936b-0015177cf92a.eli  ONLINE
			gptid/e3ec98be-5834-11e7-b3d2-0015177cf92a.eli  ONLINE
			gptid/53ba17dd-c07d-11e5-936b-0015177cf92a.eli  ONLINE
			gptid/58aff366-c07d-11e5-936b-0015177cf92a.eli  ONLINE
			gptid/5d7dd3a1-c07d-11e5-936b-0015177cf92a.eli  ONLINE

		Additional devices are known to be part of this pool, though their
		exact configuration cannot be determined.


Thank you so much guys!
 
Last edited:
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Okay, so you dodged the bullet and tripped into the giant meat grinder. Your only hope is to get that drive recognized and mostly working.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Does it show up with camcontrol devlist?
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Did you download the keys after adding it?
 
Status
Not open for further replies.

Similar threads

P
  • Locked
Upgrade FreeNAS 11.0-U4 -> 11.1 failed + db> Shell after Upgrades
Replies
3
Views
2K
peacepipe
P
M
  • Locked
FreeNAS 11.1-U3 Update: "MiddlewareError" - no install
Replies
10
Views
3K
mazab
M
A
Unable to upgrade to 11.2 from 11.1-u6
Replies
1
Views
912
dlavigne
D
C
Freenas update 11.1u6 -> 11.1u7 fails
Replies
11
Views
1K
Cd228
C
D
problem after upgrade to 11.1-u7
Replies
2
Views
995
dlavigne
D
Top