Migrating encrypted pool to new (larger) encrypted pool

[MacLemon]

I need to migrate my existing encrypted zpool (RAIDZ2) onto a new, larger and also encrypted zpool. Luckily I do have a test system which I can test the whole procedure on and trash it as many times as I like.

I have read many threads on migration of pools but none of them actually takes into account encrypted pools. I haven't found actually working instructions regarding this in the documentation, this forum, with $ search engine on the interwebs either. (Links are provided at the bottom of this posting.)

Hardware details are also at the bottom of this posting. I don't consider the hardware even remotely being part of the problem. This is purely an issue of process or software.

The problem I'm running into is that I cannot import/remount an encrypted pool anymore, not with the geli keys, not with the geli_recovery keys.

Here's the procedure, based off of the thread Howto: migrate data from one pool to a bigger pool which deals with unencrypted source and destination pools only.

0. Scrubbing all the pools to make sure all my source data storage is fine.
Everything is OK.

1. I'm moving the system-dataset to the freenas-boot drive.
System > System Dataset > System Dataset pool: freenas-boot [V]

2. I'm taking a snapshot of the freenas-boot zpool just in case.
zfs snapshot -r freenas-boot@migration

3. I'm creating a backup of my system config using the GUI.
System > General > [Save Config]
Export Password Secret Seed: [X]
[OK]

4. I'm creating a backup of the key files in /data/geli.
cp -r /data/geli /data/geli_migration
I copy those keys off of the FreeNAS box via SSH to my laptop to have backup.

5. I'm creating a backup of the keys via the GUI to my laptop.
Storage [Volumes]: Select zsource [Download key]
I make sure that I know which pool this key (file named geli.key) belongs to.

I [add recovery key] as well and save the geli_recovery.key to my laptop as well.

I do the same for the freshly created zdestination zpool as well and make sure that I know these keys are the GELI keys for the destination pool I want to migrate to.

6. I'm using the command line to create a snapshot.
zfs snapshot -r zsource@migrate

I'm sending the data from zsource to zdestination.
zfs send -R zsource@migrate | pv | zfs receive -F zdestination

pv(1) is only there to get progress indication so I know that data is transferring and how fast it's moving. There's no difference regarding the zfs send | zfs receive without using pv(1) to show progress.

Yes, this is deliberate overwriting the destination pool which is empty. (Created with the GUI with the Volume manager.)

--
So far, this has been simple and working fine.

Now the next steps would be to rename the zsource pool to old-source and the zdestination pool to become the new zsource.

7. Swap the pools by renaming them
Export both pools.
zpool export zsource
[cmdzpool export zdestination[/cmd
Reimport the old source pool and rename it to source-old.
zpool import zsource zsource-old
I'll leave it imported for the moment.

Reimport the new zdestination pool to become the new zsource.
zpool import zdestination zsource





Steps that I would do after I managed to successfully reimport the new, encrypted pool into FreeNAS:

8. Exporting both pools again.
zpool export zsource
zpool export zsource-old
The zsource-old pool will be removed, as in physically taking the disks out when everything has been done successfully.


9. Import the new pool via the GUI to make FreeNAS aware of it.
Storage > [Import Volume]
Select all 8 disks used up by the new pool
O decrypt Disks
Use the matching geli.key
Provide the correct passphrase
[Import]

10. Restore the configuration backup created in step 3.
System > General: [Upload Config]

This should restore my SMB and AFP Sharepoints to a working state.
Reboot the whole box and test if everything is ok.

11. Live happily ever after.
ZFS should be happy that the capacity for the pool is below 80% again.
I shall be happy that I've got a larger zpool now which I can use to store data.



---

What keeps me from doing this with my production setup yet?

I've never managed to reimport an encrypted pool in FreeNAS. Neither by using geli.key from the corresponding pool not with geli_recovery.key. (This would result in a complete loss of data, which I obviously want to avoid.)

I'm obviously making some mistake in the whole process, yet I've most likely overlooked it too many times now to being able to find it.

The problem I run into is somewhere with steps 7 to 9 (including).

Any help pointing my brain in the direction of my error is welcome. I can test all suggestions before actually touching any production data.

Thanks!
Best regards
MacLemon


---
Short Hardware overview
Test System
HP Proliant Microserver G8
CPU: intel G1610T, 16GB RAM, FreeNAS 11.1-RELEASE on USB stick, 2 HDDs for testing (HGST Deskstar NAS v2 6TB) (Nothing fancy, just a tiny test box.)

Production system
This is the box I actually want to migrate.

SOC: SUPERMICRO A1SAM-2750F
Memory: 32GB RAM (4 * 8GB Samsung)
HBA: IBM ServeRAID M1015, crossflashed to LSI 9211-8i in IT mode
FreeNAS: 11.1-RELEASE (stable train) on SATA-DOM
zpool (“zsource”): 6 wide RAIDZ2 HGST Deskstar NAS 4TB.
zpool (“zdestination”): 8 wide RAIDZ2 HGST Deskstar NAS v2 6TB



---
Links I've considered before posting this:

From the FreeNAS Documentation:
8.1.6.1 Importing an Encrypted Pool
8.1.8.1. Managing Encrypted Volumes


From this forum:
Howto: migrate data from one pool to a bigger pool


From the FreeNAS Bug Tracker:
https://forums.freenas.org/index.ph...a-gui-but-able-to-via-cli.59430/#post-420655 Unable to import encrypted pool via GUI, but able to via CLI

Suggests, that 11-RELEASE-U5 does have a bug with importing encrypted pools.
11.1-RC3 is said to be fixed already. (I'm running FreeNAS 11.1-RELEASE now.)


All the nitty gritty tech hardware details
An extensive description of the exact hardware used in my production system can be found at [Feedback sought] Enthusiastic NAS build [18 month usage update 2017] in the Serve the home forums.

 

[wblock]

The encryption used by FreeNAS is based on GELI, which encrypts entire devices. After that encrypted device has been mounted, there is no difference in replication. ZFS does not see any difference, because the encryption happens on an underlying layer.

We should have some standard warning to include here about using encryption. It sounds good, but people often mistake the benefits and risks.

 

[MacLemon]

I'm completely aware of the risks of encryption as well as the benefits, which is why I'm testing before working on live data. Also I'm not having any problem with the replication ( zfs send|zfs receive) at all.

I'm explicitly talking about the encryption and that FreeNAS fails to decrypt/import an encrypted zpool it created, even though I have all the keys that FreeNAS allows me to export/save/backup.

If you could be so kind to point out the things I've not made clear in my posting, I'm happy to add the missing details or clarify what is giving me trouble.

 

[Ericloewe]

What's the output of geli dump /dev/adaXp2, for the relevant drives?

 

[MacLemon]

Since these are just test volumes there is no need to redact key material.

Code:

[root@freenas ~]# geli dump /dev/ada1p2																							
Metadata on /dev/ada1p2:																											
	magic: GEOM::ELI																											  
   version: 7																													  
	flags: 0x0																													
	ealgo: AES-XTS																												
   keylen: 256																													
  provsize: 5999027552256																										  
sectorsize: 4096																													
	 keys: 0x01																													
iterations: -1																													
	 Salt: 4de53e0852db7bf50cb9db8e4b041f1f1f165067fcf26b6d702d23c4e143720e2bd4ec1fa54b5a78c75c03be07125b23bebd370acabb15e65b6ef5c8
11ad96ac																															
Master Key: 6ebdbe154a1c66755b6ecfef4abb9fe6c26b17deb2113e548d133ca621c3fe0e8743fdee4afa3254646dd968bafd0bab3c8573bf14f137eacbbe4b31
6ca5ebde873ffe8bf618d67eec1ccfbe5c44653c92203b23406cf8111ae6c0c0136e4651b1185a427caf3020bf7f4acb336e27573572d9c151cc27ee19349286f9fe
3949cc6a86511ff8b6906f23b3070a8a28deb2f27522d87c7ebc16ffa8c374322e697ad0009d74c9f11a1f9c0470b5a262931dc73cd5240640a194bc8ce810748403
273af8c137908860e4aa17a71db6d52e8cb2a2a9ca90dee031dcadd1bffbaacda38a16aed0c8d1a4770ee2ee9ac3319f043c17560c394372b90900fc245d01e6a467
1c76e6f6896b0b40548e7e90cf1a4f038d44d71ec8def6005637751db65bae6786eda712e8ff87412e7778d2bebd1688bb063baade66929e5513c94dc8e7c243a289
51e7f55323d86244ea07b2037cc35a1010ae8b30d6d22f3fd6ef2e35802252ef6a5b5743da6c3a868b08890e23e2cbf80beb6b59cffb62ebec46c4ae			
  MD5 hash: aea9b58f47b6ee1a9e416bc25a2ee574																						
																																  
[root@freenas ~]# geli dump /dev/ada2p2																							
Metadata on /dev/ada2p2:																											
	magic: GEOM::ELI																											  
   version: 7																													  
	flags: 0x0																													
	ealgo: AES-XTS																												
   keylen: 256																													
  provsize: 5999027552256																										  
sectorsize: 4096																													
	 keys: 0x01																													
iterations: -1																													
	 Salt: 71036daf976686f54191c5286e960dabd495d96dc078dc291703c952cfc131839de7b32cf2d5c308a7e678c31cf1608f52484e30040a791b1089ef2f
d811fbf5																															
Master Key: af5a6a87ff3b0958a03e502373f2c4883ed2526d38d2947f4d761714d59263a12dc68936796fc23183f2ee12985845e5ae619a542b3b897c9393ca88
2f9e1876e2aaf92874132a6c0498d657124d2f5f38cd2b8d5fbe61f6fceedffeba5c9e6f8335f2e970477603d3dd33f03bb87aa69e3d50cb7682992a78f79dd04e08
9a90ef1ef9c2cad44a09ce4c5259ee0bcfc2bfc60c17491ddd19ac08653098cc7eda7af25bee285c428699ba378e9beb9a5148d3709d3635cb4bde85b6bb39ae4cac
3c0dd035bdf04220eef414280f35445bab0e798eb775ba71f7edd523e649ea18aaa78f20faeda18a338581fb81ce0861d87c3e035bc633e7a2a86ed3ec3706304881
45ff4972354a43df7c2d210547cb50f4f1da74f20bc0df0c41583da245d8d69f4335793e80bd4591a48e2d4a1cc8f3b9a6ccbf371bf561f8304fb86aa64ca98c4e11
55f129ae5a98d7daf242d4d8ebc0a8fb92b969dcf4e66424d5b5cb1d2b38a71332a1b930ca613200aec7881d23606868e7b0aab0223112e35dff8824			
  MD5 hash: fc33c3e05ecdc2d4aeb903b8d30f4f58

I've also tested with a clean direct installation of 11.1-RELEASE now and I can import a volume after detaching it. This only works when using geli.key but not when using geli_recovery.key. Reproducible with two test-volumes. The recovery key doesn't work, the normal key does work.

(Edit: Added note about recovery keys.)

 

[Ericloewe]

Okay, so the metadata is there (which is fairly obvious since you can decrypt the drives in some cases). That rules out the more nefarious bugs we've seen lately.

 

[IceBoosteR]

Hi @MacLemon,
I have seen your bugticket regarding this. I was shocked when I first read thorugh the Bug and the forum post.
I am not 100% sure, are you able to recreate the error? What is in you opinion the problem? That the GPID has changed after detaching? I may be able to test this also on my testsystem with an upgrade from 11.0-U4 to 11.1.
Can you post in short what you have done with your detached pool? Why did you detached it? :D

I have only encrypted pools and I fear every update because of Coral (the big fail) as I lost my pool because of a developer mistake. So I want to make 100% sure that all will be fine, and when I can reproduce the error, I might can also help you.
Cheers
Ice