Old drive not removed when replacing dead (offlined) GELI drive

[nilsb]

Hello, I've posted this as a bug at https://bugs.freenas.org/issues/20489
...but I also need help fixing this short term as I'm not sure what happens after a reboot...

Situation:
- FreeNAS-9.10.2-U1 (86c7ef5)
- 4x2TB disk raidz2 in one pool (the only one on the system), where one disk (/dev/ada1) failed and went directly offline
- I had never set a passphrase on the pool
- the disk was replaced after powering off, and after booting, the pool resilvered when running the "replace" action
- the system still looks for the old disk ( gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c N/A ada1p2 )
- none of https://doc.freenas.org/9.10/storage.html#replacing-an-encrypted-drive or https://doc.freenas.org/9.10/storage.html#managing-encrypted-volumes work, with the following errors:

Code:

Jan 20 22:15:50 freenas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to delete key 1 on gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: geli: Cannot open gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: No such file or directory.
]
Jan 20 22:16:05 freenas notifier: 1+0 records in
Jan 20 22:16:05 freenas notifier: 1+0 records out
Jan 20 22:16:05 freenas notifier: 64 bytes transferred in 0.000067 secs (955286 bytes/sec)
Jan 20 22:16:05 freenas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to set passphrase on gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: geli: Cannot open gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: No such file or directory.
]
Jan 20 22:16:05 freenas manage.py: [middleware.notifier:1228] [MiddlewareError: Unable to set passphrase on gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: geli: Cannot open gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: No such file or directory.
]
Jan 20 22:16:05 freenas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: geli: Cannot open gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: No such file or directory.
]]
Jan 20 22:16:17 freenas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to delete key 1 on gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: geli: Cannot open gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c: No such file or directory.
]

Code:

[me@freenas /data/geli]$ glabel status
  Name  Status  Components
gptid/bdef8a68-95c2-11e5-a2cd-d8d385636e4c  N/A  ada0p2
gptid/bf6c6035-95c2-11e5-a2cd-d8d385636e4c  N/A  ada2p2
gptid/c032a980-95c2-11e5-a2cd-d8d385636e4c  N/A  ada3p2
gptid/dfdc6107-75b5-11e5-9e2e-d8d385636e4c  N/A  da0p1
gptid/dfe4fb42-75b5-11e5-9e2e-d8d385636e4c  N/A  da0p2
gptid/e0004b33-75b5-11e5-9e2e-d8d385636e4c  N/A  da1p1
gptid/e008ef26-75b5-11e5-9e2e-d8d385636e4c  N/A  da1p2
gptid/16fbe186-df2e-11e6-857a-d8d385636e4c  N/A  ada1p2
[me@freenas /data/geli]$ ll *key
-rw-r--r--  1 root  wheel  64 Nov 28  2015 a88a09ba-4dbb-495c-979c-41c11836b74b.key

The exported storage_encrypteddisk table (the only one in /data/freenas-v1.db where I still see that key) looks as follows, with no serial for the missing drive:

Code:

$ cat storage_encrypteddisk.csv
"encrypted_volume_id","id","encrypted_provider","encrypted_disk_id"
1,1,"gptid/bdef8a68-95c2-11e5-a2cd-d8d385636e4c","{serial}WD-WCC4Mnnnnnnn"
1,2,"gptid/beadf442-95c2-11e5-a2cd-d8d385636e4c",
1,3,"gptid/bf6c6035-95c2-11e5-a2cd-d8d385636e4c","{serial}MK0271Ynnnnnnn"
1,4,"gptid/c032a980-95c2-11e5-a2cd-d8d385636e4c","{serial}YGHnnnnnnn"
1,5,"gptid/16fbe186-df2e-11e6-857a-d8d385636e4c","{serial_lunid}PK2234nnnnnnnn"

If any other info is needed, please let me know.

 

[nilsb]

Solved... see bug report for steps.