Zpool, removing geli encryption and replacing drive

[jyavenard]

Hi.

I'm in the process of removing on all my disks (12 of them) the geli encryption.
so I did:
`zpool offline pool gptid/8bd09417-36ac-11ee-8e52-3cecef479fec.eli`
`geli detach gptid/8bd09417-36ac-11ee-8e52-3cecef479fec.eli`
`zpool replace pool gptid/8bd09417-36ac-11ee-8e52-3cecef479fec.eli gptid/8bd09417-36ac-11ee-8e52-3cecef479fec`

so far so good. Done 10 so far.
Now one disk failed in the process right after I got spurious smart errors.
So I replaced the disk with a spare. But whenever I attempt to replace the existing disk from the zpool, which is now offline ; TrueNAS keeps replacing it with a a geli one.

I can offline that one for sure and do it manually. But that got me curious. What is the way to replace a disk without getting geli involved?

If I detach /dev/ata2 (where the drive being replaced is connected to) then TrueNAS also delete the GPT partition too and I'm left with just as /dev/ata2 device.
I've always been told to not plainly give a /dev/ata2 device to zfs, it makes it hard to track the disk in the future, and additionally linux has trouble with those zpool.

So how can I create a similar schema on the disk as what truenas is doing when replacing a disk, but without geli ?

Thanks

 

[jyavenard]

I ended up copying what was done to another disk of the same type:
so I had:```

# gpart show -l da0
=> 40 7814037088 da0 GPT (3.6T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 7809842688 2 freebsd-zfs (3.6T)
7814037120 8 - free - (4.0K)

and I did:

# gpart create -s GPT ada2`
# gpart add -t freebsd-swap -b 128 -s 2G ada2`
# gpart add -t freebsd-zfs -s 7809842688 ada2`

# gpart show -lada2
=> 40 7814037088 ada2 GPT (3.6T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 7809842688 2 freebsd-zfs (3.6T)
7814037120 8 - free - (4.0K)

glabel status showed me the gptid for ada2p2

zpool replace pool gptid/95e55ec6-b9d2-11eb-b10e-002590875a70.eli gptid/57c9de14-36b1-11ee-8e52-3cecef479fec

Only thing is that the GUI isnt showing me `ada2` as the drive being used like all other disks, but the GPT partition instead.

 

[sretalla]

If you're wanting the GUI to be happy with something, you need to take all actions on it in the GUI.

Replace it with itself in the GUI if that's important to you.

 

[jyavenard]

Replace it with itself in the GUI if that's important to you.

that's possible? fairly sure it will tell me that the disk is already part of the pool

 

[sretalla]

that's possible? fairly sure it will tell me that the disk is already part of the pool

not if you offline it first and then wipe it.

 

[jyavenard]

not if you offline it first and then wipe it.

but won't that put back the geli encrypted one? why would it be different from the first time it was added?

 

[sretalla]

but won't that put back the geli encrypted one? why would it be different from the first time it was added?

If you remove the GELI one, that should leave that member disk as unencrypted... there's no way the GUI can put legacy encryption back.

 

[jyavenard]

there's no way the GUI can put legacy encryption back.

I can guarantee you that it does.
If I add a new disk, replace the broken disk with that new disk: what is added is the geli device

 

[jyavenard]

there's no way the GUI can put legacy encryption back.

So turned out the replacement disk was broken too. So I replaced it again with another.

Here is my pool status before the replacement:

pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/268a0421-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/27c38d4a-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/28ff3184-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2a31a873-461b-11e9-a077-002590875a70 ONLINE 0 0 0
replacing-4 DEGRADED 0 0 0
gptid/2aee2466-461b-11e9-a077-002590875a70.eli OFFLINE 0 0 0
gptid/2aee2466-461b-11e9-a077-002590875a70 ONLINE 0 0 0 (resilvering)
gptid/e0298a14-b9d3-11eb-b10e-002590875a70.eli ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gptid/2d0b57b3-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2dda82a5-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2e9f200f-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2fd04551-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/95e55ec6-b9d2-11eb-b10e-002590875a70.eli OFFLINE 0 0 0
gptid/3245e6d5-461b-11e9-a077-002590875a70.eli ONLINE 0 0 0

so I inserted a new disk, it's now in /dev/ada2

disk got replaced:

And this is what TrueNAS actually replaced with:

NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/268a0421-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/27c38d4a-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/28ff3184-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2a31a873-461b-11e9-a077-002590875a70 ONLINE 0 0 0
replacing-4 DEGRADED 0 0 0
gptid/2aee2466-461b-11e9-a077-002590875a70.eli OFFLINE 0 0 0
gptid/2aee2466-461b-11e9-a077-002590875a70 ONLINE 0 0 0 (resilvering)
gptid/e0298a14-b9d3-11eb-b10e-002590875a70.eli ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gptid/2d0b57b3-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2dda82a5-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2e9f200f-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2fd04551-461b-11e9-a077-002590875a70 ONLINE 0 0 0
replacing-4 DEGRADED 0 0 0
gptid/95e55ec6-b9d2-11eb-b10e-002590875a70.eli OFFLINE 0 0 0
gptid/cf02d32f-3716-11ee-8e52-3cecef479fec.eli ONLINE 0 0 0 (resilvering)
gptid/3245e6d5-461b-11e9-a077-002590875a70.eli ONLINE 0 0 0

seem a bit silly to tell that TrueNAS 13's geli is legacy and to stop using it, only to have the GUI use it again.

 

[sretalla]

Just to be clear, from what I can see, the disks that replaced the .eli encrypted ones also become .eli... that's what I expect.

But if, as I said, you removed the .eli disk from the pool already and the only replica of that member is not .eli, then it can only give you back an unencrypted one.

Anyway, it's a little hard to read what you posted there to confirm that or not, but it's what I expect.

 

[jyavenard]

Alright. After almost 2 weeks, this process is complete. None of my disks in the pool are using geli.

However, in the GUI it still shows as being "Legacy Encryption"

pool: pool
state: ONLINE
scan: resilvered 1.73T in 11:07:11 with 0 errors on Sat Aug 12 20:54:54 2023
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/268a0421-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/27c38d4a-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/28ff3184-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2a31a873-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2aee2466-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/e0298a14-b9d3-11eb-b10e-002590875a70 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/2d0b57b3-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2dda82a5-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2e9f200f-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/2fd04551-461b-11e9-a077-002590875a70 ONLINE 0 0 0
gptid/5c8e8276-37f6-11ee-b159-3cecef479fec ONLINE 0 0 0
gptid/3245e6d5-461b-11e9-a077-002590875a70 ONLINE 0 0 0

Is there anything more I need to do to complete the process?

JY
Edit: BTW, I got two disks failing to resilver (heaps of smart errors after a day or so), both new and both turned out to be SMR. Lesson learned there.

 

[jyavenard]

Follow-up: I followed the steps of this site:

KB450999 - Removing GELI Encryption from ZFS Pool - 45Drives Knowledge Base

You are here: KB Home FreeNAS KB450999 – Removing GELI Encryption from ZFS Pool Table of Contents Scope/DescriptionPrerequisitesStepsGet gptid's for each driveSteps for each drive in the poolOffline a drive using the gptidReplace drive with itselfExport the PoolVerification Scope/Description...

knowledgebase.45drives.com

worked great. all good now