SOLVED (physically) removing a drive from zpool and reattaching it without replacing?

[Skunky]

First of all: Hi to everyone, I'm new :)

I'm currently looking for the best software RAID solution for my 3x 6TB WD RED.
I already tested XPEnology and Windows Server (Diskmgmgt RAID 5 and Storage Spaces).

Today I set up a Freenas 11.0-U2 VM on my ESXi 6.5 server and did some drive failure simulations with VMDKs. (3x 8GB; RAIDZ)
I noticed, when I detach one of the drives from the VM, it's state changes to 'REMOVED' or 'UNAVAIL'. I haven't found a way to bring it back online after reattaching it to the VM, though.
The only thing I can do is replacing it, which means the drive is being wiped and rebuilt, even though the data on it is completely fine, right?

Is there actually no other way? This would be kind of worrying.
E.g Windows would notice that I reattached one of the pool drives and set it back online. Even if I removed 2 of 3 drives.

I tried the following commands to get it back online:



root@freenas:~ # zpool status test
pool: test
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: none requested
config:

NAME STATE READ WRITE CKSUM
test DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
2398268917499464536 UNAVAIL 0 0 0 was /dev/gptid/966cf301-7c64-11e7-91af-000c295e13ce.eli
gptid/96a1f57a-7c64-11e7-91af-000c295e13ce.eli ONLINE 0 0 0
gptid/96d74475-7c64-11e7-91af-000c295e13ce.eli ONLINE 0 0 0

errors: No known data errors



root@freenas:~ # zpool clear test 2398268917499464536



root@freenas:~ # zpool online test 2398268917499464536
warning: device '2398268917499464536' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present



root@freenas:~ # fmadm repaired zfs://pool=name/vdev=guid (obviously not a Freenas command)



I hope someone can help me here.
Thanks!


Skunky

 

[rs225]

It is probably the encryption. Fastest solution is probably to reboot.

 

[Skunky]

Wow, you are right! It was actually the encryption. Works flawlessly without it...
Is this intended behavior, or is there anything I can do about it? Encryption would be really important to me.

I tried rebooting of course :D Didn't change anything.
Sorry, forgot to mention that.

 

[Skunky]

Ok, I solved the problem. For anyone with the same issue:
I found out, that not the zpool itself is encrypted, but the entire disks, before the zpool is even created.
That means, when I reattach the disk, the zpool has no clue that this disk is one of it's own, because it is still encrypted and therefore unreadable.
Solution:

Code:

root@freenas: # geli attach -k "/tmp/geli.key" "/dev/da1p2"
Enter passphrase:

After that, the zpool instantly jumped back to healthy.

If more than one drive is detached and reattached you have to run this command after decrypting all drives:

Code:

root@freenas: # zpool clear poolname