SOLVED Resilvered two failed encrypted drives from RaidZ2. Now I'm not able to Re-Key

[Spacemarine]

I'm running a 4x4TB RaidZ2 that is encrypted, but only contains test-data at the moment. Today, I received to more 4 TB drives, so now I want to run a 6x4TB RaidZ2. Since that is a perfect oppotunity to simulate the failure of two drives and try to recover from that, this is what I spent my evening with.

I simply pulled the plug on two random drives and replaced them with the two new drives. Everything resilvered perfectly, I just followed the manual. All my shares worked perfectly the whole time, so ZFS seems to have done it's job well.

HOWEVER: The next step before a reboot needs to be an encryption Re-Key and that's where things went wrong. When I tried to re-key, I got the following error message:

Code:

Oct  1 00:44:41 freenas notifier: 1+0 records in
Oct  1 00:44:41 freenas notifier: 1+0 records out
Oct  1 00:44:41 freenas notifier: 64 bytes transferred in 0.000089 secs (719666 bytes/sec)
Oct  1 00:44:41 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set passphrase on gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: No such file or directory.
]
Oct  1 00:44:41 freenas manage.py: [middleware.notifier:1241] [MiddlewareError: Unable to set passphrase on gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: No such file or directory.
]
Oct  1 00:44:41 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/d659fa19-67bf-11e5-96f8-0cc47a310a74: No such file or directory.
]]

What to do? What went wrong?
I'm glad this is just a test with no important data. However, it worries me that this might happen in the future to my real data.

 

[SweetAndLow]

Good for you to test it before trusting it but this is why i think people should avoid encryption unless the law requires that data to be encrypted for compliance reasons.

To help you more you need to provide freenas version. Also post the output of zpool status in code tags.

 

[Spacemarine]

I'm sorry, I forgot to specify the freenas version, I'm on stable-9.3.

Just out of curiosity I shut down the machine and rebooted it this morning. So my surprise, I was able to unlock the drives and connect to my shares. Even pool status told me that everything is ok:

Code:

  pool: test
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Oct  1 00:24:26 2015
config:

    NAME                                                STATE     READ WRITE CKSUM
    test                                                ONLINE       0     0     0
      raidz2-0                                          ONLINE       0     0     0
        gptid/f3d77352-67c1-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/d6c8fa93-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/d7b0b758-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0

errors: No known data errors

I thought that the two new drives were supposed to have a "wrong" encryption key and therefore shouldn't be able to be unlocked before the re-key? Or is the re-key just a precaution measure?
However, after the reboot I can perfectly re-key and change the passphrase, everything works perfectly now.

I know that it is safer and thus recommended, to leave freenas unencrypted and encrypt the data with truecrypt and other programs directly from the clients. However, there are several reasons, why this is unpractical to me. (Multiple clients accessing the same data at the same time, not all clients support the same encryption software, backup programs and other plugins need to run on the freenas and access the data)

 

[Spacemarine]

I just repeated the whole procedure and received the exact same error. Here is the error message:

Code:

Oct  1 22:53:50 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set passphrase on gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: No such file or directory.
]
Oct  1 22:53:50 freenas manage.py: [middleware.notifier:1241] [MiddlewareError: Unable to set passphrase on gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: No such file or directory.
]
Oct  1 22:53:50 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: geli: Cannot open gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74: No such file or directory.
]]

And zpool status:

Code:

  pool: test                                                                                                                       
state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Oct  1 22:53:09 2015                                                         
config:                                                                                                                            
                                                                                                                                   
        NAME                                                STATE     READ WRITE CKSUM                                             
        test                                                ONLINE       0     0     0                                             
          raidz2-0                                          ONLINE       0     0     0                                             
            gptid/5d3f672e-687e-11e5-8e47-0cc47a310a74.eli  ONLINE       0     0     0                                             
            gptid/d6c8fa93-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0                                             
            gptid/3ae081a2-687e-11e5-8e47-0cc47a310a74.eli  ONLINE       0     0     0                                             
            gptid/d7b0b758-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0                                             
                                                                                                                                   
errors: No known data errors 

Maybesome else can try to reproduce this? Maybe this is a systematic bug?

 

[Spacemarine]

This time a reboot didn't resolve the situation. Two of the drives were not readable due to wrong encryption key. (So the warning of the manual is correct, you should re-key BEFORE rebooting) Thanks to RaidZ2 I was able to mount the pool with only 2 out of 4 drives. So I resilvered them AGAIN and this time I was able to re-key.

So in both tests I would have been able to recover my data. However, this is still a very unsatifying situation...

 

[fta]

It's a known bug: https://bugs.freenas.org/issues/11356

 

[fta]

Just out of curiosity I shut down the machine and rebooted it this morning. So my surprise, I was able to unlock the drives and connect to my shares. Even pool status told me that everything is ok:

Code:

  pool: test
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Oct  1 00:24:26 2015
config:

    NAME                                                STATE     READ WRITE CKSUM
    test                                                ONLINE       0     0     0
      raidz2-0                                          ONLINE       0     0     0
        gptid/f3d77352-67c1-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/d6c8fa93-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/c5f4f5a9-67c1-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0
        gptid/d7b0b758-67bf-11e5-96f8-0cc47a310a74.eli  ONLINE       0     0     0

errors: No known data errors

I thought that the two new drives were supposed to have a "wrong" encryption key and therefore shouldn't be able to be unlocked before the re-key? Or is the re-key just a precaution measure?
However, after the reboot I can perfectly re-key and change the passphrase, everything works perfectly now.

This is expected. The only way you're going to screw up your pool after a replace is to re-key and have it fail, which is why the manual is way wrong about needing to do a re-key after a replace.