Can't import/unlock encrypted zpool anymore

[arameen]

Background
Lately i had issues with one of my drives in my encrypted zpool. I was getting read and writeerrors and pool degradation. After doing some smarttests, short and long one, nothing seemed strange. I cleared the faults. But freeNAS kept complaining and degrading the pool and after a few clearings of every second day i removed the drive from the system. Put it in my PC and let Seagate seatools check the driver very carefully. Nothing wrong this time either.
I put back the drive to my system that continues to complain about my drive, and suddenly even another drive. Same issue, only read and write errors, even day or second day. no CRC errors.
I suspected the cable and tried to connect different drivers to different SATA ports, didnt help much. suspected my IBM ServeRAID M1015.
I got tired of trying to figure this out and orderd a new drive to substitute the one of with most faults, most convinient way to find out if the drive is faulty or not. Suddenly today from nowhere one of the USB sticks got faults too and my boot pool got degraded.
I tried to remove that faulty USB stick, but apperently removed the healthy one. (there is no easy way to see what usb stick is what dev when having dual boot). After putting back the USB stick and restarting, freeNAS won't boot anymore from any of those 2 USB sticks. it was telling me this is a nas disk, not a boot disk. I found it very strange that both USB sticks stopped booting when they before removal where booting perfectly. Installed freeNAS 9.10 on a new usb stick, at that moment i wasn't very sure what version of FreeNAS i had. So when i booted the system with the new installed 9.10 i got warnings about "firmware verison 16 doesn't not support driver version 20". I even managed to try to import my encrypted zpool once, but it didnt work out. And I realized I have to downgrade my freeNAS version because I had no time to update the firmware of the IBM M1505.
I installed the version 9.3 of freeNAS, the one I had last time working, and booted the system. My first zpool was found without any issues as earlier. But my second and encrypted zpool could not be unlocked anymore.
No matter what I tried I didn't manage to unlock the volume. I uploaded my previous freeNAS configuration prior to unlocking, didn't help. I tried with earlier configurations, didn't help. I tried to detach and import the volume, with different dated configurations, didn't help. I used several different backup copies of the encryption key, stored on different locations, in case any could been corrupted. IT didnt help. I tried all this steps with one suspected faulty disc disconnected and later both suspected faulty disk disconnect, in vain. I looked on the forum and used google, not much help when it comes to issues with importing encrypted zpools. Only thing I realized so far is, that I should never have encrypted my zpool.
I never had any issues before to be honest, I have substituted drives on the encrypted zpool without issues. But all that is to no help when I can not unlock my zpool anymore.
So what can I do now?
As mentioned, I do have copies of the key. Still freeNAS says when i try to unlock the volume: "Unable to geli attach gptid/........ geli: Cannot open keyfile /data/geli/......: No such file or directory."
The key can not be wrong. So what could be wrong ? yesterday the pool worked perfectly, the boot volumes stop booting today and suddenly its impossible to reimport an encrypted volume. I doubt this is supposed to work like this with freeNAS encryption. I mean if I had missed to backup the key, i could understand it. But several backups of the keys and configruations makes it hard to realize that I did something wrong and i should start accept several TBs of data is gone.
I read somewhere that it could be possible to extract the key from the old USB stick. I could try if i knew the commands. Want to try it, even thought i think something else is wrong here. Why would that key be the one to unlock while all the copies I have are suddenly not unlocking the volume?
Could the faulty drives or switching SATA connections to drives affect? its hard to think that could affect ?

Problem
So guys, what can I do now ? I am stuck with freeNAS telling me when i try to import the volume
"Unable to geli attach gptid/........ geli: Cannot open keyfile /data/geli/......: No such file or directory."
while i am sure i have the right key that worked hours ago when everything was working :(



My hardwaresetup:
SuperMicro MBD-X10SL7-F-O
Intel Core i3-4150 3,5GHz 4MB Socket 1150 (encryption support)
32GB ECC RAM (4x Samsung 8GB DDR3L ECC 1600MHz 1.35V UDIMM)
IBM ServeRAID M1015 (IT Mode)
Encrypted pool: 11x 4TB Seagate NAS drives

 

[m0nkey_]

Did you download a copy recovery key as per the documentation when you created your pool? Have you replaced a drive since and performed a re-key?

If not, your only option is to recover from backup.

 

[arameen]

Yes i did download several copys of the key to different locations as I mentioned before.
What do you mean with re-key ? I saw you linked to latest version of freeNAS. I am using FreeNAS 9.3
Yes I have replaced drives since the creating of the encrypted pool and saved the key. Its never been any issue and the unlocking after adding disks more than once have always been working until today.

 

[m0nkey_]

If the keys are not working, your only recourse is to restore from backup.

 

[arameen]

I doubt there is no more to do.
I mean everything worked hours ago and now suddenly the keys or something else is not valid anymore

By the way, there is no backup, this was the backup. With several copies of the keys, i thought nothing can happen except a fire or earthquake

 

[arameen]

Someone on the forum adviced me to try to unlock the drives from shell. I am not used at all to the shell or know many commands, i mostly use freeNAS through the GUI and I choosen freeNAS bacause of ZFS.
Anyway he told me to use "geli attach -p -k any.key ada1p2" And that is what i did on all 10 drives.

1 drive of the 11, mentioned earlier, is disconnected physically since before. The reason is that freeNAS wouldnt boot with it connected. This is the first eventually failing drive that freeNAS complained about.
Remaning is 10 drives. I tried to unlock/decrypt those 10 from shell with that command and got the following results:
- 1 would not unlock with reason: "Cannot read metadata from da... Invalid argument" That is the second propably failing disc that freeNAS complained about as i mentioned in my first post. So I could understant why this one is not unlocking.
- 5 drives wouldn't unlock with the reason "Wrong key for da.... " Same as what happens when i try to unlock through the GUI for alla drives.
- 4 drives SEEM to unlock. I don't recieve "wrong key" message for those when unlocking, I don't recieve any message at all when i execute the unlock command. So I assume those 4 got unlocked.

So 5 disk will not unlock with the key while 4 will unlock with the same key.
How is that possible?
What can I do now?
I still have the 2 USB sticks that I used before this all started, the ones that could unlock the whole pool without issues. Can they be used

I need the 5 disks unlocked before I can import the whole pool (in a degraded state, 9 of 11 disks in a raidz3 pool) and start moving the data somewhere else.