SOLVED Updated HBA firmware, can't decrypt volume, can't import volume

[Cody994]

Hey everyone.

I have a Dell T710 running Freenas 9.10 stable. I have an 15 drives (3 x 5 drive Z1s striped together) in an Dell MD1000 connected via an LSI SAS9200-8e. I purchased another SAS9200-8e to connect an HP D2700. I was getting warnings about my first card being the wrong firmware P7 and and that it should be P20. I shut everything down and upgraded both cards. I used a separate computer to update the cards just to play it safe.

I put both cards back in the T710 and connected one of each cable of the MD1000 to each HBA and did the same with the D2700. After I rebooted the system, I went to decrypt my volume and it wouldn't decrypt with the password or recovery key. I decided to detatch the volume and reimport, but now when I try to import, nothing shows up. Running zpool import and zpool import -v does nothing. Is there something I'm missing or anything I can do to recover my volume?

Thanks for any and all help in advance. This community is awesome!

 

[m0nkey_]

http://doc.freenas.org/9.10/storage.html#importing-an-encrypted-pool

 

[Cody994]

http://doc.freenas.org/9.10/storage.html#importing-an-encrypted-pool

I have tried this, but no disks show up. But if I hit volume manager, I can see all 15 drives to create a new volume.

 

[philhu]

Never, EVER do a firmware upgrade of a controller if you volume is encrypted. The read before and read after controller stuff can change, making the drives unreadable using the old read values

 

[m0nkey_]

Never, EVER do a firmware upgrade of a controller if you volume is encrypted. The read before and read after controller stuff can change, making the drives unreadable using the old read values

All the more reason not to use encrypted pools (I wasn't aware of this, so thanks).

This is actually pretty bad. A firmware update on a HBA should not have any impact what-so-ever, if it is, that's pretty screwed up.

@Cody994, suggest you open a bug ticket at https://bugs.freenas.org and report this. Something is clearly broken and needs to be fixed.

 

[Robert Trevellyan]

upgraded both cards

Was the pre-upgrade firmware IR mode or IT mode? What about the post-upgrade firmware?

 

[Cody994]

Was the pre-upgrade firmware IR mode or IT mode? What about the post-upgrade firmware?

I believe the pre-upgrade firmware was in IT mode, but when I went onto get the firmware here, I only see one version for P7. The new firmware was P20 IT.

Another note: I tried flashing my first card back to P7, but still had the same results.
Also, I have a new section under storage called multipathing after I upgraded the cards. I thought this was because I was hooked into two separate HBAs, but after I connect everything back to just one card, it still says multipathing.

 

[Cody994]

I ended up solving my issue. The multipathing seems to have been causing the issue. For each drive in that volume, I needed to run

gmultipath destroy x​

Where x is the disk name. After running this for all 15 disks, I was able to decrypt and import my volume. But now I am on back on the P7 firmware. I've been working on this remote, so when I get back I'll see what I can do to get everything updated and working.

Thanks for the help everyone.

 

[SweetAndLow]

Never, EVER do a firmware upgrade of a controller if you volume is encrypted. The read before and read after controller stuff can change, making the drives unreadable using the old read values

I'm not sure i believe this, do you have some evidence of things changing from controller upgrade when in IT mode?

 

[Ericloewe]

I'm not sure i believe this, do you have some evidence of things changing from controller upgrade when in IT mode?

I have to agree, it's a rather dubious statement.

 

[philhu]

I have to agree, it's a rather dubious statement.

I did not say in IT mode. I said in general. Dell Perc 5/i and c100 controllers were notorious for changing their caching between firmware upgrades. So I just never do it as a rule of thumb

 

[Ericloewe]

I did not say in IT mode. I said in general. Dell Perc 5/i and c100 controllers were notorious for changing their caching between firmware upgrades. So I just never do it as a rule of thumb

Ah, well, with HW RAID and their shenanigans, all bets are off.

With direct-attach (that's what the da designator stands for), this should never be a problem.