SOLVED Unable to complete enrypted drive unlock

[cedarosaur]

I began to notice very poor performance of my FreeNAS earlier today (taking ~20min to do an rsync dry run that would usually be done in 20 seconds). After trying a couple of times, I discovered that my previously unlocked encrypted volume had locked itself. When I try to unlock it, I get stuck in a loop trying to read from /dev/ada2 and am unable to complete unlocking the encrypted volume. From the logs, I can see there is definitely an issue with the disk and it will require replacing.

However, as I am unable to unlock the volume, I am not able to safely offline the affected drive as per the manual. I am also not able to check the zpool status, as it requires the volume to be unlocked.

I have so far attempted to unlock via password, with the recoery key, and detatching and re-attaching the volume. But it always gets stuck on /dev/ada2.

My question is: Is it possible to power-off the machine, and replace the affected drive WIHTOUT offlining it?

Some additional background:
The hardware is aging. i5 CPU and motherboard from 2011, RAM is a bit newer.
RAIDZ-1 volume with 5x4TB disks. 4/5 disks are about 5 years old, the other disk is 1 year old.
I replaced 1 drive in this volume about 12 months ago and resilvered successfully
Up until this week was running FreeNAS 9.3, but upgraded to 11.1 in the last few days.
All important data stored on the volume is backed up.

/var/log/messages:
https://pastebin.com/KQV5Dp6w

Output of smartctl -A:
https://pastebin.com/F33bKZbk

 

[joeschmuck]

Agree, your drive ada2 has failed. Do you not run normal testing on your drives? With so many sector errors I'd think that this problem would have been identified a long time ago.

I'd power off your system, replace drive Serial Number WD-WCC4E0172952, and follow the manual instructions after that point.

But to be honest, if it were me, I'd backup all my important data before proceeding. This would be my #1 thing to do because if it fails then you can at least restore your data.

 

[cedarosaur]

I was able to replace the failing drive and get the volume back up and running.

Once I had removed the failing drive, I was able to unlock the volume (in a degraded state). From there I was able to follow the manual to get the resilvering going.

In case anyone else has this issue, I have included the error lines here so its searchable:

Code:

Jun 15 18:27:49 andromeda (ada2:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 18 48 b8 c5 40 bd 00 00 00 00 00
Jun 15 18:27:49 andromeda (ada2:ahcich1:0:0:0): CAM status: ATA Status Error
Jun 15 18:27:49 andromeda (ada2:ahcich1:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jun 15 18:27:49 andromeda (ada2:ahcich1:0:0:0): RES: 41 40 48 b8 c5 40 bd 00 00 00 00
Jun 15 18:27:49 andromeda (ada2:ahcich1:0:0:0): Retrying command
Jun 15 18:27:53 andromeda (ada2:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 18 48 b8 c5 40 bd 00 00 00 00 00
Jun 15 18:27:53 andromeda (ada2:ahcich1:0:0:0): CAM status: ATA Status Error
Jun 15 18:27:53 andromeda (ada2:ahcich1:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jun 15 18:27:53 andromeda (ada2:ahcich1:0:0:0): RES: 41 40 48 b8 c5 40 bd 00 00 00 00