Error 22? Failing drive?

[EvanVanVan]

I received the following error emails this morning.

Code:

freenas.local kernel log messages:
> (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 ca be b8 c0 00 00 08 00
> (da4:mps0:0:4:0): CAM status: SCSI Status Error
> (da4:mps0:0:4:0): SCSI status: Check Condition
> (da4:mps0:0:4:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
> (da4:mps0:0:4:0): Info: 0xcabeb8c0
> (da4:mps0:0:4:0): Error 22, Unretryable error
> GEOM_ELI: Crypto WRITE request failed (error=22). gptid/2c67faef-b790-11e3-b9da-002590d65107.eli[WRITE(offset=1739418796032, length=4096)]

-- End of security output --

Code:

Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  7.19G  2.32G  4.86G         -      -    32%  1.00x  ONLINE  -
volume1       21.8T  10.6T  11.1T         -     7%    48%  1.00x  ONLINE  /mnt

  pool: volume1
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 4K in 0h0m with 0 errors on Thu Mar 24 18:55:46 2016
config:

        NAME                                                STATE     READ WRITE CKSUM
        volume1                                             ONLINE       0     0     0
          raidz2-0                                          ONLINE       0     0     0
            gptid/2a0e67aa-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2a82eeea-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2afb7434-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2b79a9e5-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2bf1472f-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2c67faef-b790-11e3-b9da-002590d65107.eli  ONLINE       0     1     0
            gptid/2cedc0a0-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0
            gptid/2d7a9191-b790-11e3-b9da-002590d65107.eli  ONLINE       0     0     0

errors: No known data errors

-- End of daily output --

I see the da4 HDD had the error, but I'm not sure what I do next. What is Error 22? The link in the second email talks about the drive failing.

Thanks

Specs:
SUPERMICRO MBD-X9SCL-F-O Intel Xeon E3 Server
Intel Xeon E3-1230 V2
Kingston Technology ValueRAM 32GB DDR3 1600MHz PC3 12800 ECC
WD RED (8x) 3GB
IBM ServeRAID M1015 (crossflashed into IT Mode) (All eight of my HDs are plugged into it)

FreeNAS 9.3-STABLE

edit: cleanup

 

[joeschmuck]

Check out the Hard Drive Toubleshooting Guide in the troubleshooting guide section of the forums. You will end up running a SMART long test on all your drives and then examine the results. Once you have done these steps, if you have any further questions/comments, please post them here in this thread. I'd like to know if the guide helps or if I need to make it more clear.

https://forums.freenas.org/index.ph...uide-basic-common-failures.41026/#post-260852

 

[EvanVanVan]

Awesome, thanks will do. I should have realized that was the procedure from last time, but I only saw it quickly this morning and didn't find much on Error 22 with a quick Google search.

 

[joeschmuck]

I'm not sure what you will find out, I'm not sure what error 22 is either but the guide should at least identify if you have a physical drive failure or not.

Your LSI card, is it flashed to P20 firmware?

Additionally, ensure you backup all your important data before you reflash your LSI card or swap out a drive, it looks like you have encryption (I could be wrong) but why risk loosing all your data is all I say.

 

[EvanVanVan]

I'm running a long test on da4. I won't see the results until tomorrow.

I just ran sas2flash -listall and it looks like the LSI card's firmware is only P16. I thought it was higher, but when I updated to 9.3 I had the "Firmware version match" error/alert and downgraded the firmware from P18 to P16. Has FreeNas started using P20? Keep in mind I'm only on 9.3 stable (FreeNAS-9.3-STABLE-201506042008), not 9.3.1. I could update (FreeNAS and LSI's firmware) after this all gets settled though.

I am encrypted. In my naivety when setting up my pool, I didn't realize that encryption caused more of a hassle than it's worth for my uses (since I'm not storing anything truly important).

I shouldn't lose anything flashing, correct? I think I was OK disconnecting and reconnecting the drives the last time while downgrading.

Thanks for the help. So far so good on the guide too, so thanks for that!

 

[joeschmuck]

1) Yes, FreeNAS now recommends using P20 firmware in the more current version of 9.3 Stable (since mid-2015).
2) Upgrading your controller will have no impact on the safety of your data, unless human error gets in the way.
3) I found one reference on the internet to your error 22 issue and the problem seemed to be tracking to the LSI firmware. It had nothing to do with FreeNAS but sometimes you need to take that leap to find these difficult problems.

With respect to your pool being encrypted, I would eventually copy off all your data and destroy the pool and recreate it without encryption and save the headache, although you might wait for FreeNAS 10 to come out, I hear there may be an easier implementation of encryption however I don't know any details.

Hopefully the drive tests good and you can upgrade the firmware and all is good in the world, and it's the quick fix too.

 

[EvanVanVan]

Aright, the drive long tested fine and I'm safely upgraded to FreeNAS-9.10-STABLE-201603252134 (412fb1c) and LSI FW P20. The critical alert ended up clearing after my post 9.10 update reboot. Hopefully it doesn't come back.

Thanks again. The guide was very helpful, especially since I had struggled to search/learn all the same information in the past, but now it was conveniently located all in one place and I didn't need to second guess myself all over again.

Now, I think I'm going to follow this guide (https://forums.freenas.org/index.ph...om-a-zfs-volume-while-keeping-the-data.16467/) and remove encryption once and for all... seems straight forward enough.

 

[joeschmuck]

That encryption removal is for a Volume, not a Pool.;) Backup your data before you try anything like this!

 

[EvanVanVan]

I thought I understood the difference between a volume and the pool, but maybe not lol. From the guide it seems like it's removing the encryption/(need for the geli.key entirely) from each hard drive one at a time, until it's removed completely, correct?

I would think the guide OP is correct that with RAID2Z and healthy drives it should be relatively safe to complete the process?

 

[danb35]

That encryption removal is for a Volume, not a Pool.;) Backup your data before you try anything like this!

Come again? A volume is a pool, and a pool is a volume. There's no difference between the two terms.

 

[joeschmuck]

Like to hear how it comes out.

 

[EvanVanVan]

Well Joe, three days later and I'm fully un-encrypted! Thanks for your help throughout the entire process.

Code:

[root@freenas ~]# zpool status                                                                                                     
  pool: volume1                                                                                                                    
state: ONLINE                                                                                                                     
  scan: resilvered 1.30T in 7h4m with 0 errors on Tue Mar 29 14:30:13 2016                                                         
config:                                                                                                                            
                                                                                                                                   
        NAME                                            STATE     READ WRITE CKSUM                                                 
        volume1                                         ONLINE       0     0     0                                                 
          raidz2-0                                      ONLINE       0     0     0                                                 
            gptid/2a0e67aa-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2a82eeea-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2afb7434-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2b79a9e5-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2bf1472f-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2c67faef-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2cedc0a0-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
            gptid/2d7a9191-b790-11e3-b9da-002590d65107  ONLINE       0     0     0                                                 
                                                                                                                                   
errors: No known data errors

 

[joeschmuck]

I'm very glad it worked out and I actually think your data is safer now. Replacing an encrypted drive has just such a high risk of failure.