Faulty hard drive makes system unresponsive

[cuvy]

Hi there!

One of my hard drive just died.

Code:

  pool: source05
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: resilvered 327G in 4h44m with 0 errors on Fri Jul  3 17:40:12 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    source05                                        DEGRADED     0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/ddb271f6-da25-11e4-8bfe-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/db319104-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/dbd2ec75-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/dc6c39c1-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/dd028adf-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/dd9a284c-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/de451451-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/deddd207-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/df85902d-a051-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
      raidz2-1                                      ONLINE       0     0     0
        gptid/33c28822-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/345ca565-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/491d5be2-eeb0-11e4-8bfe-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/35954b42-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5a52fdf2-d341-11e4-a970-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/e16be3e9-ccc9-11e4-88d2-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/3767933f-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/38689417-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/390826ee-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
      raidz2-2                                      ONLINE       0     0     0
        gptid/69bae444-0a16-11e5-a31d-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/59515a7e-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/59ecc290-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/7e120af2-d67b-11e4-a970-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5b265a03-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5bc01151-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/f71c27d5-1f54-11e5-bd8c-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5d062b46-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5da86c2f-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
      raidz2-3                                      DEGRADED     0     0     0
        gptid/8991e658-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/90fb1d51-989b-11e2-91fd-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/8ae02062-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/cbcf0ee5-ce51-11e4-88d2-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/4b054ed1-aa01-11e3-993a-0025904e8462  FAULTED      0    57     0  too many errors
        gptid/8cdc6e68-a052-11e1-96da-90e2ba0f1104  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/d7557b80-a0bd-11e3-8b78-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/5d3d3856-8db1-11e3-8b78-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/77594fd6-08a1-11e5-a31d-0025904e8462  ONLINE       0     0     0  block size: 512B configured, 4096B native
      raidz2-4                                      ONLINE       0     0     0
        gptid/9a420dd5-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/9b878e25-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/9cc3a285-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/9dc4c4e1-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/e8cf7521-1c0a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/a1b9ca77-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/a379da6f-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/a489a9f9-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
        gptid/a5bc5e59-16c0-11e5-a31d-0025904e8462  ONLINE       0     0     0
      raidz2-5                                      ONLINE       0     0     0
        gptid/1c771f26-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/1d53115a-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/4f41938a-21a4-11e5-bd8c-0025904e8462  ONLINE       0     0     0
        gptid/1f989ab7-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/20a2e7ff-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/21c772c7-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/22ccdfff-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/23e9531a-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0
        gptid/25333d87-1b8a-11e5-8388-0025904e8462  ONLINE       0     0     0

The WebGUI is not responding so I tried to use command line but glabel and gpart are both hanging. I'm pretty sure I know what device it is based on the server console:

However I need to know the serial number of the drive because I've labeled all my drive with their serial number.. Is there a way to find the serial number if glabel/gpart are not responding?

 

[cuvy]

camcontrol fails too.

Code:

~# camcontrol identify da26
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

 

[cuvy]

I found a debug configuration I had downloaded recently and I found the serial number in the hardware dump. Phew. That being said, do I have to make the drive offline or being FAULTY makes it offline automatically?

 

[cuvy]

When the status is FAULTY, it is already offline.

All good, resilvering now :) Hopefully this might help someone in the future.

Cheers!

 

[SweetAndLow]

Hmmm... I feel like things shouldn't go unresponsive. Maybe this would be a good bug?

 

[Ericloewe]

Since it's already faulted, it should not affect your system. My hunch is that there's a second misbehaving drive.

 

[cuvy]

I had bunch of disk alerts, I updated the system, rebooted and now all errors are gone. Nothing in the logs is showing since then... I'm monitoring it though.

 

[Ericloewe]

I had bunch of disk alerts, I updated the system, rebooted and now all errors are gone. Nothing in the logs is showing since then... I'm monitoring it though.

Error counts don't persist across reboots, so nothing else would be expected.