Volume DEGRADED, no SMART failure.

dvc9 · Jun 26, 2019

Hey!
I have a bit of a bug here. We have one VDEV, that is degraded,
but mostly all disks shows great stats in SMART, and SMART -short tests.
One drive /dev/da46 did not pass smart, so we are now replacing it.

I have printed out here gpart, camcontrol, zpool status, and messages.

FreeNAS-11.2-U3
Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (12 cores)
at the moment only 32 GB RAM. ( ECC )

What have triggered it ?
We did upgrade the RAM, but now we have taken out it all just to leave 2 RAM peaces, just in case its RAM.
We have installed a SolarFlare NIC 10 GbE.

that's the only two things done lately.

So the question, The pool works even if its degraded...
but how can I identify and clear the errors.

zpool status

Code:

root@storage01[~]# zpool status
  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:39 with 0 errors on Thu Jun 20 03:45:39 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da59p2    ONLINE       0     0     0

errors: No known data errors

  pool: storage
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jun 26 17:03:45 2019
    4.71T scanned at 1.41G/s, 2.83T issued at 867M/s, 350T total
    85.0G resilvered, 0.81% done, 4 days 20:44:08 to go
config:

    NAME                                              STATE     READ WRITE CKSUM
    storage                                           DEGRADED     0     0    49
      raidz3-0                                        ONLINE       0     0     0
        gptid/8e01b0d1-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/c963ad9f-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/cc5b0b90-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/cf4575fc-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/cfde91dd-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/d06725f3-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/d0e465af-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/d9cd9dac-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/dcca36b4-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/dfd2f587-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
        gptid/e2c9b66d-561e-11e9-bf14-00155d00d702    ONLINE       0     0     0
      raidz3-1                                        DEGRADED     0     0    98
        spare-0                                       DEGRADED     0     0     0
          gptid/e36cea07-561e-11e9-bf14-00155d00d702  DEGRADED     0     0     0  too many errors
          gptid/e8a41b2e-9805-11e9-b951-0cc47af6f6ac  ONLINE       0     0     0
        gptid/e68cd9ab-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        replacing-2                                   ONLINE       0     0     0
          gptid/e999f903-561e-11e9-bf14-00155d00d702  ONLINE       0     0     0
          gptid/59feb134-9805-11e9-b951-0cc47af6f6ac  ONLINE       0     0     0
        gptid/ec8eaa6b-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/ef9c22e2-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     1  too many errors
        gptid/f2aff507-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/f5b7357f-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/f8ce24f9-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/fbdc0edf-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/fc7aca89-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
        gptid/ffab9d10-561e-11e9-bf14-00155d00d702    DEGRADED     0     0     0  too many errors
      raidz3-3                                        ONLINE       0     0     0
        gptid/863e7c71-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/8f6d7470-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/98ae9014-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/a1eadaed-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/aad285d5-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/b3b03999-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/bcec7d49-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/c71aad9f-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/d07b3742-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/dab4672c-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/e3e20440-6125-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
      raidz3-4                                        ONLINE       0     0     0
        gptid/1830b175-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/26e9e6c9-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/354c43a9-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/43a99e42-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/523971eb-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/60dc2c21-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/6f7b8a59-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/7e036820-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/8c9dbc5c-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/9b71df84-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/aa4201e0-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
      raidz3-5                                        ONLINE       0     0     0
        gptid/c05e896f-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/cf4055f8-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/de3a0da0-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/ed4898f2-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/fc5d66d3-6126-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/0b2889ed-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/1a4dd43b-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/292eb181-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/38274469-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/4709b007-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
        gptid/55f09b9b-6127-11e9-b0b7-0cc47af6f6ac    ONLINE       0     0     0
    logs
      mirror-2                                        ONLINE       0     0     0
        gptid/f4282a88-6084-11e9-a3f5-0cc47af6f6ac    ONLINE       0     0     0
        gptid/fb3c0c8f-6084-11e9-a3f5-0cc47af6f6ac    ONLINE       0     0     0
    spares
      2145511691674371321                             INUSE     was /dev/gptid/e8a41b2e-9805-11e9-b951-0cc47af6f6ac

errors: 2 data errors, use '-v' for a list

dvc9 · Jun 26, 2019

camcontrol devlist

Code:

root@storage01[~]# camcontrol devlist
<LSI SAS3x48 0a01>                 at scbus1 target 9 lun 0 (pass0,ses0)
<LSI SAS3x48 0a01>                 at scbus1 target 10 lun 0 (pass1,ses1)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 11 lun 0 (pass2,da0)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 12 lun 0 (pass3,da1)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 13 lun 0 (pass4,da2)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 15 lun 0 (pass5,da3)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 16 lun 0 (pass6,da4)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 18 lun 0 (pass7,da5)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 19 lun 0 (pass8,da6)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 20 lun 0 (pass9,da7)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 21 lun 0 (pass10,da8)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 22 lun 0 (pass11,da9)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 23 lun 0 (pass12,da10)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 24 lun 0 (pass13,da11)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 26 lun 0 (pass14,da12)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 27 lun 0 (pass15,da13)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 28 lun 0 (pass16,da14)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 29 lun 0 (pass17,da15)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 30 lun 0 (pass18,da16)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 31 lun 0 (pass19,da17)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 32 lun 0 (pass20,da18)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 33 lun 0 (pass21,da19)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 34 lun 0 (pass22,da20)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 35 lun 0 (pass23,da21)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 36 lun 0 (pass24,da22)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 37 lun 0 (pass25,da23)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 38 lun 0 (pass26,da24)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 39 lun 0 (pass27,da25)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 40 lun 0 (da26,pass28)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 41 lun 0 (pass29,da27)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 42 lun 0 (pass30,da28)
<ATA TOSHIBA MG04ACA6 FS3B>        at scbus1 target 43 lun 0 (pass31,da29)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 44 lun 0 (pass32,da30)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 71 lun 0 (pass33,da31)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 72 lun 0 (pass34,da32)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 73 lun 0 (pass35,da33)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 74 lun 0 (pass36,da34)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 75 lun 0 (pass37,da35)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 76 lun 0 (pass38,da36)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 77 lun 0 (pass39,da37)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 78 lun 0 (pass40,da38)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 79 lun 0 (da39,pass41)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 80 lun 0 (pass42,da40)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 81 lun 0 (pass43,da41)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 82 lun 0 (pass44,da42)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 83 lun 0 (pass45,da43)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 84 lun 0 (pass46,da44)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 85 lun 0 (pass47,da45)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 86 lun 0 (pass48,da46)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 87 lun 0 (pass49,da47)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 88 lun 0 (pass50,da48)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 89 lun 0 (pass51,da49)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 90 lun 0 (pass52,da50)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 91 lun 0 (pass53,da51)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 92 lun 0 (pass54,da52)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 93 lun 0 (pass55,da53)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 94 lun 0 (pass56,da54)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 95 lun 0 (pass57,da55)
<ATA ST14000NM0018-2H SN02>        at scbus1 target 96 lun 0 (pass58,da56)
<ATA SAMSUNG MZ7KM240 003Q>        at scbus13 target 4 lun 0 (pass59,da57)
<ATA SAMSUNG MZ7KM240 003Q>        at scbus13 target 5 lun 0 (pass60,da58)
<Kingston DataTraveler 3.0 >       at scbus15 target 0 lun 0 (pass61,da59)

gpart show

Code:

root@storage01[~]# gpart show
=>         40  11721045088  da0  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  27344764848  da1  GPT  (13T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  27340570456    2  freebsd-zfs  (13T)

=>         40  11721045088  da2  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da3  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  27344764848  da4  GPT  (13T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  27340570456    2  freebsd-zfs  (13T)

=>         40  11721045088  da5  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da6  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da7  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da8  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da9  GPT  (5.5T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  11716850696    2  freebsd-zfs  (5.5T)

=>         40  11721045088  da10  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da11  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  11721045088  da12  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da13  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da14  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da15  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da16  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  11721045088  da17  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da18  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da19  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  11721045088  da20  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da21  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da22  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da23  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da24  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  11721045088  da25  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da27  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  11721045088  da28  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  11721045088  da29  GPT  (5.5T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)

=>         40  27344764848  da30  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  27343716272  da31  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da32  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da33  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da34  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da35  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da36  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da37  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da38  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da40  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da41  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da42  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da43  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da44  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da45  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da46  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da47  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da48  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da49  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da50  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da51  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27343716272  da52  GPT  (13T) [CORRUPT]
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27339521880     2  freebsd-zfs  (13T)

=>         40  27344764848  da53  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  27344764848  da54  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  27344764848  da55  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  27344764848  da56  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>       40  468862048  da57  GPT  (224G)
         40         88        - free -  (44K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  464667656     2  freebsd-zfs  (222G)

=>       40  468862048  da58  GPT  (224G)
         40         88        - free -  (44K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  464667656     2  freebsd-zfs  (222G)

=>      40  30218768  da59  GPT  (14G)
        40    532480     1  efi  (260M)
    532520  29655040     2  freebsd-zfs  (14G)
  30187560     31248        - free -  (15M)

=>         40  27344764848  da26  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

=>         40  27344764848  da39  GPT  (13T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  27340570456     2  freebsd-zfs  (13T)

/var/log/messages

Code:

Jun 26 13:34:25 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 13:34:25 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 14:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 14:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 14:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 14:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 14:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 14:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 14:34:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 14:34:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 15:01:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 15:01:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 15:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 15:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 15:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 15:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 15:34:39 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 15:34:39 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 16:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 16:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 16:04:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 16:04:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 16:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 16:31:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 16:34:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 16:34:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 17:01:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 17:01:44 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 17:03:08 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=12839626188744565204
Jun 26 17:03:32 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=756982453269821394
Jun 26 17:04:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 17:04:38 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 17:06:30 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=11704843481163072900
Jun 26 17:08:04 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=11252445962263860787
Jun 26 17:08:17 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=4920594368887248432
Jun 26 17:08:33 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=4747051055621338426
Jun 26 17:09:00 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=15336992827793764220
Jun 26 17:09:28 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=9070670237534172754
Jun 26 17:09:54 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=6248470898710237435
Jun 26 17:10:11 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=1891426674323798208
Jun 26 17:11:10 storage01 ZFS: vdev state changed, pool_guid=10470013019122752882 vdev_guid=4604440798865722533
Jun 26 17:31:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 17:31:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 17:34:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 17:34:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 18:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 18:01:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 18:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 18:04:37 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 18:13:28 storage01 ctld[2665]: exiting on signal
Jun 26 18:31:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 18:31:43 storage01 smartd[25629]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors
Jun 26 18:34:40 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Currently unreadable (pending) sectors
Jun 26 18:34:40 storage01 smartd[27206]: Device: /dev/da46 [SAT], 8 Offline uncorrectable sectors

dvc9 · Jun 26, 2019

We had just updated to 128 GB RAM, and one of the chips where faulty.
We then downgraded to 32 GB RAM, and it did not get new checksum errors,
but the old ones where still there...

I therefor believed that its a "lag" of some sorts, as the files where idle,
and the corruption has actually been removed.

So I followed up on Oracles documentation. ( Repairing ZFS Storage Pool-Wide Damage )

and ran the
zpool clear -F PoolNAME

This is as the errors where only checksum,
no smart errors, and no disk errors that I could find.

If the error occurs again, then I must reconsider.

Pitfrr · Jun 26, 2019

It's a good practice to burn-in new hardware (especially RAM and HDD). Did you burn-in the new memory before using it?
How did you identify the faulty memory?
To know if this could have been detected by a burn in...

danb35 · Jun 26, 2019

So zpool clear doesn't actually fix anything, it just takes away the report of errors. A better thing to have done would have been a scrub, as it would check if the errors had actually been resolved (and if so, reset the counter).

dvc9 · Jun 26, 2019

Yeah, so did not test the RAM before adding it ( shame on me )
The scrub will be started as soon as the Resilver of the failed drive is done.

Important Announcement for the TrueNAS Community.

Volume DEGRADED, no SMART failure.

dvc9

Explorer

dvc9

Explorer

dvc9

Explorer

Pitfrr

Wizard

danb35

Hall of Famer

dvc9

Explorer

Similar threads

Important Announcement for the TrueNAS Community.

Volume DEGRADED, no SMART failure.

dvc9

Explorer

dvc9

Explorer

dvc9

Explorer

Pitfrr

Wizard

danb35

Hall of Famer

dvc9

Explorer

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Volume DEGRADED, no SMART failure."

Similar threads