zpool clear for a Single Checksum Error?

[dorfoninternet]

I recently replaced 5 drives in my pool due to 4 of them failing S.M.A.R.T. tests and the 5th just having a sudden death. This was after a series of power outages over the last month or so. I also replaced the CableMatters SFF-8087::SFF-8087 with a genuine Supermicro cable. With so many drives failing (3 failed in one day) I thought it might be a bad cable. Though that CablesMatters cable had been in service for about 5 years without issue...

The day after getting through the last drive/cable replacement one of the new drives threw a checksum error. The pool immediately went into repairing mode. It finished the repair but the drive is still showing that 1 checksum error.

What are your thoughts? Run a zpool clear? Replace the drive? Put the original cable back?

pool: vol0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 16K in 18:21:38 with 0 errors on Sun Jul 18 18:21:39 2021
config:

Code:

 pool: vol0
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 16K in 18:21:38 with 0 errors on Sun Jul 18 18:21:39 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        vol0                                            ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/1224482e-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/1297ba9c-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/1315e6dd-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/6f2dd230-bb1b-11ea-ab54-0cc47a86a3e4  ONLINE       0     0     0
            gptid/140c3e88-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/147bebeb-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/1b48ba9f-e26b-11eb-b4d8-0cc47abab4b1  ONLINE       0     0     1
            gptid/157fce85-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/1684a5fc-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/179edafc-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/18b89709-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/716b97f6-e1b5-11eb-b4d8-0cc47abab4b1  ONLINE       0     0     0
            gptid/1adcdb80-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/9f74095c-9736-11e9-ac69-0cc47a86a3e4  ONLINE       0     0     0
            gptid/a30f7412-e6ed-11e9-bd5b-0cc47a86a3e4  ONLINE       0     0     0
            gptid/46c3563d-7aa1-11ea-91d8-0cc47a86a3e4  ONLINE       0     0     0
            gptid/1e596246-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/49f230f3-e05b-11eb-b4d8-0cc47abab4b1  ONLINE       0     0     0
            gptid/20902071-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0
            gptid/21a978a1-e136-11e6-b6f4-0cc47a86a3e4  ONLINE       0     0     0

errors: No known data errors

Code:

root@freenas01:~ # glabel status
                                      Name  Status  Components
gptid/439e1e3f-77bb-11eb-859b-0cc47abab4b1     N/A  ada0p1
gptid/4390265c-77bb-11eb-859b-0cc47abab4b1     N/A  ada1p1
gptid/1224482e-e136-11e6-b6f4-0cc47a86a3e4     N/A  da0p2
gptid/1297ba9c-e136-11e6-b6f4-0cc47a86a3e4     N/A  da1p2
gptid/1315e6dd-e136-11e6-b6f4-0cc47a86a3e4     N/A  da2p2
gptid/6f2dd230-bb1b-11ea-ab54-0cc47a86a3e4     N/A  da3p2
gptid/9f74095c-9736-11e9-ac69-0cc47a86a3e4     N/A  da4p2
gptid/1adcdb80-e136-11e6-b6f4-0cc47a86a3e4     N/A  da5p2
gptid/18b89709-e136-11e6-b6f4-0cc47a86a3e4     N/A  da6p2
gptid/179edafc-e136-11e6-b6f4-0cc47a86a3e4     N/A  da7p2
gptid/1684a5fc-e136-11e6-b6f4-0cc47a86a3e4     N/A  da8p2
gptid/21a978a1-e136-11e6-b6f4-0cc47a86a3e4     N/A  da9p2
gptid/20902071-e136-11e6-b6f4-0cc47a86a3e4     N/A  da10p2
gptid/1e596246-e136-11e6-b6f4-0cc47a86a3e4     N/A  da11p2
gptid/46c3563d-7aa1-11ea-91d8-0cc47a86a3e4     N/A  da12p2
gptid/a30f7412-e6ed-11e9-bd5b-0cc47a86a3e4     N/A  da13p2
gptid/140c3e88-e136-11e6-b6f4-0cc47a86a3e4     N/A  da14p2
gptid/157fce85-e136-11e6-b6f4-0cc47a86a3e4     N/A  da15p2
gptid/147bebeb-e136-11e6-b6f4-0cc47a86a3e4     N/A  da16p2
gptid/49f230f3-e05b-11eb-b4d8-0cc47abab4b1     N/A  da17p2
gptid/716b97f6-e1b5-11eb-b4d8-0cc47abab4b1     N/A  da18p2
gptid/1b48ba9f-e26b-11eb-b4d8-0cc47abab4b1     N/A  da19p2
gptid/9f65d774-9736-11e9-ac69-0cc47a86a3e4     N/A  da4p1
gptid/6f1288ff-bb1b-11ea-ab54-0cc47a86a3e4     N/A  da3p1
gptid/130d3302-e136-11e6-b6f4-0cc47a86a3e4     N/A  da2p1
gptid/128f5d4d-e136-11e6-b6f4-0cc47a86a3e4     N/A  da1p1
gptid/12194ec3-e136-11e6-b6f4-0cc47a86a3e4     N/A  da0p1

Code:

root@freenas01:~ # smartctl --all /dev/da19
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Pro
Device Model:     WDC WD6003FFBX-68MU3N0
Serial Number:    
LU WWN Device Id: 
Firmware Version: 83.00A83
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jul 18 23:23:12 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 648) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   133   133   054    Old_age   Offline      -       92
  3 Spin_Up_Time            0x0007   100   100   024    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       165
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       12
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       12
194 Temperature_Celsius     0x0002   136   136   000    Old_age   Always       -       44 (Min/Max 25/48)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       142         -
# 2  Short offline       Completed without error       00%       118         -
# 3  Short offline       Completed without error       00%        94         -
# 4  Short offline       Completed without error       00%        72         -
# 5  Short offline       Completed without error       00%        48         -
# 6  Short offline       Completed without error       00%        35         -
# 7  Short offline       Completed without error       00%        11         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Ericloewe]

The timing is awkward, but generally speaking a single checksum error is not a huge deal.
Run zpool clear and keep an eye on it.

 

[c77dk]

I would do as @Ericloewe suggests + do a long SMART test (and make sure to run it on all drives if not done)

 

[Ericloewe]

That actually reminds me of an important point: make sure all your disks are set for the SMART tasks, after having replaced them.

 

[dorfoninternet]

Thanks for the replies. I have the "all disks" checked for the S.M.A.R.T. tests. I just went in and validated that is still checked after all the replacements. I currently have the long S.M.A.R.T. test set to run monthly so I'll kick one off manually now.