Problem in pool (Device experienced unrecoverable error)

[Temereen64]

Greetings Everyone,

I was migrating data from one freenas server to my new server. About halfway through the transfer an error message popped up. Am I correct in reading that one of the drives had some type of read error? What steps should I take from here? I matched the gptid to the disk in question, is the disk one that I should replace/send back for warranty?

warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@truenas[~]# zpool status -v
pool: Bigstore
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

NAME STATE READ WRITE CKSUM Bigstore ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/af61b10e-58e6-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/b73fc259-58e6-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/b5f2200a-58e6-11eb-aa88-003990e6d7d4 ONLINE 1 0 0
gptid/b9674cfc-58e6-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/bbe88e08-58e6-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/ba3c9b91-58e6-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/0498b937-58e7-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/023fe247-58e7-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/075ga0ba-58e7-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/0867f9a6-58e7-11eb-aa88-003990e6d7d4 ONLINE 0 0 0
gptid/03abfa21-58e7-11eb-aa88-039590e6d7d4 ONLINE 0 0 0
gptid/0a404231-58e7-11eb-aa88-039590e6d7d4 ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0

 

[HoneyBadger]

Am I correct in reading that one of the drives had some type of read error? What steps should I take from here? I matched the gptid to the disk in question, is the disk one that I should replace/send back for warranty?

Entirely possible, but can we see the output of a smartctl -a /dev/daX command, posted within "CODE" tags?

A single read error could be an errant cosmic ray causing that "1 in 10^14 chance" read error, something as simple as a loose cable, or a drive giving you early warning on a potential failure.

 

[Temereen64]

Good evening everyone,

Thanks again to you all for your help. I have learned a heck of a lot on this forum great community and lots of very helpful information. Without this site I definitely would not have taken the dive into Freenas. Shout out to all the great people here. Anyway, is the output telling me that I need to replace the drive/send it back for warranty? or is it just one of those one in a million blips? What should I do from here? (Dismiss the alert or do I need to get to the store ASAP and get a new disk and send this for warranty?

The output of smartctl is listed below:

Code:

 root@truenas[~]# smartctl -a /dev/da2
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:  
LU WWN Device Id
Firmware Version: 82.00A82
User Capacity:  
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jan 19 23:48:07 2021 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (43860) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 465) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   162   162   021    Pre-fail  Always       -       6900
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       27
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       417
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       27
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       18
194 Temperature_Celsius     0x0022   122   118   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       4
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       212         -
# 2  Extended offline    Completed without error       00%       170         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay. 

 

[HoneyBadger]

The only value that's showing something there would be the UDMA_CRC_Error_Count - your reallocated/pending sectors are still zero, and even your seek error rate is zero. I'd check for a loose or kinked SATA cable inside the case, perhaps replace it as a precaution. It's a young drive and you seem to have periodic SMART tests running; so if it alerts again I would consider sending it back for warranty as drives usually follow the "bathtub curve" - either early in their life due to manufacturing defect, or late due to mechanical failure.

 

[Etorix]

The drive hasn't had a test for some time, so the next steps would be to run smartctl -t short /dev/da2, and then -t conveyance and finally -t long (this last one will take 8 hours, launch it overnight). Check smartctl -a afterwards. If the output has changed (other than spin time and reporting more tests, of course), you have a reason to inquire with WD about replacing the drive. Else it'a a cable or cosmic blip.

For peace of mind, you may want to run the same tests on all drives.