16 Disk Errors at once

[M86]

Hello,

since a few weeks i have a freenas nas in use, started with some problems i got my system now up and running with the old drives from my prevois (qnap) nas..
But yesterday i got my first error(S!) - 16 at once!
It looks like its all the same, it happend all at the same time. A day before another drive gots kinda hot and freenas reported it - shouldnt happen after tomorrow since i got a new chase with hdd fans and more space.
I also bought another harddrive since the 4 4tb hdds are nearly full - but now im not sure... can i use the new hdd for extension, or should i use it (or prepare it) for replacement?

Heres the smartctl output:

Code:

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Enterprise Capacity 3.5 HDD
Device Model:     ST4000NM0035-1V4107
Serial Number:    ZC121GNK
LU WWN Device Id: 5 000c50 0a3114de2
Firmware Version: TN02
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Sep 21 00:50:28 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  575) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                       ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 411) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_                       FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   079   064   044    Pre-fail  Always       -                              85810768
  3 Spin_Up_Time            0x0003   094   093   000    Pre-fail  Always       -                              0
  4 Start_Stop_Count        0x0032   097   097   020    Old_age   Always       -                              3889
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -                              38
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -                              1495661096
  9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -                              17441 (229 150 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -                              0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -                              61
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -                              0
187 Reported_Uncorrect      0x0032   084   084   000    Old_age   Always       -                              16
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -                              0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -                              0
190 Airflow_Temperature_Cel 0x0022   065   044   040    Old_age   Always       -                              35 (Min/Max 20/35)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -                              1795
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -                              182
193 Load_Cycle_Count        0x0032   063   063   000    Old_age   Always       -                              75023
194 Temperature_Celsius     0x0022   035   056   000    Old_age   Always       -                              35 (0 16 0 0 0)
195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always       -                              85810768
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -                              0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -                              0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -                              0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -                              14848h+38m+07.167s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -                              18646693844
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -                              197785125297

SMART Error Log Version: 1
ATA Error Count: 16 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 16 occurred at disk power-on lifetime: 17437 hours (726 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                       .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 e8 ff ff ff 4f 00  24d+05:59:38.707  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:38.707  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:38.707  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:38.707  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:38.707  READ FPDMA QUEUED

Error 15 occurred at disk power-on lifetime: 17437 hours (726 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                       .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 e8 ff ff ff 4f 00  24d+05:59:36.581  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:36.581  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:36.581  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:36.581  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:36.581  READ FPDMA QUEUED

Error 14 occurred at disk power-on lifetime: 17437 hours (726 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                       .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 e8 ff ff ff 4f 00  24d+05:59:34.449  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:34.449  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:34.449  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:34.449  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:34.449  READ FPDMA QUEUED

Error 13 occurred at disk power-on lifetime: 17437 hours (726 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                       .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 e8 ff ff ff 4f 00  24d+05:59:32.281  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:32.281  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:32.281  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:32.281  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:32.281  READ FPDMA QUEUED

Error 12 occurred at disk power-on lifetime: 17437 hours (726 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle                       .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 e8 ff ff ff 4f 00  24d+05:59:29.999  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:29.999  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:29.999  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:29.999  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  24d+05:59:29.999  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):

 

[joeschmuck]

This hard drive is failing but not for what you reported.

First a warning to configure SMART Testing your drives... If you do not configure to run routine SMART testing then you are looking to be surprised by a drive failure. I recommend a daily SMART Short test and a weekly SMART Long test.

This drive has 38 bad sectors (ID5). Replace the drive. (See the link in my signature)

Also, run a SMART Long test and check the results. You could have other drives failing.

 

[M86]

Thanks!

The cmd for long term does not work for me.. always tells me wrong parameters :/
I did a short test on all other hdds, no errors there.
I setup daily short and weekly "offline" tests" - are that the long?

Are 38 bad sectors much? how that affect it the system and data? should i replace the disk right now, or can i wait some days to weeks for another to order?
I wanted to use the 5th hdd to expand the raid since i need some more space. but before the risk is to high ill use the 5th hdd as a replacement for the broken hdd.

How can i replace the disk in the raid? just switch to new hdd and add the disk to the pool? do i need to "start" the data writing to the new empty hdd? or is this automatic while in raid?

thanks a lot :)

 

[joeschmuck]

The cmd for long term does not work for me.. always tells me wrong parameters :/

it should be smartctl -t long /dev/ada0 as an example. For your drive it takes 411 minutes minimum to run.

I setup daily short and weekly "offline" tests" - are that the long?

Short self-test routine recommended polling time: ( 1) minutes.
Extended self-test routine recommended polling time: ( 411) minutes.
Conveyance self-test routine recommended polling time: ( 2) minutes.

The Offline test is not going to give you much value over the Short test and is definitely not the same as the Long/Extended test.
You need to set up SMART "Long" tests weekly and "Short" tests daily. Ignore Offline tests unless someone else tells you that you need to use it. Try using from the command line to run the long test as I indicated above.

Maybe you should post what the forum rules ask for, the hardware of your system and FreeNAS version you are running.

Are 38 bad sectors much? how that affect it the system and data? should i replace the disk right now, or can i wait some days to weeks for another to order?

38 sectors is bad. If you are able to run a SMART Long test I'm pretty confident it will fail. Anything more than 1 sector bad and many people will replace their drive, I wait until I have 5 sectors and the SMART Long test passes. If the test fails then I replace the drive because there is a hardware failure occurring. Without knowing your system configuration I can't tell you if your data is at risk, if you have a RAIDZ1 setup then you likely have good data. you can run a SCRUB to validate your data in good.

How can i replace the disk in the raid? just switch to new hdd and add the disk to the pool? do i need to "start" the data writing to the new empty hdd? or is this automatic while in raid?

Read the user manual. All the information you need it in there and well written.