Error on a disk

[Max90]

I have noticed critical errors reported by TrueNAS. Can you help me on what to do in these cases? seems to be talking about the disk / dev / ada2.
These are the errors:
error
CRITICAL
Pool Volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
2022-02-20 00:12:30 (Europe / Rome)
Expel
notifications_active
CRITICAL
Device: / dev / ada2, 8 Currently unreadable (pending) sectors.
2022-02-25 20:01:12 (Europe / Rome)
Expel
notifications_active
CRITICAL
Device: / dev / ada2, 8 Offline uncorrectable sectors.
2022-02-25 20:01:12 (Europe / Rome)

what is happening? do i have to do some tests on the discs or do they need to be replaced?

 

[Etorix]

You can read the SMART report
smartctl -a /dev/ada2
run short and long tests
smartctl -t short /dev/ada2
smartctl -t long /dev/ada2
but the message already indicates that ada2 is developing bad sectors so replacement should be considered as soon as possible.
Take this opportunity to look at other drives as well.

 

[rogerh]

I'd just say that if the disk shows few or no errors with smartctl -a and the long test passes it is possible that the errors are arising elsewhere in hardware such as cables or possibly elsewhere in the hardware that you haven't told us about. It is recommended to give the forum details of your hardware when asking a question and it can make a difference to the answer. Have you done a scrub since the warning message? Whether you should or not depends a bit on the disk findings and how much redundancy you have.

 

[Max90]

You can read the SMART report
smartctl -a /dev/ada2
run short and long tests
smartctl -t short /dev/ada2
smartctl -t long /dev/ada2
but the message already indicates that ada2 is developing bad sectors so replacement should be considered as soon as possible.
Take this opportunity to look at other drives as well.

I'd just say that if the disk shows few or no errors with smartctl -a and the long test passes it is possible that the errors are arising elsewhere in hardware such as cables or possibly elsewhere in the hardware that you haven't told us about. It is recommended to give the forum details of your hardware when asking a question and it can make a difference to the answer. Have you done a scrub since the warning message? Whether you should or not depends a bit on the disk findings and how much redundancy you have.

I have a HP ProLiant Microserver hstns 5151 (AMD Turion™ II NEO N40L processor (1.5 GHz,15W, 2MB)) with 4 disk of 1 TB. FreeNAS-11.3-U5 to be updated but I am waiting for this reason.

this is the result of smartctl -a /dev/ada2:

smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162 Serial Number: Z1D4B8KH LU WWN Device Id: 5 000c50 0506cc3d0 Firmware Version: CC46 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Mar 12 12:43:17 2022 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 111) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 46103512 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 140 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 230859132 9 Power_On_Hours 0x0032 049 049 000 Old_age Always - 45464 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 140 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 074 051 045 Old_age Always - 26 (Min/Max 25/36) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 137 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 208 194 Temperature_Celsius 0x0022 026 049 000 Old_age Always - 26 (0 13 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 45466h+42m+27.810s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 67658962779 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 184096853658 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing

This is the result of smartctl -t short /dev/ada2:
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Short self-test routine immediately in off-linemode". Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 1 minutes for test to complete. Test will complete after Sat Mar 12 12:48:42 2022 CET Use smartctl -X to abort test.

for the long test he told me it will take 111 minutes .. but how can I see what it reports?
If i read on the disk results smart:
Short FAILED:
Remaining:
0.9
Lifetime:
45464
Error:
1953523104

Extended RUNNING:
Remaining:
0.9
Lifetime:
45465
Error:
N/A

 

[Redcoat]

but how can I see what it reports?

With

smartctl -a /dev/ada2

 

[Max90]

both tests fail:
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162 Serial Number: Z1D4B8KH LU WWN Device Id: 5 000c50 0506cc3d0 Firmware Version: CC46 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Mar 12 15:08:41 2022 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 113) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 111) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 46176296 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 140 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 230894753 9 Power_On_Hours 0x0032 049 049 000 Old_age Always - 45467 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 140 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 072 051 045 Old_age Always - 28 (Min/Max 25/36) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 137 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 208 194 Temperature_Celsius 0x0022 028 049 000 Old_age Always - 28 (0 13 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 45469h+07m+53.074s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 67661626923 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 184096854122 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 10% 45466 1953523104 # 2 Short offline Completed: read failure 90% 45464 1953523104 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 113) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 111) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 46176296 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 140 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 230894753 9 Power_On_Hours 0x0032 049 049 000 Old_age Always - 45467 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 140 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 072 051 045 Old_age Always - 28 (Min/Max 25/36) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 137 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 208 194 Temperature_Celsius 0x0022 028 049 000 Old_age Always - 28 (0 13 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 45469h+07m+53.074s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 67661626923 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 184096854122 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 10% 45466 1953523104 # 2 Short offline Completed: read failure 90% 45464 1953523104

 

[Max90]

SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[danb35]

Code tags would be a better way to post this output than the inline code you're using. But that aside, if it's failing SMART self-tests, IMO it's time to replace the disk. It's been running nonstop for over 5 years; it's fully depreciated. A couple of other points:

• A single-digit number of bad sectors, by itself, isn't grounds IMO for immediate disk replacement, if the number is stable. If it starts climbing, you should be looking for a replacement disk.
• Until you ran them during the time of this thread, you'd never run SMART self-tests on this disk. When you replace it, be sure to set up a regular test schedule--I run short tests daily, long tests weekly.
• When you get your replacement disk, be sure to burn it in to avoid a DOA disk eating your data. See Uncle Fester's Guide (link in my sig) for details.

 

[Etorix]

for the long test he told me it will take 111 minutes .. but how can I see what it reports?

Wait as long as it takes for the test to complete (large drives can take 10-12 hours to perform a long test) and run smartctl -a /dev/ada2 again ;)
Some drives abort the test and report failure if smartctl is invoked on them during a test, so you may try running the test again.
Anyway, this drive is over 5 years old and has 8 dubious sectors. Old age is kicking in; it's time to replace it. Hopefully the warning came early enough to replace the drive before it fails completely.

 

[Max90]

Code tags would be a better way to post this output than the inline code you're using. But that aside, if it's failing SMART self-tests, IMO it's time to replace the disk. It's been running nonstop for over 5 years; it's fully depreciated. A couple of other points:

• A single-digit number of bad sectors, by itself, isn't grounds IMO for immediate disk replacement, if the number is stable. If it starts climbing, you should be looking for a replacement disk.
• Until you ran them during the time of this thread, you'd never run SMART self-tests on this disk. When you replace it, be sure to set up a regular test schedule--I run short tests daily, long tests weekly.
• When you get your replacement disk, be sure to burn it in to avoid a DOA disk eating your data. See Uncle Fester's Guide (link in my sig) for details.

You're right, I've never taken the tests. I will definitely schedule the tests after I have replaced the disk.
Where do I read the number of bad sectors? anyway better, so I have the time to buy myself a record calmly.
What part of the guide am I interested in? I thought it was enough to remove the failed disk and put the new one, once the nas was turned off.

Wait as long as it takes for the test to complete (large drives can take 10-12 hours to perform a long test) and run smartctl -a /dev/ada2 again ;)
Some drives abort the test and report failure if smartctl is invoked on them during a test, so you may try running the test again.
Anyway, this drive is over 5 years old and has 8 dubious sectors. Old age is kicking in; it's time to replace it. Hopefully the warning came early enough to replace the drive before it fails completely.

I tried to retake the long test but it always fails..

 

[Etorix]

You're right, I've never taken the tests. I will definitely schedule the tests after I have replaced the disk.
Where do I read the number of bad sectors? anyway better, so I have the time to buy myself a record calmly.

5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 definitively discarded
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 suspicious (normally ends up under #5 after further attempts)

What part of the guide am I interested in? I thought it was enough to remove the failed disk and put the new one, once the nas was turned off.

The one about replacing drives…
If you have a spare SATA port, it is better to plug in the new drive while retaining the old one, and then use the GUI to replace (Storage>Pool>(wheel)>Status>(old disk)>(…)>Replace). That way, the pool resilvers whithout any loss of redundancy.

 

[Max90]

5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 definitively discarded
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 suspicious (normally ends up under #5 after further attempts)

The one about replacing drives…
If you have a spare SATA port, it is better to plug in the new drive while retaining the old one, and then use the GUI to replace (Storage>Pool>(wheel)>Status>(old disk)>(…)>Replace). That way, the pool resilvers whithout any loss of redundancy.

i can't find it, i found this:

/core/coretutorials/storage/disks/diskreplace/

that is fine?
maybe i can recover a 1tb hard disk at work, so i might already replace it.
Otherwise I will buy a 2TB disk at least, so I will increase the space in the future.
Don't risk losing any data during the rebuild?

 

[Etorix]

With a disk that is still readable and has a limited number of errors, you're actually in the "special case" where the old disk can remain online—and helps reducing the risk of data loss.
Make sure to burn-in the new drive and then proceed with the replacement.

 

[danb35]

Where do I read the number of bad sectors?

In addition to what @Etorix mentioned, attribute 198, "offline uncorrectable sectors" is also an indicator.

What part of the guide am I interested in?

Of Fester's guide? The HDD validation part.

 

[Max90]

In addition to what @Etorix mentioned, attribute 198, "offline uncorrectable sectors" is also an indicator.

Of Fester's guide? The HDD validation part.

Thank you.
I found some discs at work that I can use.
I can choose between WD Red, HP or Toshiba.
I had opted for the WD Red but have seen it spinning at 5400rpm while my barracudas spin at 7200rpm.
Do you think it is better to use it all the same being WD Red or opt for HP \ Toshiba running at 7200rpm?

 

[NugentS]

Make sure the WD isn't SMR

BTW - please post your system spec as per the forum rules

 

[Max90]

Make sure the WD isn't SMR

BTW - please post your system spec as per the forum rules

I wrote above the features.

The WD Red is the WD10EFRX-68FYTN0 model. It should be CMR

 

[Etorix]

I had opted for the WD Red but have seen it spinning at 5400rpm while my barracudas spin at 7200rpm.
Do you think it is better to use it all the same being WD Red or opt for HP \ Toshiba running at 7200rpm?

There's no need for drives to be identical, or even similar, although any vdev will obviously be limited by its least performing member.
Note, though that WD drives are merely marketed as "5400 rpm-class", whatever that means, and actually spin at 7200 rpm.

Western Digital is trying to redefine the word “RPM”

Well those measurements of rotational speed are just, like, your opinion, man...

arstechnica.com

 

[Max90]

There's no need for drives to be identical, or even similar, although any vdev will obviously be limited by its least performing member.
Note, though that WD drives are merely marketed as "5400 rpm-class", whatever that means, and actually spin at 7200 rpm.

Western Digital is trying to redefine the word “RPM”

Well those measurements of rotational speed are just, like, your opinion, man...

arstechnica.com

so do you recommend to use the same wd red?
Otherwise I use the hp \ toshiba which I am sure they run at 7200 rpm.
it is certainly more reliable than HP \ Toshiba.
Keep in mind that in addition to the data I also use it as a Plex server for 4K videos, I don't think it affects anyway.

 

[Max90]

In addition to what @Etorix mentioned, attribute 198, "offline uncorrectable sectors" is also an indicator.

Of Fester's guide? The HDD validation part.

this just to do the test before changing it?
I took the WD Red test and it's 2 years old and passed all the tests, including the extended SMART (from the WD software though).

I would like to understand what is the guide to follow to replace it..just replace from TrueNAS, remove the old disk and add the new one in its place? and does the reconstruction by itself? Can it be hot then?
I have never done such a thing thank you if you give me some advice .. Currently I have mastered NTFS MBR but I think TrueNAS will burn it once inserted anyway ..

unfortunately I have no other sata available..
Is this guide ok? https://www.truenas.com/docs/core/storage/disks/diskreplace/