Errors on 2 disks - not sure where to start

[wnoble]

Received this error today. Not sure where to start:

Mar 22 20:00:08 nas-5 smartd[2782]: Device: /dev/da2 [SAT], 24 Currently unreadable (pending) sectors
Mar 22 20:00:08 nas-5 smartd[2782]: Device: /dev/da4 [SAT], 16 Currently unreadable (pending) sectors
Mar 22 20:00:08 nas-5 smartd[2782]: Device: /dev/da4 [SAT], 2 Offline uncorrectable sectors



System Info:
Build: FreeNAS-9-10.1-U2
Platform: Intel Xeon CPU X5675 @ 3.07GHz
Memory: 12259MB

Volume: VMware has the disks with errors and is the production voluem (MainShare is not being used and can use the disks as needed)

 

[sretalla]

It looks like da2 and da4 are having issues...

You can start by running:
smartctl -a /dev/da2
smartctl -a /dev/da4

And share the output.

 

[wnoble]

Thanks so much for the fast response. Here is the output.

[root@nas-5] ~# smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD6002FRYZ-01WD5B0
Serial Number: K1G37GGB
LU WWN Device Id: 5 000cca 255c17924
Firmware Version: 01.01M02
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Mar 23 07:28:02 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 113) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 774) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 138 138 054 Pre-fail Offline - 100
3 Spin_Up_Time 0x0007 131 131 024 Pre-fail Always - 500 (Average 502)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 39
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 096 096 000 Old_age Always - 32288
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
192 Power-Off_Retract_Count 0x0032 099 099 000 Old_age Always - 1377
193 Load_Cycle_Count 0x0012 099 099 000 Old_age Always - 1377
194 Temperature_Celsius 0x0002 153 153 000 Old_age Always - 39 (Min/Max 20/45)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 24
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



[root@nas-5] ~# smartctl -a /dev/da4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD6002FRYZ-01WD5B0
Serial Number: K1G35UDB
LU WWN Device Id: 5 000cca 255c172f5
Firmware Version: 01.01M02
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Mar 23 07:29:24 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 113) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 763) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 138 138 054 Pre-fail Offline - 100
3 Spin_Up_Time 0x0007 140 140 024 Pre-fail Always - 470 (Average 469)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 39
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 1
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 096 096 000 Old_age Always - 32288
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
192 Power-Off_Retract_Count 0x0032 099 099 000 Old_age Always - 1376
193 Load_Cycle_Count 0x0012 099 099 000 Old_age Always - 1376
194 Temperature_Celsius 0x0002 171 171 000 Old_age Always - 35 (Min/Max 20/40)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 16
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 2
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[sretalla]

197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 24

For da2, this is an issue. As it's a pending operation, there is a possibility it's related to some interface errors and we don't see any other read/write/reallocate errors, so it's possible the disk is still good and we just have a cabling or controller error (maybe related to the other disk having issues?)

196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 16
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 2

For da4, this is more of an issue and is clearly internal to the disk. You should consider replacing this disk.

It also seems you're not running scheduled SMART tests on either of the disks (possibly not for the others either in that case). This could mean that there are more errors not yet detected.

 

[wnoble]

Thank you so much. This is stressing me out like there is no tomorrow.

I have 2 disks coming in for Tuesday.

Can I just replace da4 and the rebuild will automatically start? I don't think the controller is hot swap.
Or better yet, can I break down the volume "MainShare" and use one of those disks by doing a "replace"?

 

[sretalla]

The RAID controller is FreeNAS... FreeNAS doesn't just assume a replaced drive is intended for use in the same pool so you need to tell it so in the GUI. The procedure is well documented:

10. Storage — FreeNAS®11.3-U1 User Guide Table of Contents

www.ixsystems.com

 

[wnoble]

Ok. Got it (I think). It's been years since dealing with swapping disks. Looking at what disks are on the server, da7 is not being used at all. When I look at da4, the "replace" option is there already. And when I select "replace", da7 is available.

Logically from the guide, I should be able to simply replace da4 with da7 and all would resume with "resilvering" the array. Correct? Anything I need to look out for?

 

[wnoble]

To me da7 looks like it is a good candidate to use

[root@nas-5] ~# smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD6002FRYZ-01WD5B0
Serial Number: K1G36LYB
LU WWN Device Id: 5 000cca 255c175ee
Firmware Version: 01.01M02
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Mar 23 09:38:29 2020 EDT
recommended polling time: ( 780) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 138 138 054 Pre-fail Offline - 100
3 Spin_Up_Time 0x0007 135 135 024 Pre-fail Always - 486 (Average 487)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 38
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 096 096 000 Old_age Always - 32288
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 051 051 000 Old_age Always - 59302
193 Load_Cycle_Count 0x0012 051 051 000 Old_age Always - 59302
194 Temperature_Celsius 0x0002 150 150 000 Old_age Always - 40 (Min/Max 20/47)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[sretalla]

To me da7 looks like it is a good candidate to use

I agree. Although the forum recommends doing burn-in on new disks first, so perhaps consider that. (search for badblocks burnin test and there's plenty of instructions on how-to if you want to do that... it will take a long time)

Also note that you haven't run any SMART tests on that disk either... you should consider setting up that scheduled task if you love your data.

 

[wnoble]

Running a long test now. Going to take roughly 7 hours. Guess I just need to keep my fingers crossed until the morning before replacing da4. Or would it be safe to do the short test and make the replacement now.

Still searching for the scheduled tests. Found this -> https://www.ixsystems.com/community/threads/scrub-and-smart-testing-schedules.20108/

 

[sretalla]

The long test goes to each block, so if you want confidence in the disk, you need to do it.

https://en.wikipedia.org/wiki/S.M.A.R.T.#Self-tests

SMART drives may offer a number of self-tests:

Short Checks the electrical and mechanical performance as well as the read performance of the disk. Electrical tests might include a test of buffer RAM, a read/write circuitry test, or a test of the read/write head elements. Mechanical test includes seeking and servo on data tracks. Scans small parts of the drive's surface (area is vendor-specific and there is a time limit on the test). Checks the list of pending sectors that may have read errors, and it usually takes under two minutes.

Long/extended A longer and more thorough version of the short self-test, scanning the entire disk surface with no time limit. This test usually takes several hours, depending on the read/write speed of the drive and its size.

 

[sretalla]

Still searching for the scheduled tests. Found this -> https://www.ixsystems.com/community/threads/scrub-and-smart-testing-schedules.20108/

8. Tasks — FreeNAS®11.3-U1 User Guide Table of Contents

www.ixsystems.com

I'm not sure what else you need, but happy to help in addition to that.

 

[wnoble]

Understand completely. Thank you so much for your patience with me. I am just so stressed over this I am getting sick to my stomach.

If all goes well with the long test, I should be good doing a da4 to da7 "replace"?
Is there anything I should look for or possibly any kind of error I could see upon making the change?

Once this disk has been replaced, I will certainly look more into scheduled tests. But right now, I can hardly even think straight. Thanks again through this.

 

[sretalla]

Once this disk has been replaced, I will certainly look more into scheduled tests

Clearly the priority is to ensure the integrity of the pool now, worrying about scheduling tests after that's covered.

If all goes well with the long test, I should be good doing a da4 to da7 "replace"?

If you don't get any read/write errors during the long test (or any others), it's a good start (although officially not a burn-in).

Depending how critical your pool is, you can then go ahead if you're comfortable taking a small risk that the new disk will fail during or shortly after the addition to the pool. (if it's more critical, you will still want to do the badblocks testing in line with the burnin process https://github.com/Spearfoot/disk-burnin-and-testing)

Is there anything I should look for or possibly any kind of error I could see upon making the change?

During a resilver, a bunch of code is used which is common to the scrub process and resilver, so you're able to be confident that if the zpool status -v says you're good after resilvering is done, your data is in great shape.

You can run an additional scrub if it helps you sleep better.

 

[wnoble]

Long test on da7 went well without any errors. Attempted to do the replace and received the error below. I am 99% sure this disk was never used or ever used in a volume. Is it safe to force?

 

[sretalla]

Only you can be sure of your own disks. If you are confident it's empty, go ahead and force it.

 

[wnoble]

Thanks. After posting, I found a thread with same issue and they forced without problems. I proceeded with a force.

 

[wnoble]

Ok. "resilvering" completed. I did get 1 chksum error.

[root@nas-5] ~# smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD6002FRYZ-01WD5B0
Serial Number: K1G36LYB
LU WWN Device Id: 5 000cca 255c175ee
Firmware Version: 01.01M02
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Mar 24 11:25:46 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 113) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 780) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 138 138 054 Pre-fail Offline - 100
3 Spin_Up_Time 0x0007 135 135 024 Pre-fail Always - 486 (Average 487)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 38
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 096 096 000 Old_age Always - 32314
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 051 051 000 Old_age Always - 59321
193 Load_Cycle_Count 0x0012 051 051 000 Old_age Always - 59321
194 Temperature_Celsius 0x0002 139 139 000 Old_age Always - 43 (Min/Max 20/47)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 32302 -
# 2 Short offline Completed without error 00% 32290 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[wnoble]

For da2, this is an issue. As it's a pending operation, there is a possibility it's related to some interface errors and we don't see any other read/write/reallocate errors, so it's possible the disk is still good and we just have a cabling or controller error (maybe related to the other disk having issues?)

Still seeing the da2 error:
Mar 24 11:41:05 nas-5 smartd[82889]: Device: /dev/da2 [SAT], 24 Currently unreadable (pending) sectors

Should I wait and see what the happens on the next scrub? Scheduled for this weekend. I also added a short and long test in the SMART tasks.

 

[sretalla]

Still seeing the da2 error:

Keep watching it, if it goes up, replace the disk.

Checksum errors can be associated with cabling (or disks which misbehave... like when they have pending operations).

Check your cables carefully. You can do a zpool clear on the pool to remove the CRC for now so you can see easily if another one arrives.