SOLVED Repeating message from TrueNAS - what does it mean?

[NumberSix]

Hi
Like most home TrueNAS users, I run my system headless almost all of the time. Unlike most home TrueNAS users, I'm not too hot on understanding or using the CLI. So when I attached a monitor to TrueNAS lately, the screen is always filled with the following message about something that's corrupted (I think?), although the system appears to function otherwise normally.

What does this output mean, and what action should I take to address it?

Thank you.

 

[Kris Moore]

Means a bad disk most likely, ada2 is throwing errors. Off chance it could be cable or the motherboard as well (If this is onboard SATA)

 

[NumberSix]

Means a bad disk most likely, ada2 is throwing errors. Off chance it could be cable or the motherboard as well (If this is onboard SATA)

Thank you Kris. I see that ada2 is in fact one of my mirrored HDDs - less than a year old. So what course of action should help this? Should I try to trigger a 'scrub' (if I have the terminology correct)? If so, how should I actually do that? Thank you!

EDIT:
Actually I found and tried that option - running a scrub. Curiously, it did nothing. I would have imagined it would at least map out the offending sector, but no, apparently not. So what can I do? I can't replace an almost new, little used 8 Tb HDD which cost me a small fortune because something called 'CAM' (whatever that is) is constantly complaining about one sector, whose impact on performance isn't measurable. Yet on the other hand, I naturally want to silence this issue if a way to do so can be found? What would you suggest please?

 

[NugentS]

Remove the drive from the pool
Run a smart test (long) on the drive
When it fails - return under warranty

 

[NumberSix]

Remove the drive from the pool
Run a smart test (long) on the drive
When it fails - return under warranty

Hi Nugent!
Thanks for that. You can't underestimate my abilities here, so can I ask for some more detail please? Thank you.

1. If I remove the drive - how can I run a smart test on it? (Edit: from the GUI, it looks like I can run a smart test on just this drive, in situ?).
2. I've looked at SMART test results before and to a noobie it's all pretty meaningless. Especially, there is no clear 'pass or fail' output, so how would I determine pass or fail?
Thanks again for your support.

 

[Etorix]

You can run a SMART test on a live drive, no need to remove. TrueNAS allows to schedule SMART tests on live pools, and doing so is strongly recommended.
When test smartctl -t long /dev/ada2 is done, post the output of smartctl -a /dev/ada2 in CODE tags. The most critical items are typically #5 and #196-198 for bad and/or suspicious sectors. Parity/CRC errors could also be issues with the controller or cable.

 

[NugentS]

Remove the drive from the pool, I didn't say remove the drive from the system
As an alternative to interpreting the smart stats on FreeBSD, install the drive on a windows box and use the program HDDScan or the Windows Seatools Utils to test the drive, even Crystal DiskInfo will read the smart stats

 

[NumberSix]

You can run a SMART test on a live drive, no need to remove. TrueNAS allows to schedule SMART tests on live pools, and doing so is strongly recommended.
When test smartctl -t long /dev/ada2 is done, post the output of smartctl -a /dev/ada2 in CODE tags. The most critical items are typically #5 and #196-198 for bad and/or suspicious sectors. Parity/CRC errors could also be issues with the controller or cable.

Hi!
Thank you for your detailed advice. I did as you suggested. Unfortunately getting the results to you didn't work out. Shell won't allow any copy/pastes. I read that I should use ssh and putty. After going to great lengths to do that, I ran the same command in putty, but it fails there, as you see here.

However, I finally worked out how to copy most of the shells output, but some had to be painfully transcribed by hand/eye. Here are the hard won results.

---

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST8000VN004-2M2101
Serial Number:    WSD0JH83
LU WWN Device Id: 5 000c50 0c8a31fe4
Firmware Version: SC60
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Mar  3 14:16:26 2022 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (  41) The self-test routine was interrupted
                                        by the host with a hard or soft reset.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 722) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   073   064   044    Pre-fail  Always       -       19102840
  3 Spin_Up_Time            0x0003   082   080   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       245
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   076   060   045    Pre-fail  Always       -       38085481
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       4109
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       127
 18 Head_Health             0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   029   000    Old_age   Always       -       1719
190 Airflow_Temperature_Cel 0x0022   070   058   040    Old_age   Always       -       30 (Min/Max 29/35)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       10
193 Load_Cycle_Count        0x0032   086   086   000    Old_age   Always       -       29531
194 Temperature_Celsius     0x0022   030   042   000    Old_age   Always       -       30 (0 13 0 0 0)
195 Hardware_ECC_Recovered  0x001a   073   064   000    Old_age   Always       -       19102840
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1836
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1742 (183 64 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       17996686346
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       49745955719

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Interrupted (host reset)      00%      4089         -
# 2  Short offline       Completed without error       00%      3999         -
# 3  Short offline       Completed without error       00%      3831         -
# 4  Short offline       Completed without error       00%      3684         -
# 5  Short offline       Completed without error       00%      3518         -
# 6  Short offline       Completed without error       00%      3275         -
# 7  Short offline       Completed without error       00%      3107         -
# 8  Short offline       Completed without error       00%      2999         -
# 9  Short offline       Completed without error       00%      2863         -
#10  Short offline       Completed without error       00%      2733         -
#11  Short offline       Completed without error       00%      2545         -
#12  Short offline       Completed without error       00%      2377         -
#13  Short offline       Completed without error       00%      2208         -
#14  Short offline       Completed without error       00%      2040         -
#15  Short offline       Completed without error       00%      1890         -
#16  Short offline       Completed without error       00%      1785         -
#17  Short offline       Completed without error       00%      1706         -
#18  Short offline       Completed without error       00%      1607         -
#19  Short offline       Completed without error       00%      1465         -
#20  Short offline       Completed without error       00%      1198         -
#21  Short offline       Completed without error       00%      1147         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

---

Can you 'read' this output? Does it say my HDD is pooched, and if so, which part of the output says that?

Just a minor aside, I was bemused to find this result in the GUI; it's not clear to me if it means the test 'failed' to run or if the test ran to completion and it's finding is that the disk 'failed' the test. I was also surprised it doesn't actually show the test results heavily implied by the preceding link, entitled "S.M.A.R.T TEST RESULTS", but that's another matter.

Thank you for your help in decoding this.

 

[Etorix]

Seagate uses some overly sophisticated encoding for some raw values ("error rates"), so that the output is not as straightforward as for WD drives. But for what I understand it seems that your drive is fine: No reallocated (failed) or pending (suspicious) sector. UDMA-CRC errors could be a defective cable or connector: Do you have a replacement cable?

The long SMART test failed because the server was reset before the end of its 722 minutes run (yes, that's 12 hours to complete!). If you know what happened and why, it's all fine. Just run the test again, overnight.

 

[rvassar]

Replace the cable. Pay particular attention to cable bends & stress. If you have extra SATA ports, move it to a different port. Don't worry about moving ports, ZFS will find it and put it back in the pool. Then run a full 12 hour SMART test.

 

[NumberSix]

The long SMART test failed because the server was reset before the end of its 722 minutes run (yes, that's 12 hours to complete!). If you know what happened and why, it's all fine. Just run the test again, overnight.

Thank you!
I ran the test again and this time it reports success - no resets in the middle of it. Do you mind if I invite further opinion from you of what the results indicate? It's the anniversary of buying this drive next month, so if I should return it (huge pain in the bum that would be!), time is running out. If it is Ok then I can breath a huge sigh of relief and go source a new SATA cable instead - child's play to replace in comparison!

Here's the full SMART result:

Code:

root@truenas[~]# smartctl -a /dev/ada2
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST8000VN004-2M2101
Serial Number:    WSD0JH83
LU WWN Device Id: 5 000c50 0c8a31fe4
Firmware Version: SC60
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Mar  4 13:54:35 2022 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 722) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   077   064   044    Pre-fail  Always       -       47220664
  3 Spin_Up_Time            0x0003   082   080   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       245
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   076   060   045    Pre-fail  Always       -       42574584
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       4133
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       127
 18 Head_Health             0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   029   000    Old_age   Always       -       1719
190 Airflow_Temperature_Cel 0x0022   070   058   040    Old_age   Always       -       30 (Min/Max 29/35)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       10
193 Load_Cycle_Count        0x0032   086   086   000    Old_age   Always       -       29532
194 Temperature_Celsius     0x0022   030   042   000    Old_age   Always       -       30 (0 13 0 0 0)
195 Hardware_ECC_Recovered  0x001a   077   064   000    Old_age   Always       -       47220664
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1836
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1766 (208 14 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       18022038778
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       49748721111

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      4132         -
# 2  Extended offline    Interrupted (host reset)      00%      4089         -
# 3  Short offline       Completed without error       00%      3999         -
# 4  Short offline       Completed without error       00%      3831         -
# 5  Short offline       Completed without error       00%      3684         -
# 6  Short offline       Completed without error       00%      3518         -
# 7  Short offline       Completed without error       00%      3275         -
# 8  Short offline       Completed without error       00%      3107         -
# 9  Short offline       Completed without error       00%      2999         -
#10  Short offline       Completed without error       00%      2863         -
#11  Short offline       Completed without error       00%      2733         -
#12  Short offline       Completed without error       00%      2545         -
#13  Short offline       Completed without error       00%      2377         -
#14  Short offline       Completed without error       00%      2208         -
#15  Short offline       Completed without error       00%      2040         -
#16  Short offline       Completed without error       00%      1890         -
#17  Short offline       Completed without error       00%      1785         -
#18  Short offline       Completed without error       00%      1706         -
#19  Short offline       Completed without error       00%      1607         -
#20  Short offline       Completed without error       00%      1465         -
#21  Short offline       Completed without error       00%      1198         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

 

[NumberSix]

Replace the cable. Pay particular attention to cable bends & stress. If you have extra SATA ports, move it to a different port. Don't worry about moving ports, ZFS will find it and put it back in the pool. Then run a full 12 hour SMART test.

I will do that - replace the SATA cable. Thank you for explaining about it being 'found' again ok. Alas, the motherboard, the whole PC, is small and old; exactly the reason I picked it as a NAS housing, but it has 4 SATA connectors with 3 in use, so I can swop socket as well as cable. There's nowhere near me selling PC gear so it's an eBay or Amazon order or similar, and with postage delays, I'll get back to you in a week to 10 days after that! As a short term test I've swopped sockets, and reversed the cable too, thinking that should stress it in different places, if a partial break in the cable is the cause. Thank you again meanwhile.

 

[Etorix]

I may be wrong, but my bet from the SMART report is that the drive is good and the issue is with the cable.

 

[NumberSix]

I may be wrong, but my bet from the SMART report is that the drive is good and the issue is with the cable.

Hi Etorix. You may be wrong, but you may be right too! I think your instincts deserve a promotion. I don't have a replacement cable yet so I simply reversed the one I do have, thinking that would stress it's bends differently by essentially reversing them. Bam! No more constant parity error messages! I think you (and other commentators here) identified the issue. A huge thank you to you (all) - you saved me the pain & aggravation of needlessly returning my drive, and the jeapordy that would have bought to my backup arrangements, so - Thank You!

 

[Etorix]

Glad to know the situation has improved.
I'd still recommend to replace the cable though; it's a minor hurdle compared to replacing a drive.

 

[rvassar]

I'll second @Etorix comment. The connectors on retail cables are rated for all of 50 insert remove events. They get loose in no time, and any stress at all makes them deform and develop flaky connections. Understand, it's not the drive itself, it's the cheap plastic on the cable tips.

 

[NumberSix]

I'll second @Etorix comment. The connectors on retail cables are rated for all of 50 insert remove events. They get loose in no time, and any stress at all makes them deform and develop flaky connections. Understand, it's not the drive itself, it's the cheap plastic on the cable tips.

Hearing you on FM Rvasser! New cables it is then. Thank you everyone!!