Help required to problem solve smart testing mystery

[ethereal]

hello i recently had a problem with long smart tests not being run as scheduled. so i was trying to solve the problem but got some mixed smart data back.

smartctl -a /dev/da0 | grep "progress" -i -A 1

Self-test execution status: ( 241) Self-test routine in progress...
10% of test remaining.

as you can see 10% remaining but if i run

Code:

smartctl -a /dev/da0
                                        
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     37810         -
# 2  Short offline       Completed without error       00%     37787         -
# 3  Short offline       Completed without error       00%     37763         -
# 4  Short offline       Completed without error       00%     37738         -
# 5  Short offline       Completed without error       00%     37714         -
# 6  Short offline       Completed without error       00%     37690         -
# 7  Short offline       Completed without error       00%     37671         -
# 8  Short offline       Completed without error       00%     37643         -
# 9  Short offline       Completed without error       00%     37618         -
#10  Short offline       Completed without error       00%     37594         -
#11  Short offline       Completed without error       00%     37572         -
#12  Short offline       Completed without error       00%     37546         -
#13  Short offline       Completed without error       00%     36899         -
#14  Short offline       Completed without error       00%     36875         -
#15  Short offline       Completed without error       00%     36852         -
#16  Extended offline    Completed without error       00%     36847         -
#17  Short offline       Completed without error       00%     36827         -
#18  Short offline       Completed without error       00%     36803         -
#19  Short offline       Completed without error       00%     36779         -
#20  Short offline       Completed without error       00%     36755         -
#21  Short offline       Completed without error       00%     36732         -    

as you can see here all tests completed without error 0% remaining. this is happening on 6 discs out of 12 and as you can see there is no extended test run recently.
so which command to believe 10% remaing or 0% all completed ?
are there any commands to see what is what?

thank you

 

[Chris Moore]

Self-test execution status: ( 241) Self-test routine in progress...
10% of test remaining.

I would like to see the full output of smartctl -x /dev/da0 so we can see if there are any more details that would help explain this.

 

[jgreco]

The SMART log doesn't necessarily show tests in progress. Does the test eventually complete?

 

[ethereal]

Code:

root@freenas:~ # smartctl -x /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital AV-GP (AF)
Device Model:     WDC WD30EURX-64HYZY0
Serial Number:    WD-WMC4N0F63AAE
LU WWN Device Id: 5 0014ee 0594119b8
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jun 23 00:44:50 2019 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
Total time to complete Offline
data collection:                (39240) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 394) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   179   177   021    -    6050
  4 Start_Stop_Count        -O--CK   100   100   000    -    191
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   049   049   000    -    37839
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    191
192 Power-Off_Retract_Count -O--CK   200   200   000    -    180
193 Load_Cycle_Count        -O--CK   200   200   000    -    584
194 Temperature_Celsius     -O---K   117   103   000    -    33
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     37836         -
# 2  Short offline       Completed without error       00%     37810         -
# 3  Short offline       Completed without error       00%     37787         -
# 4  Short offline       Completed without error       00%     37763         -
# 5  Short offline       Completed without error       00%     37738         -
# 6  Short offline       Completed without error       00%     37714         -
# 7  Short offline       Completed without error       00%     37690         -
# 8  Short offline       Completed without error       00%     37671         -
# 9  Short offline       Completed without error       00%     37643         -
#10  Short offline       Completed without error       00%     37618         -
#11  Short offline       Completed without error       00%     37594         -
#12  Short offline       Completed without error       00%     37572         -
#13  Short offline       Completed without error       00%     37546         -
#14  Short offline       Completed without error       00%     36899         -
#15  Short offline       Completed without error       00%     36875         -
#16  Short offline       Completed without error       00%     36852         -
#17  Extended offline    Completed without error       00%     36847         -
#18  Short offline       Completed without error       00%     36827         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        DST executing in background (3)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     29/36 Celsius
Lifetime    Min/Max Temperature:      2/47 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (218)

Index    Estimated Time   Temperature Celsius
 219    2019-06-22 16:47    32  *************
 ...    ..( 75 skipped).    ..  *************
 295    2019-06-22 18:03    32  *************
 296    2019-06-22 18:04    33  **************
 ...    ..( 12 skipped).    ..  **************
 309    2019-06-22 18:17    33  **************
 310    2019-06-22 18:18    32  *************
 ...    ..(153 skipped).    ..  *************
 464    2019-06-22 20:52    32  *************
 465    2019-06-22 20:53    33  **************
 ...    ..( 18 skipped).    ..  **************
   6    2019-06-22 21:12    33  **************
   7    2019-06-22 21:13    32  *************
 ...    ..(210 skipped).    ..  *************
 218    2019-06-23 00:44    32  *************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            0  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4      2562559  Vendor specific

 

[ethereal]

i think the problem maybe solved - i posted in another thread where i outlined a problem where 5 discs became deselected in my long smart tests. i selected all discs and saved the test schedule. when i checked yesterday (saturday the day the test should run) they had not started. so i changed the time to midnight sunday morning. now everything looks good. i'll have to wait until the tests are finished to ensure they were long and completed okay.

Code:

root@freenas:~ # smartctl -a /dev/da0 | grep "progress" -i -A 1
Self-test execution status:      ( 248) Self-test routine in progress...
                                        80% of test remaining.
root@freenas:~ # smartctl -a /dev/da1 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/da2 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/da3 | grep "progress" -i -A 1
Self-test execution status:      ( 248) Self-test routine in progress...
                                        80% of test remaining.
root@freenas:~ # smartctl -a /dev/da4 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/da5 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/da6 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/da7 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/ada0 | grep "progress" -i -A 1
root@freenas:~ # smartctl -a /dev/ada1 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/ada2 | grep "progress" -i -A 1
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
root@freenas:~ # smartctl -a /dev/ada3 | grep "progress" -i -A 1

 

[ethereal]

i am having more problems with the 6x3tb pool. the long tests never got past 10 % remaining. and the scrub was taking a long time so i canceled all of the tests. this hasn't helped with the scrub - it usually takes about 15 hours - it is 18 hours gone with 1 day 12 hours to go - and the time to go keeps increasing.

as all 6 drives failed to finish the long test and the time to go for the scrub keeps increasing i'm worried i'll lose the pool - it is raidz2.

what do you suggest - i have some 4 spare 3 tb discs - should i create a new pool and copy the data to the new pool or use them as external drive and copy files over (this would be using windows and ntfs)

should i use my spare discs to start replacing discs in the pool ?

should i let the scrub continue ? can i cancel it ?

 

[ethereal]

panic over scrub eventually finished and i ran long test today which completed on time