Pool Status Unhealthy Literally No Drive Errors

[jpcapone]

Ok in my TrueNAS dashboard the Pool Status reports as being unhealthy while everything else is green.

But there are no errors

Everything is functioning nominally but it is kind of annoying. I just rebuilt my TrueNAS system - which used to be virtual - with ECC RAM because I was experiencing rando read/write/checksum errors on disks that were brand new. So now no errors but still unhealthy. Any suggestions?

 

[Samuel Tai]

Run zpool status -v <name of your pool>. It will describe what it's unhappy about.

 

[jpcapone]

Thanks for that. This is what I get:
/var/db/system/rrd-1e9984bcf13340bcb68bc263ecb0a902/localhost/cpu-1/percent-system.rrd
/var/db/system/rrd-1e9984bcf13340bcb68bc263ecb0a902/localhost/cpu-3/percent-system.rrd

Any suggestions on next steps? I would delete them but I don't think that would resolve the issue.

 

[Samuel Tai]

That's graph data for the system graphs. You don't need to delete them. Just switch the system dataset in System->System Dataset back to your boot pool. Then scrub your data pool. You may need to scrub a couple of times to clear the unhealthy status. Then move the system dataset back.

 

[jpcapone]

That's graph data for the system graphs. You don't need to delete them. Just switch the system dataset in System->System Dataset back to your boot pool. Then scrub your data pool. You may need to scrub a couple of times to clear the unhealthy status. Then move the system dataset back.

Ok I moved the System Dataset back to boot and I ran the scrub process quite a few times and I even rebooted but the pool still reports unhealthy and the status -v still shows the same problematic files. Do I need to wait until it reports healthy before I switch the System Dataset back? Any other suggestions? Here is a screener showing that I did switch the option you mentioned.

 

[jpcapone]

That's graph data for the system graphs. You don't need to delete them. Just switch the system dataset in System->System Dataset back to your boot pool. Then scrub your data pool. You may need to scrub a couple of times to clear the unhealthy status. Then move the system dataset back.

Ok I have followed your directions and I have been scrubbing the pool every day since my original post and I am still coming up with an unhealthy pool. Any other suggestions?

 

[SweetAndLow]

Ok I have followed your directions and I have been scrubbing the pool every day since my original post and I am still coming up with an unhealthy pool. Any other suggestions?

You never actually ran the zpool status -v command that was asked of you. Please do that

 

[jpcapone]

You never actually ran the zpool status -v command that was asked of you. Please do that

Maybe I am missing something but I posted the results of the zpool status -v on February 1, 2021 and Samuel Tai responded to it on the same day. I ran another one and the full results below:

Code:

        NAME                                            STATE     READ WRITE CKSUM
        DeadPool                                        ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/1b9f87fe-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/05bc6ea1-5c57-11eb-8d6f-005056ab46b3  ONLINE       0     0     8
            gptid/df60cc1b-56ec-11eb-8b64-005056ab46b3  ONLINE       0     0     8
            gptid/2464e446-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/212f21fa-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/29a80c52-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/effe0a3e-5fff-11eb-aa71-005056ab46b3  ONLINE       0     0     8
            gptid/99c6547d-521a-11eb-a710-005056ab46b3  ONLINE       0     0     8

errors: Permanent errors have been detected in the following files:

        DeadPool/.system/rrd-1e9984bcf13340bcb68bc263ecb0a902:/localhost/cpu-1/percent-system.rrd
        DeadPool/.system/rrd-1e9984bcf13340bcb68bc263ecb0a902:/localhost/cpu-3/percent-system.rrd
root@freenas[~]#

 

[Alecmascot]

There are the drive errors....
I now think we need full specs of your system as per the forum rules.

 

[SweetAndLow]

Maybe I am missing something but I posted the results of the zpool status -v on February 1, 2021 and Samuel Tai responded to it on the same day. I ran another one and the full results below:

Code:

        NAME                                            STATE     READ WRITE CKSUM
        DeadPool                                        ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/1b9f87fe-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/05bc6ea1-5c57-11eb-8d6f-005056ab46b3  ONLINE       0     0     8
            gptid/df60cc1b-56ec-11eb-8b64-005056ab46b3  ONLINE       0     0     8
            gptid/2464e446-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/212f21fa-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/29a80c52-4ecd-11eb-b807-005056ab46b3  ONLINE       0     0     8
            gptid/effe0a3e-5fff-11eb-aa71-005056ab46b3  ONLINE       0     0     8
            gptid/99c6547d-521a-11eb-a710-005056ab46b3  ONLINE       0     0     8

errors: Permanent errors have been detected in the following files:

        DeadPool/.system/rrd-1e9984bcf13340bcb68bc263ecb0a902:/localhost/cpu-1/percent-system.rrd
        DeadPool/.system/rrd-1e9984bcf13340bcb68bc263ecb0a902:/localhost/cpu-3/percent-system.rrd
root@freenas[~]#

You never posted this. Can you now post the smartctl -a info for each disk? Looks like every disk in your system has checksum errors.

 

[jpcapone]

You never posted this. Can you now post the smartctl -a info for each disk? Looks like every disk in your system has checksum errors.

Sorry for the confusion.......

freenas outup - Pastebin

Pastebin.pl is a website where you can store code/text online for a set period of time and share to anybody on earth

pastebin.pl

 

[jpcapone]

eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

You never posted this. Can you now post the smartctl -a info for each disk? Looks like every disk in your system has checksum errors.

Any ideas or recommendations regarding my issue?

 

[SweetAndLow]

eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

Any ideas or recommendations regarding my issue?

no clue that link doesn't work. Can you just post the output using [CODE ] tags?

 

[jpcapone]

no clue that link doesn't work. Can you just post the output using [CODE ] tags?

Thanks for getting back to me. When i do that it says I am over the character limit.

 

[jpcapone]

no clue that link doesn't work. Can you just post the output using [CODE ] tags?

Code:

root@freenas[~]# smartctl -a /dev/da0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Toshiba X300
 
Device Model:     TOSHIBA HDWE140
 
Serial Number:    Y043K5ZFFBRG
 
LU WWN Device Id: 5 000039 a6b8001b3
 
Firmware Version: FP1R
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ATA8-ACS (minor revision not indicated)
 
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:52:12 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (  120) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 487) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
 
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       5942
 
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       979
 
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       2456
 
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
 
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
 
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       945
 
 10 Spin_Retry_Count        0x0033   119   100   030    Pre-fail  Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       979
 
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       17
 
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       969
 
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1056
 
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       27 (Min/Max 17/47)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       146
 
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
 
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
 
222 Loaded_Hours            0x0032   098   098   000    Old_age   Always       -       870
 
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
 
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       556
 
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 
# 1  Extended offline    Completed without error       00%       357         -
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[jpcapone]

Code:

root@freenas[~]# smartctl -a /dev/da1
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Western Digital Red Pro
 
Device Model:     WDC WD4003FFBX-68MU3N0
 
Serial Number:    V1HHZNMG
 
LU WWN Device Id: 5 000cca 0bcd55c2f
 
Firmware Version: 83.00A83
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
 
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:54:07 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (   87) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 420) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0004   133   133   054    Old_age   Offline      -       93
 
  3 Spin_Up_Time            0x0007   134   134   024    Pre-fail  Always       -       328 (Average 301)
 
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       42
 
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
 
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
 
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       890
 
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       42
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       79
 
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       79
 
194 Temperature_Celsius     0x0002   214   214   000    Old_age   Always       -       28 (Min/Max 16/42)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[jpcapone]

no clue that link doesn't work. Can you just post the output using [CODE ] tags?

Code:

root@freenas[~]# smartctl -a /dev/da3
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Toshiba X300
 
Device Model:     TOSHIBA HDWE140
 
Serial Number:    Y03AK70SFBRG
 
LU WWN Device Id: 5 000039 a6bb80193
 
Firmware Version: FP1R
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ATA8-ACS (minor revision not indicated)
 
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:57:13 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (  120) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 475) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
 
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       456
 
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       64
 
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
 
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
 
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       810
 
 10 Spin_Retry_Count        0x0033   101   100   030    Pre-fail  Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       64
 
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       112
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       56
 
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       92
 
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       26 (Min/Max 15/44)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
 
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
 
222 Loaded_Hours            0x0032   098   098   000    Old_age   Always       -       805
 
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
 
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       559
 
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
 
 
root@freenas[~]# smartctl -a /dev/da4
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Toshiba X300
 
Device Model:     TOSHIBA HDWE140
 
Serial Number:    Y03AK70NFBRG
 
LU WWN Device Id: 5 000039 a6bb8018e
 
Firmware Version: FP1R
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ATA8-ACS (minor revision not indicated)
 
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:58:27 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (  120) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 484) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
 
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       5782
 
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       40
 
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
 
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
 
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       810
 
 10 Spin_Retry_Count        0x0033   100   100   030    Pre-fail  Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       40
 
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       52
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       32
 
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       61
 
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       29 (Min/Max 17/44)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
 
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
 
222 Loaded_Hours            0x0032   098   098   000    Old_age   Always       -       806
 
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
 
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       559
 
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
 
 
root@freenas[~]# smartctl -a /dev/da5
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Western Digital Red Pro
 
Device Model:     WDC WD4003FFBX-68MU3N0
 
Serial Number:    V1J3ME5G
 
LU WWN Device Id: 5 000cca 0bcddd50a
 
Firmware Version: 83.00A83
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
 
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:59:06 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (   87) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 452) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0004   132   132   054    Old_age   Offline      -       96
 
  3 Spin_Up_Time            0x0007   140   140   024    Pre-fail  Always       -       305 (Average 300)
 
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       36
 
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
 
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
 
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       727
 
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       36
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       68
 
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       68
 
194 Temperature_Celsius     0x0002   206   206   000    Old_age   Always       -       29 (Min/Max 14/40)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[jpcapone]

no clue that link doesn't work. Can you just post the output using [CODE ] tags?

Code:

root@freenas[~]# smartctl -a /dev/da6
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Western Digital Red Pro
 
Device Model:     WDC WD4003FFBX-68MU3N0
 
Serial Number:    VBGHXM6F
 
LU WWN Device Id: 5 000cca 095c73d00
 
Firmware Version: 83.00A83
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
 
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 15:59:48 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (   87) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 466) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0004   132   132   054    Old_age   Offline      -       96
 
  3 Spin_Up_Time            0x0007   154   154   024    Pre-fail  Always       -       399 (Average 386)
 
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       46
 
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
 
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
 
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       606
 
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       46
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       100
 
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       100
 
194 Temperature_Celsius     0x0002   206   206   000    Old_age   Always       -       29 (Min/Max 17/41)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
 
 
root@freenas[~]# smartctl -a /dev/da7
 
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
 
 
 
=== START OF INFORMATION SECTION ===
 
Model Family:     Western Digital Red Pro
 
Device Model:     WDC WD4003FFBX-68MU3N0
 
Serial Number:    V1J26S3H
 
LU WWN Device Id: 5 000cca 0bcdd3112
 
Firmware Version: 83.00A83
 
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 
Sector Sizes:     512 bytes logical, 4096 bytes physical
 
Rotation Rate:    7200 rpm
 
Form Factor:      3.5 inches
 
Device is:        In smartctl database [for details use: -P show]
 
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
 
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
 
Local Time is:    Tue Feb  9 16:00:28 2021 EST
 
SMART support is: Available - device has SMART capability.
 
SMART support is: Enabled
 
 
 
=== START OF READ SMART DATA SECTION ===
 
SMART overall-health self-assessment test result: PASSED
 
 
 
General SMART Values:
 
Offline data collection status:  (0x82) Offline data collection activity
 
                                        was completed without error.
 
                                        Auto Offline Data Collection: Enabled.
 
Self-test execution status:      (   0) The previous self-test routine completed
 
                                        without error or no self-test has ever
 
                                        been run.
 
Total time to complete Offline
 
data collection:                (   87) seconds.
 
Offline data collection
 
capabilities:                    (0x5b) SMART execute Offline immediate.
 
                                        Auto Offline data collection on/off support.
 
                                        Suspend Offline collection upon new
 
                                        command.
 
                                        Offline surface scan supported.
 
                                        Self-test supported.
 
                                        No Conveyance Self-test supported.
 
                                        Selective Self-test supported.
 
SMART capabilities:            (0x0003) Saves SMART data before entering
 
                                        power-saving mode.
 
                                        Supports SMART auto save timer.
 
Error logging capability:        (0x01) Error logging supported.
 
                                        General Purpose Logging supported.
 
Short self-test routine
 
recommended polling time:        (   2) minutes.
 
Extended self-test routine
 
recommended polling time:        ( 433) minutes.
 
SCT capabilities:              (0x003d) SCT Status supported.
 
                                        SCT Error Recovery Control supported.
 
                                        SCT Feature Control supported.
 
                                        SCT Data Table supported.
 
 
 
SMART Attributes Data Structure revision number: 16
 
Vendor Specific SMART Attributes with Thresholds:
 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
 
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
 
  2 Throughput_Performance  0x0004   133   133   054    Old_age   Offline      -       92
 
  3 Spin_Up_Time            0x0007   140   140   024    Pre-fail  Always       -       311 (Average 296)
 
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       29
 
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
 
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
 
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
 
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       409
 
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       29
 
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       72
 
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       72
 
194 Temperature_Celsius     0x0002   200   200   000    Old_age   Always       -       30 (Min/Max 17/38)
 
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
 
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
 
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
 
 
 
SMART Error Log Version: 1
 
No Errors Logged
 
 
 
SMART Self-test log structure revision number 1
 
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
 
 
SMART Selective self-test log data structure revision number 1
 
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
 
    1        0        0  Not_testing
 
    2        0        0  Not_testing
 
    3        0        0  Not_testing
 
    4        0        0  Not_testing
 
    5        0        0  Not_testing
 
Selective self-test flags (0x0):
 
  After scanning selected spans, do NOT read-scan remainder of disk.
 
If Selective self-test is pending on power-up, resume after 0 minute dela

 

[SweetAndLow]

Only da0 has ever had a self test run on it. I suggest running a long test on all drives and a short test on all drives.

These also look fairly new, did you do any burn in testing?

 

[SweetAndLow]

You need to post all your system hardware specs also. Read the forum rules.