Pool status " ONLINE (Unhealthy)

mitch0627

Cadet
Joined
Aug 10, 2022
Messages
2
Hello All,
im running TrueNas core 12.0.u6 with 2 (4TB) drives. and 12 GB Ram.

I notices my Pool Status was "degraded" do to 1 checksum error on each drive. I ran (zpool clear) and the pool Status went to ONLINE but now its now Unhealthy.


Code:
root@freenas[~]# zpool status -v Shared-NAS
  pool: Shared-NAS
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 04:05:26 with 6 errors on Sun Jul 10 04:05:27 2022
config:

    NAME                                            STATE     READ WRITE CKSUM
    Shared-NAS                                      ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/ba610235-21f3-11e9-943b-f8b156a6494f  ONLINE       0     0     1
        gptid/bb4b3520-21f3-11e9-943b-f8b156a6494f  ONLINE       0     0     1

errors: Permanent errors have been detected in the following files:

        <0x237>:<0xb8f>
        <0x237>:<0xbd5>
        <0x237>:<0xbe4>
        /var/db/system/rrd-1c36823397d74dbc9e7d4fbefe8b455e/localhost/interface-bridge0/if_octets.rrd
        /var/db/system/rrd-1c36823397d74dbc9e7d4fbefe8b455e/localhost/df-mnt-Shared-NAS-iocage-jails-nextcloudjail/df_complex-free.rrd
        /var/db/system/rrd-1c36823397d74dbc9e7d4fbefe8b455e/localhost/disk-da1/disk_time.rrd
root@freenas[~]#



Code:
root@freenas[~]# smartctl -a /dev/ada0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZGY3N6K8
LU WWN Device Id: 5 000c50 0b424f197
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug 10 15:19:06 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  581) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 604) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x50bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.


SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       184774333
  3 Spin_Up_Time            0x0003   095   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       140
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1054932374
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       13317 (46 246 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       140
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   066   059   040    Old_age   Always       -       34 (Min/Max 23/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       80
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       182
194 Temperature_Celsius     0x0022   034   041   000    Old_age   Always       -       34 (0 10 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       13313 (42 243 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       36514130746
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       29030057884


SMART Error Log Version: 1
No Errors Logged


SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     13297         -


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



Code:
root@freenas[~]# smartctl -a /dev/ada1
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZGY3NN0M
LU WWN Device Id: 5 000c50 0b4248664
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug 10 15:20:40 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  591) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 651) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x50bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       184571102
  3 Spin_Up_Time            0x0003   095   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       140
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1429765648
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       13317 (48 209 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       140
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   069   062   040    Old_age   Always       -       31 (Min/Max 23/32)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       82
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       176
194 Temperature_Celsius     0x0022   031   040   000    Old_age   Always       -       31 (0 11 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       13312 (10 36 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       36514519694
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       28958839863

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 

Attachments

  • Screenshot from 2022-08-10 16-13-45.png
    Screenshot from 2022-08-10 16-13-45.png
    78.8 KB · Views: 190
  • Screenshot from 2022-08-10 16-14-23.png
    Screenshot from 2022-08-10 16-14-23.png
    17.7 KB · Views: 196

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
Post your hardware please - as per forum rules.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
For the time being you should work under the assumption that a serious problem exists in your setup. Until the details have been figured out, please make a backup immediately.

Whether one of your disks is about to die, you have a cabling issue, or some other problems needs further investigation. And, as @NugentS pointed out, we need more information.
 

mitch0627

Cadet
Joined
Aug 10, 2022
Messages
2
Hardware:
  • DELL optiplex 7010
  • intel i3
  • RAM 12 Gb
  • Hard drives:
  • Seagate 4TB Irenwolf NAS (x2)
  • 16 gb usb boot drive.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
OK - well the usual suspects haven't appeared (other than non-server grade kit). How are the disks connected to the computer. I would assume via on-board SATA at this point - but it would nice if that were confirmed please.
You don't say if the RAM is ECC or not - I guess probably not
The 16GB USB Boot drive is not ideal - but isn't the cause of this issue - do you have a backup of the TrueNAS config? From before this issue occurred preferably?

chksum errors are usually (but not always) caused by bad cables. So I would reseat / replace the SATA cables

Your pool is hosed - but not is a totally disastrous way. You appear to have lost some metadata and some of the system dataset.

Do you have a backup of your data? If not then make one immediately.
 
Top