Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Need help with "Unhealthy Pool" (Novice)

riddle0

Neophyte
Joined
Feb 11, 2021
Messages
4
Good day!
I'm new to TrueNAS, this is my first server for home use.
  • Motherboard – z170 gigabyte gaming5 miniitx
  • Processor – intel pentium G4560 4 threads
  • RAM – 8GB ddr4 2400 (2x4gb)
  • System SSD – 120gb NVMe
  • HDD 1 – 8TB Seagate Ironwolf ST8000VN0022
  • HDD 2 – 8TB Seagate Ironwolf ST8000VN004
  • HDD 3 – 1.5TB S24083A6
The pool in which the problem occurred consists of two 8TB disks in the mirror.
  1. ada0 – 8TB Seagate Ironwolf ST8000VN0022
  2. ada2 – 8TB Seagate Ironwolf ST8000VN004
Снимок экрана 2021-02-23 в 12.11.11.png

Снимок экрана 2021-02-23 в 12.11.26.png

The system is updated to the latest version, the server rebooted, tried to use the methods described here Hard Drive Troubleshooting Guide (All Versions of FreeNAS), but I did not understand if there is something that applies to my case.
I tried getting SMART from the GUI and got this
ada0 (ST8000VN0022):
Снимок экрана 2021-02-23 в 12.12.22.png


ada2 (ST8000VN004)
Снимок экрана 2021-02-23 в 12.21.31.png

Please tell me what is going on and what I need to do to avoid data loss in this pool

UPD. Today I also noticed that my second pool changed its status to unhealthy. It consists of a single disk. Tests that reached the end-it seems that there were no errors, now I have launched a long test.
Снимок экрана 2021-02-23 в 12.26.34.png

Снимок экрана 2021-02-23 в 12.26.43.png

Снимок экрана 2021-02-23 в 12.27.03.png


Снимок экрана 2021-02-23 в 12.27.50.png
 

Attachments

sretalla

Wizened Sage
Joined
Jan 1, 2016
Messages
3,597
You're going to need to move to the CLI for more information.

Try starting with zpool status -v and share the results back here with code tags.

Also you can have a look at smartctl -a /dev/ada0 smartctl -a /dev/ada1 smartctl -a /dev/ada2 and share here in code tags.
 

RegularJoe

Senior Member
Joined
Aug 19, 2013
Messages
284
What cotroller are you using? Are these USB drives or eSATA via a PM?

Once you have it so the GUI says healthy I would reboot, any errors at boot about zfs db issues and you have a corrupt zfs filesystem and it is best to restore/rsync to a new zvol and destroy your damaged one.

ZFS is not a backup.
 

riddle0

Neophyte
Joined
Feb 11, 2021
Messages
4
Thanks for the answer
Try starting with zpool status -v and share the results back here with code tags.
Code:
root@Cloud Vault[~]# zpool status -v
  pool: Cloudvault2x8
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 384K in 05:53:50 with 10 errors on Thu Feb 11 00:08:06 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        Cloudvault2x8                                   ONLINE       0     0 0
          mirror-0                                      ONLINE       0     0 0
            gptid/faee83d5-396b-11eb-8138-408d5c5afa73  ONLINE       0     0 1
            gptid/fb1bf2fb-396b-11eb-8138-408d5c5afa73  ONLINE       0     0 1

errors: Permanent errors have been detected in the following files:

        /mnt/Cloudvault2x8/Vault/VAULT/Мультимедия/Изображения/Фото/2014/IMG_20140331_144025.jpg
        <0x11a3a>:<0x2308>
        <0x11a3a>:<0x2226>
        <0x11a3a>:<0x2439>
        <0x11a3a>:<0x1c68>
        <0x11a3a>:<0x1d8e>
        <0x11a3a>:<0x2196>
        <0x11a3a>:<0x1c9d>
        <0x11a3a>:<0x21be>
        <0x11a3a>:<0x20dd>

  pool: KUBERA
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 02:12:32 with 0 errors on Sun Feb 21 02:12:32 2021
config:

        NAME                                          STATE     READ WRITE CKSUM
        KUBERA                                        ONLINE       0     0     0
          gptid/428b2f9c-3a4f-11eb-8f83-408d5c5afa73  ONLINE       0    10     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Wed Feb 17 03:45:05 2021
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          nvd0p2    ONLINE       0     0     0

errors: No known data errors
root@Cloud Vault[~]#

Also you can have a look at smartctl -a /dev/ada0 smartctl -a /dev/ada1 smartctl -a /dev/ada2 and share here in code tags.
Code:
root@Cloud Vault[~]# smartctl -a /dev/ada0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST8000VN0022-2EL112
Serial Number:    ZA19RD7B
LU WWN Device Id: 5 000c50 0a5e9e6e5
Firmware Version: SC61
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Feb 23 16:50:19 2021 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 800) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   064   044    Pre-fail  Always       -       235441344
  3 Spin_Up_Time            0x0003   087   084   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   097   097   020    Old_age   Always       -       3456
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   092   060   045    Pre-fail  Always       -       1533228957
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       10868
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   020    Old_age   Always       -       1953
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       2
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   061   049   040    Old_age   Always       -       39 (Min/Max 36/42)
191 G-Sense_Error_Rate      0x0032   097   097   000    Old_age   Always       -       7701
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       544
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       407690
194 Temperature_Celsius     0x0022   039   051   000    Old_age   Always       -       39 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   010   001   000    Old_age   Always       -       235441344
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       3
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       8673 (39 146 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       86322977530
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       162635753846

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     10592         -
# 2  Extended offline    Aborted by host               90%     10575         -
# 3  Short offline       Completed without error       00%         4         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@Cloud Vault[~]#
Code:
root@Cloud Vault[~]# smartctl -a /dev/ada2
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST8000VN004-2M2101
Serial Number:    WKD1XKCT
LU WWN Device Id: 5 000c50 0d026daae
Firmware Version: SC60
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Feb 23 16:53:22 2021 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 723) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   070   064   044    Pre-fail  Always       -       11126240
  3 Spin_Up_Time            0x0003   086   079   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       202
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   045    Pre-fail  Always       -       57301866
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       2052
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       41
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   062   057   040    Old_age   Always       -       38 (Min/Max 36/43)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       729
194 Temperature_Celsius     0x0022   038   043   000    Old_age   Always       -       38 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   070   064   000    Old_age   Always       -       11126240
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1869 (215 125 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       18110565290
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       14397691178

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1775         -
# 2  Short offline       Completed without error       00%      1759         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@Cloud Vault[~]#
Code:
root@Cloud Vault[~]# smartctl -a /dev/ada1
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST1500DM003-9YN16G
Serial Number:    S24083A6
LU WWN Device Id: 5 000c50 04a900b54
Firmware Version: CC4C
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Feb 23 16:56:03 2021 MSK

==> WARNING: A firmware update for this drive is available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 205) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   099   006    Pre-fail  Always       -       75706848
  3 Spin_Up_Time            0x0003   093   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   081   081   020    Old_age   Always       -       20126
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   047   044   030    Pre-fail  Always       -       1477484217272
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       13258
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   020    Old_age   Always       -       3006
183 Runtime_Bad_Block       0x0032   001   001   000    Old_age   Always       -       1469
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   098   098   000    Old_age   Always       -       2
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   095   095   000    Old_age   Always       -       5
190 Airflow_Temperature_Cel 0x0022   066   050   045    Old_age   Always       -       34 (0 1 37 32 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       994
193 Load_Cycle_Count        0x0032   042   042   000    Old_age   Always       -       117524
194 Temperature_Celsius     0x0022   034   050   000    Old_age   Always       -       34 (128 0 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       907
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       4549h+38m+05.168s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       56749194385815
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       124959309593570

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     13255         -
# 2  Short offline       Completed without error       00%     12964         -
# 3  Short offline       Aborted by host               90%      1263         -
# 4  Short offline       Aborted by host               90%      1263         -
# 5  Short offline       Aborted by host               90%      1263         -
# 6  Short offline       Aborted by host               90%      1263         -
# 7  Short offline       Aborted by host               90%      1263         -
# 8  Short offline       Aborted by host               90%      1263         -
# 9  Short offline       Aborted by host               90%      1263         -
#10  Short offline       Aborted by host               90%      1263         -
#11  Short offline       Aborted by host               90%      1263         -
#12  Short offline       Aborted by host               90%      1263         -
#13  Short offline       Aborted by host               90%      1263         -
#14  Short offline       Aborted by host               90%      1263         -
#15  Short offline       Aborted by host               90%      1263         -
#16  Short offline       Aborted by host               90%      1263         -
#17  Short offline       Aborted by host               90%      1263         -
#18  Short offline       Aborted by host               90%      1263         -
#19  Short offline       Aborted by host               90%      1263         -
#20  Short offline       Aborted by host               90%      1263         -
#21  Short offline       Aborted by host               90%      1263         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@Cloud Vault[~]#
 

riddle0

Neophyte
Joined
Feb 11, 2021
Messages
4
I also forgot to specify that some of the files stored on the disk were copied from a dead HDD, maybe that's why it refers to the fact that some files are damaged. But they have been stored there for a long time and before the status of the pool was normal
 

riddle0

Neophyte
Joined
Feb 11, 2021
Messages
4
What cotroller are you using? Are these USB drives or eSATA via a PM?
The 3.5" drives are connected directly to the NAS motherboard's SATA ports. The SATA controller is standard for the Z170 Intel chipset. I do not use a hardware raid, but use a mirror one created inside TrueNAS
ZFS is not a backup.
I use zfs pool as the main file storage, so I created it in the mirror for safety
 

Yorick

Dedicated Sage
Joined
Nov 4, 2018
Messages
1,759
Raw read error rate is high. If that keeps going up, it looks like all three disks are failing.

CloudVault unfortunately has metadata errors. If you don't have a backup, get the data off *now* that you can. You may need to rebuild the pool if these metadata errors can't be resolved.
 
Top