Is there any way I can replace those drives?

[BarneyGold]

Hi everyone and my apologies for the evasive title.

I have a 4 drives raidz1 pool which I have been running on Scale 22.02.0.1.
One day the pool started to appear as Degraded, after what I performed short and long SMART tests on all drives without any error.
Truenas could however identify two of the drives to be faulty, one in a critical state and the other "only" in a degraded state (I don't remember the exact warning messages as a few days passed since then).
I remember still being able to read and write files on the associated SMB share with no problem.
I then offlined the critical error drive and ordered two other drives.

I have performed a fresh install of Scale on my SSD (also 22.02.0.1, with a drive format) and can see my pool in the import selection list, but importing it causes an infinite loading.
I can however import it in read only mode through the shell via zpool import -fo readonly=on MAIN

Here is the status of the pool :

Code:

root@truenas[~]# zpool status -v MAIN
  pool: MAIN
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub in progress since Sun Apr 10 09:57:34 2022
        0B scanned at 0B/s, 0B issued at 0B/s, 24.8T total
        0B repaired, 0.00% done, no estimated completion time
config:

        NAME                                      STATE     READ WRITE CKSUM
        MAIN                                      DEGRADED     0     0     0
          raidz1-0                                DEGRADED     0     0     0
            6ff51e1c-6e68-11ec-8a70-3cecef6779ea  ONLINE       0     0     0
            6ff6d45f-6e68-11ec-8a70-3cecef6779ea  DEGRADED     0     0     0  too many errors
            70070cc4-6e68-11ec-8a70-3cecef6779ea  OFFLINE      0     0     0
            70142708-6e68-11ec-8a70-3cecef6779ea  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x5657>

My question is, can I still somehow replace the offline disk and use one of the two new drives I bought to resilver the pool, then replace the second failing drive to resilver a second time?

Thanks in advance and have a nice day!

 

[joeschmuck]

I performed short and long SMART tests on all drives without any error.

My question is, can I still somehow replace the offline disk and use one of the two new drives I bought to resilver the pool, then replace the second failing drive to resilver a second time?

Why replace drives that do not fail?

Please use the link above to the Forum Rules, you can find a lot of help from previous postings on how to possibly correct this problem without replacing the drives. If you need to ask for specific help because you cannot locate it in the forums or just do not understand it, then follow the Forum Rules and post your system information, we will want to see the output of all your hard drives SMART data to verify the drives are good or bad as this will help us give you correct advice. Please understand that your problem is not unfamiliar, this happens often when people have component failure. It happens to us all at some point in time.

My first advice, backup your data before proceeding, you do have one drive that is offline and another degraded. Should the degraded drive die, your data is gone.

Good Luck

 

[BarneyGold]

Thanks for the reply.

I didn't think posting my system specs would help which is why I didn't want to flush too much info.

Show : here's more details on my system

Xeon E5-2630 v3 CPU | 32GB Samsung ECC Registered | Supermicro X10SRl-F | 128GB SSD Boot Drive | 250GB SSD Drive for VMs | Four 10TB WD Red Plus and two Seagate Ironwolf 10TB connected directly to the mobo (Two WD drives and the Seagate drives are in the "MAIN" radz1 pool)

I ran smartctl -a /dev/sdX for the four drives as you suggested providing the output of the SMART tests, hopefully this was the right command.

Show : Command results

Code:

root@truenas[~]# smartctl -a /dev/sdb 
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.93+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD101EFBX-68B0AN0
Serial Number:    
LU WWN Device Id: 5 000cca 0b0dd9850
Firmware Version: 85.00A85
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 22 14:21:29 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1001) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   129   129   054    Old_age   Offline      -       104
  3 Spin_Up_Time            0x0007   174   174   024    Pre-fail  Always       -       491 (Average 444)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       713
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       3446
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       276
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1477
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1477
194 Temperature_Celsius     0x0002   154   154   000    Old_age   Always       -       42 (Min/Max 20/58)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3357         -
# 2  Short offline       Completed without error       00%      2234         -
# 3  Short offline       Completed without error       00%      2101         -
# 4  Short offline       Completed without error       00%      1932         -
# 5  Extended offline    Completed without error       00%      1836         -
# 6  Short offline       Completed without error       00%      1773         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

root@truenas[~]# smartctl -a /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.93+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD101EFBX-68B0AN0
Serial Number:    
LU WWN Device Id: 5 000cca 0b0df5009
Firmware Version: 85.00A85
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 22 14:27:49 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1004) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   128   128   054    Old_age   Offline      -       108
  3 Spin_Up_Time            0x0007   182   182   024    Pre-fail  Always       -       497 (Average 396)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       212
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       2134
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       136
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       367
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       367
194 Temperature_Celsius     0x0002   151   151   000    Old_age   Always       -       43 (Min/Max 20/45)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      2045         -
# 2  Short offline       Completed without error       00%       922         -
# 3  Short offline       Completed without error       00%       789         -
# 4  Short offline       Completed without error       00%       620         -
# 5  Extended offline    Completed without error       00%       525         -
# 6  Short offline       Completed without error       00%       461         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

root@truenas[~]# smartctl -a /dev/sdc
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.93+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST10000VN0004-1ZD101
Serial Number:    
LU WWN Device Id: 5 000c50 0b6572d41
Firmware Version: SC60
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 22 14:28:34 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  575) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 861) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   065   064   044    Pre-fail  Always       -       3539645
  3 Spin_Up_Time            0x0003   092   086   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   098   098   020    Old_age   Always       -       2693
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       8
  7 Seek_Error_Rate         0x000f   094   060   045    Pre-fail  Always       -       2347347543
  9 Power_On_Hours          0x0032   076   076   000    Old_age   Always       -       21683 (45 18 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       631
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       68720525337
189 High_Fly_Writes         0x003a   001   001   000    Old_age   Always       -       126
190 Airflow_Temperature_Cel 0x0022   068   049   040    Old_age   Always       -       32 (Min/Max 24/32)
191 G-Sense_Error_Rate      0x0032   090   090   000    Old_age   Always       -       21297
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       255
193 Load_Cycle_Count        0x0032   053   053   000    Old_age   Always       -       95492
194 Temperature_Celsius     0x0022   032   051   000    Old_age   Always       -       32 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   002   001   000    Old_age   Always       -       3539645
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Pressure_Limit          0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10897 (160 59 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       151914175164
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       276354299084

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     21661         -
# 2  Short offline       Completed without error       00%     21659         -
# 3  Short offline       Completed without error       00%     21593         -
# 4  Short offline       Completed without error       00%     21592         -
# 5  Extended offline    Interrupted (host reset)      00%     21592         -
# 6  Short offline       Completed without error       00%     20472         -
# 7  Short offline       Completed without error       00%     20338         -
# 8  Short offline       Completed without error       00%     20169         -
# 9  Extended offline    Completed without error       00%     20073         -
#10  Short offline       Completed without error       00%     20010         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

root@truenas[~]# smartctl -a /dev/sdd
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.93+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST10000VN0008-2JJ101
Serial Number:    
LU WWN Device Id: 5 000c50 0c5b11087
Firmware Version: SC60
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 22 14:29:53 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 977) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       180724271
  3 Spin_Up_Time            0x0003   093   087   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   099   099   020    Old_age   Always       -       1287
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   087   060   045    Pre-fail  Always       -       538782418
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       10313
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       608
 18 Head_Health             0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   070   043   040    Old_age   Always       -       30 (Min/Max 24/30)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       368
193 Load_Cycle_Count        0x0032   098   098   000    Old_age   Always       -       5128
194 Temperature_Celsius     0x0022   030   057   000    Old_age   Always       -       30 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   083   064   000    Old_age   Always       -       180724271
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Pressure_Limit          0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       9319 (34 144 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       78268151288
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       105546121048

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     10293         -
# 2  Short offline       Completed without error       00%     10293         -
# 3  Short offline       Completed without error       00%     10292         -
# 4  Short offline       Completed without error       00%     10291         -
# 5  Short offline       Completed without error       00%     10226         -
# 6  Short offline       Completed without error       00%     10225         -
# 7  Extended offline    Interrupted (host reset)      00%     10225         -
# 8  Short offline       Completed without error       00%      9104         -
# 9  Short offline       Completed without error       00%      8970         -
#10  Short offline       Completed without error       00%      8802         -
#11  Extended offline    Completed without error       00%      8705         -
#12  Short offline       Completed without error       00%      8643         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

As you can see, they all display the SMART result as PASSED even though some results regarding the Seagate drives do seem alarming to me, hence why I ordered two other drives to replace them.

I do understand that compromised pools posts are abundant on the forums, I just didn't find anyone in my situation which is not being able to import the pool in order to scrub ot replace drives. I cannot do much with a pool that's read only.

 

[Jailer]

You supplied the smart data for 4 drives but you say you have 6 drives attached to your motherboard. Are you sure you are providing the smart data for the correct 4 drives?

Also, why the fresh install? This isn't windows we are dealing with here.

 

[joeschmuck]

As you can see, they all display the SMART result as PASSED

I recommend that you do some searching on the forums and reading up on SMART results and they they mean. Unfortunately the fact that it says "PASSED" is misleading, do not rely on this data. What I noticed was while you may have done an extended test, most of those were quite a long time ago. You should rerun a SMART Long/Extended test on all your drives. The only drive you have had any issues with was S/N: , well shit, you redacted the serial numbers, I still don't get why people remove the serial numbers but whatever, you should at least leave the last 4 digits so we can track the drive by the serial number. The drive with 8 sectors that were relocated, but no other issues so those may have been a one time issue. As of right now, none of those drives are suspect to me, but you need to run the long test and ensure something doesn't fail.

And @Jailer is correct, you have six drives, where are the other two? Also, Clean Install? What was previously used? Can you roll back?

Out of curiosity, which two drives were you planning to replace and how did you conclude which ones those were? I'm asking a serious question because I want to know your thought process as I could help me help others later.

Something else you could try, grab a USB flash drive (stick) with 16GB (or similar), disconnect your boot drive, install TrueNAS Core 12.0-U8 to the USB Flash drive, boot the system and do a minimal configuration, and then import your pool. You might have a compatibility issue with Debian so try FreeBSD instead. I'm pretty sure the ZFS features (software) is the same. Do not upgrade your pool if asked, you do not need to do that.

 

[BarneyGold]

Also, why the fresh install? This isn't windows we are dealing with here.

To make sure this wasn't the root cause. I had my config backed up so it didn't cost anything to try.

And @Jailer is correct, you have six drives, where are the other two? Also, Clean Install? What was previously used? Can you roll back?

As stated in my original post I was running Scale 22.02.0.1 and did a fresh install of that same Scale version.

Something else you could try, grab a USB flash drive (stick) with 16GB (or similar), disconnect your boot drive, install TrueNAS Core 12.0-U8 to the USB Flash drive, boot the system and do a minimal configuration, and then import your pool. You might have a compatibility issue with Debian so try FreeBSD instead.

I just did, and I have the same problem as when running Scale, which is an infinite pool import. This permitted me however to get the error messages again. I attached the screenshots, see java viewer.png for the pool import error,

Out of curiosity, which two drives were you planning to replace and how did you conclude which ones those were? I'm asking a serious question because I want to know your thought process as I could help me help others later.

and pool state degraded.png for the assumptions about the drives.

I just started a long test on all six drives. I'll make sure to post the results as soon as I can get them, including the S/N this time.
A nice day to both of you!


edit : in pool state degraded.png, the original message I had stated that drive ZPW05AG8 was CRITICAL, before I offlined it.

 

[joeschmuck]

Make sure you backup any data you desire to retain. If the SMART long tests show no issues then I would destroy the pool and start all over. I only say that because I do not know how to fix a pool that you can only mount as read-only. I guess you could try Ubuntu and try to mount the pool but I'm not very hopeful. Someone else may have better help information.