Drive faulted when trying to resilver

[patrickRx]

Hello, I had a drive that started giving read and write errors so I warrantied it. When trying to get resilver the drive, it faults with multiple read and write errors. I've tried this multiple times. So apparently not the drive. I've tried with a different cable as well. Can anyone help me begin to troubleshoot this? Here is the output of zpool status.

Code:

root@freenas[~]# root@freenas[~]# zpool status data01
  pool: data01
 state: DEGRADED
  scan: resilvered 257G in 0 days 11:48:40 with 0 errors on Tue Mar 10 00:28:06 2020
config:

        NAME                        STATE     READ WRITE CKSUM
        data01                      DEGRADED     0     0     0
          raidz2-0                  DEGRADED     0     0     0
            da0p2                   ONLINE       0     0     0
            da1p2                   ONLINE       0     0     0
            da2p2                   ONLINE       0     0     0
            da3p2                   ONLINE       0     0     0
            da4p2                   ONLINE       0     0     0
            replacing-5             UNAVAIL      0     0     0
              18047286444837006076  OFFLINE      0     0     0  was /dev/da7p1
              da6                   FAULTED      6 9.45K     0  too many errors

Thank you.

 

[hervon]

Did you test the new drive before inserting it in the pool ? Might be faulty too. Since you did not respect the forum guidelines about your current hardware it is hard to help much further.

 

[patrickRx]

Sorry I clearly missed that step. Hardware is as follows:

CPU: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
Motherboard: ASUS P8H77-M LE LGA 1155
Ram: 16GB non-ECC
Drive controller: SAS-9220-8i flashed to interface mode
Drives: 6x Seagate Barracuda 4tb

As for testing the drive, would you recommend running a SMART long test?

Thanks again.

 

[hervon]

Long Smart test is a way to go. Also search the forums for disk burn-in test before putting them in pool.

 

[Redcoat]

Use badblocks as a burn-in. See Uncle Fester's Guide in the resource section for a very full description on how to run it.

 

[patrickRx]

Hello, I did run all the tests in the Fester's Guide with no errors. The SMART results are below. I'm concerned that this in fact a problem with my controller card. I tried with two different cables, and I'm getting read and write errors with no checksum errors on the new drive, same as the old drive. Any other ideas besides buying a new card? Thank you!

Code:

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM004-2CV104
Serial Number:    ZFN0EC4Y
LU WWN Device Id: 5 000c50 0a48f6b68
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Mar 11 17:53:33 2020 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 493) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   067   006    Pre-fail  Always       -       179112192
  3 Spin_Up_Time            0x0003   099   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   070   060   045    Pre-fail  Always       -       9607330
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       211 (26 15 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       2
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   098   000    Old_age   Always       -       0 0 13
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   073   070   040    Old_age   Always       -       27 (Min/Max 23/29)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       10
194 Temperature_Celsius     0x0022   027   040   000    Old_age   Always       -       27 (0 23 0 0 0)
195 Hardware_ECC_Recovered  0x001a   083   067   000    Old_age   Always       -       179112192
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       203h+35m+31.700s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3962422120
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       164740

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%       211         -
# 2  Short offline       Completed without error       00%       211         -
# 3  Extended offline    Completed without error       00%       186         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Redcoat]

I guess, given the time, you didn't yet run badblocks? Anyway, I suggest you might switch ports on your card and try again to see if it's a specific port problem. Maybe switch the slot the card is in also as another test.

 

[patrickRx]

Thank you for your response. I did run badblocks, although if you think it was too quick perhaps I did something incorrectly? I used the following command:

Code:

badblocks –b 4096 –vws /dev/da6

I have tried both ports on the controller card with two different cables. Unfortunately I only have one PCIE slot long enough on my motherboard. Are there any ways to test the controller card without buying a new one? My one option I can think of would be to use the motherboards SATA slots and see if that solves the problem, however there are only 6 so I would need to move my boot partition to a USB drive first (still supported on FreeNAS?).

Thank you again.

 

[Redcoat]

The badblocks line looks fine. My 4TB WD Reds took approx 24 hours per test and it seems yours was shorter...

Your suggestion is an interesting option for a test at least. USB's not preferred these days because of short life-expectancy, but not forbidden.

 

[patrickRx]

Thanks I’ll give it a try!

 

[patrickRx]

It was the card. Thank you all for your help.