Random disk errors

autisten · May 10, 2016

One of my Freenas installations have begun to act quite strange. The disks seems to experience random smart errors. These errors cause CKSUM errors in the zvol.
As mentioned, the errors are completely random. I have 9 3TB disks in raidz1, 7 WD Reds and 2 Seagates. The WD Reds are not very old, a couple of them have only been installed for 2 months.
The setup has been running fine for years, it is only recently that these issues started.
If I reboot the system, the errors vanish. But as soon as I put load on the system, they come back.
I have tried the following to solve the issue:
1: Change SATA cables
2: Install a new SATA controller
3: Install a new 850W PSU
My hardware is:
SuperMicro X8SIL-F
8GB ECC RAM
9 x 3TB Disks
I have attached a couple of screenshots showing the zpools current status, and the output of smartctl -a on one of the disks.

danb35 · May 11, 2016

9 x 3 TB disks in RAIDZ1? You must not care very much about your data.

Please post the full output of smartctl -x on one of the drives, in code tags.

autisten · May 11, 2016

I know about the raidz1, I did not do the initial install. Fortunately, there is a backup of all the data.

Output of smartctl -x:

Code:

smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.3-RELEASE amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N5LAPTXK
LU WWN Device Id: 5 0014ee 2621e7904
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed May 11 12:47:42 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (39540) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 397) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   173   173   021    -    6308
  4 Start_Stop_Count        -O--CK   100   100   000    -    16
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   100   253   000    -    0
  9 Power_On_Hours          -O--CK   097   097   000    -    2243
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    15
192 Power-Off_Retract_Count -O--CK   200   200   000    -    3
193 Load_Cycle_Count        -O--CK   200   200   000    -    1549
194 Temperature_Celsius     -O---K   109   107   000    -    41
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 3054 (device log contains only the most recent 24 errors)
    CR     = Command Register
    FEATR  = Features Register
    COUNT  = Count (was: Sector Count) Register
    LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
    LH     = LBA High (was: Cylinder High) Register    ]   LBA
    LM     = LBA Mid (was: Cylinder Low) Register      ] Register
    LL     = LBA Low (was: Sector Number) Register     ]
    DV     = Device (was: Device/Head) Register
    DC     = Device Control Register
    ER     = Error register
    ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3054 [5] occurred at disk power-on lifetime: 1309 hours (54 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 46 00 00 00 00 00 00 40 00  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  ef 00 03 00 46 00 00 00 00 00 00 40 00  5d+14:25:54.064  SET FEATURES [Set transfer mode]
  ef 00 03 00 46 00 00 00 00 00 00 40 00  5d+14:25:54.061  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 00 00 00 00 00 40 00  5d+14:25:54.061  IDENTIFY DEVICE
  b0 00 da 00 00 00 00 00 c2 4f 00 e0 00  5d+11:20:55.080  SMART RETURN STATUS
  b0 00 d8 00 00 00 00 00 c2 4f 00 e0 00  5d+11:20:54.523  SMART ENABLE OPERATIONS

Error 3053 [4] occurred at disk power-on lifetime: 1309 hours (54 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 46 00 00 00 00 00 00 40 00  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  ef 00 03 00 46 00 00 00 00 00 00 40 00  5d+14:25:54.061  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 00 00 00 00 00 40 00  5d+14:25:54.061  IDENTIFY DEVICE
  b0 00 da 00 00 00 00 00 c2 4f 00 e0 00  5d+11:20:55.080  SMART RETURN STATUS
  b0 00 d8 00 00 00 00 00 c2 4f 00 e0 00  5d+11:20:54.523  SMART ENABLE OPERATIONS
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:46.513  READ VERIFY SECTOR(S) EXT

Error 3052 [3] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:46.513  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:37.998  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:29.483  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:20.968  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:12.453  READ VERIFY SECTOR(S) EXT

Error 3051 [2] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:37.998  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:29.483  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:20.968  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:12.453  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:03.938  READ VERIFY SECTOR(S) EXT

Error 3050 [1] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:29.483  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:20.968  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:12.453  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:03.938  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:55.423  READ VERIFY SECTOR(S) EXT

Error 3049 [0] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:20.968  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:12.453  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:03.938  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:55.423  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:46.908  READ VERIFY SECTOR(S) EXT

Error 3048 [23] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:12.453  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:03.938  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:55.423  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:46.908  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:38.393  READ VERIFY SECTOR(S) EXT

Error 3047 [22] occurred at disk power-on lifetime: 1306 hours (54 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 61 00 01 00 00 00 fa c5 3e e0 00  Device Fault; Error: ABRT at LBA = 0x00fac53e = 16434494

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:20:03.938  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:55.423  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:46.908  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:38.393  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 00 00 fa c5 3e e0 00  5d+11:19:29.878  READ VERIFY SECTOR(S) EXT

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      1153         -
# 2  Short offline       Completed without error       00%      1057         -
# 3  Extended offline    Completed without error       00%       825         -
# 4  Short offline       Completed without error       00%       769         -
# 5  Short offline       Completed without error       00%       650         -
# 6  Extended offline    Completed without error       00%       489         -
# 7  Short offline       Completed without error       00%       411         -
# 8  Short offline       Completed without error       00%       315         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    41 Celsius
Power Cycle Min/Max Temperature:     37/41 Celsius
Lifetime    Min/Max Temperature:      2/43 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (399)

Index    Estimated Time   Temperature Celsius
400    2016-05-11 04:50    41  **********************
...    ..(148 skipped).    ..  **********************
  71    2016-05-11 07:19    41  **********************
  72    2016-05-11 07:20    37  ******************
...    ..( 83 skipped).    ..  ******************
156    2016-05-11 08:44    37  ******************
157    2016-05-11 08:45    38  *******************
...    ..( 22 skipped).    ..  *******************
180    2016-05-11 09:08    38  *******************
181    2016-05-11 09:09    39  ********************
...    ..( 19 skipped).    ..  ********************
201    2016-05-11 09:29    39  ********************
202    2016-05-11 09:30    40  *********************
...    ..( 25 skipped).    ..  *********************
228    2016-05-11 09:56    40  *********************
229    2016-05-11 09:57    41  **********************
...    ..(169 skipped).    ..  **********************
399    2016-05-11 12:47    41  **********************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           12  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           11  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4        57189  Vendor specific

[root@filer] ~#

Bidule0hm · May 11, 2016

Looks like the SATA controller is the problem (or the cables). How the drives are connected to the system?

danb35 · May 11, 2016

...and what are you using for a SATA controller, anyway?

Nothing looks grossly wrong with the SMART output. The drive is a little warmer than we like to see (40C max for best life), and has more load cycles than I'd expect (over 1500, for only 15 power cycles), but neither of those is causing your problem. A bigger problem is that this drive hasn't seen a SMART test in over 1000 hours. These should be scheduled regularly--I run a short test daily and a long test weekly, but you could go with intervals as long as weekly for the short tests and monthly for the longs.

With that said, though, the most common cause of these kinds of problems is either the controller or the cables.

autisten · May 11, 2016

6 of the drives are connected directly to the motherboard SATA controller. The rest of the drives are connected to a STLabs SATA card.
I have a suspicion that the fault lies with the onboard controller. But I do not know why it will suddenly fail like this. There are no obvious hardware errors.
I will try to see, if I can narrow it down to the controller itself. If it is faulty, a new motherboard might be the sane solution.

danb35 · May 11, 2016

With the exception of LSI SAS HBAs, I don't think experience with "SATA cards" has been very positive here. Is there any pattern to the drives showing the faults? Are they always the ones on the motherboard? Always the ones on the SATA card? Or some combination?

An LSI 9211-8i and a pair of breakout cables would give you connectivity for 8 of the drives with a known-good solution, but would cost around $100 here.

autisten · May 11, 2016

Yeah I know people here do not approve SATA cards, but I have not had a fault on several different SATA cards the past many years.
Right now it seems like it's only the onboard ports. But I will test that later today.

Important Announcement for the TrueNAS Community.

Random disk errors

autisten

Cadet

Attachments

danb35

Hall of Famer

autisten

Cadet

Bidule0hm

Server Electronics Sorcerer

danb35

Hall of Famer

autisten

Cadet

danb35

Hall of Famer

autisten

Cadet

Similar threads