Had a strange drive failure and errors I've never seen before

[MDKAOD]

[Edit] Sorry, this should probably be in the Sorage sub-forum. I mis-posted.

Hey all, so last week I was asking about preventative failure drive replacement, and after getting all of the information together I triggered a replacement of da0 with da8. After resilvering da8 then scrubbing the volume, I had a drive in the good array (da7) that went crazy, throwing all sorts of errors:

Code:

May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00 length 65536 SMID 312 terminated ioc 804b scsi 0 state c xfer 0
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00 length 65536 SMID 785 terminated ioc 804b scsi 0 state c xfer 0
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 00 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Error 5, Retries exhausted
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 82 88 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Error 5, Retries exhausted
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 83 08 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 74 70 68 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 74 70 6a 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 83 90 00 01 00 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 83 08 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 74 70 68 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 74 70 6a 90 00 00 10 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 83 90 00 01 00 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 54 3f 83 08 00 00 80 00
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:45 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)

<there are a lot of these lines>

May 23 01:16:46 library kernel: (da7:mps1:0:3:0): Retrying command (per sense data)
May 23 01:16:46 library kernel: (da7:mps1:0:3:0): READ(10). CDB: 28 00 74 70 6d 87 00 00 01 00
May 23 01:16:46 library kernel: (da7:mps1:0:3:0): CAM status: SCSI Status Error
May 23 01:16:46 library kernel: (da7:mps1:0:3:0): SCSI status: Check Condition
May 23 01:16:46 library kernel: (da7:mps1:0:3:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
May 23 01:16:46 library kernel: (da7:mps1:0:3:0): Error 5, Retries exhausted

zpool status showed a bunch of readwrite errors and a couple checksum errors. I replaced the "failed" da7 disk with da9, resilvered and everything was happy. da7 seems to still be working fine:

Code:

[root@library] ~# smartctl -x /dev/da7
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p15 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Blue (SATA 6Gb/s)
Device Model:     WDC WD10EZEX-00KUWA0
Serial Number:    WD-WCC1S7329586
LU WWN Device Id: 5 0014ee 2b421c8ad
Firmware Version: 15.01H15
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May 26 11:51:31 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (10500) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 115) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30b5) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   180   175   021    -    1966
  4 Start_Stop_Count        -O--CK   100   100   000    -    16
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   086   086   000    -    10911
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    16
192 Power-Off_Retract_Count -O--CK   200   200   000    -    15
193 Load_Cycle_Count        -O--CK   200   200   000    -    0
194 Temperature_Celsius     -O---K   115   111   000    -    28
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     10876         -
# 2  Extended offline    Completed without error       00%     10807         -
# 3  Short offline       Completed without error       00%     10781         -
# 4  Short offline       Completed without error       00%     10733         -
# 5  Short offline       Completed without error       00%     10685         -
# 6  Short offline       Completed without error       00%     10637         -
# 7  Short offline       Completed without error       00%     10589         -
# 8  Short offline       Completed without error       00%     10541         -
# 9  Short offline       Completed without error       00%     10493         -
#10  Extended offline    Completed without error       00%     10472         -
#11  Short offline       Completed without error       00%     10445         -
#12  Short offline       Completed without error       00%     10397         -
#13  Short offline       Completed without error       00%     10349         -
#14  Short offline       Completed without error       00%     10301         -
#15  Short offline       Completed without error       00%     10253         -
#16  Short offline       Completed without error       00%     10205         -
#17  Short offline       Completed without error       00%     10157         -
#18  Short offline       Completed without error       00%     10109         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    28 Celsius
Power Cycle Min/Max Temperature:     28/31 Celsius
Lifetime    Min/Max Temperature:     22/32 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (334)

Index    Estimated Time   Temperature Celsius
335    2015-05-26 03:54    31  ************
...    ..(171 skipped).    ..  ************
  29    2015-05-26 06:46    31  ************
  30    2015-05-26 06:47    30  ***********
...    ..( 11 skipped).    ..  ***********
  42    2015-05-26 06:59    30  ***********
  43    2015-05-26 07:00    29  **********
...    ..( 55 skipped).    ..  **********
  99    2015-05-26 07:56    29  **********
100    2015-05-26 07:57    28  *********
...    ..(205 skipped).    ..  *********
306    2015-05-26 11:23    28  *********
307    2015-05-26 11:24    31  ************
...    ..( 26 skipped).    ..  ************
334    2015-05-26 11:51    31  ************

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            1  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            2  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       296933  Vendor specific

[root@library] ~#

Searching around the forums for this error set is tricky, but it seems that it might be a cable problem? Potentially something else, like an actual failure? Thanks for any insight!

 

[Bidule0hm]

I think it was a bad connexion. You've probably moved the cables a bit when you've replaced da0.

 

[cyberjock]

Actually, that problem looks like it might be in the mps controller.

Can I take a guess that if you log into the WebGUI and click on the stoplight it tells you that your firmware has a mismatch?

 

[MDKAOD]

@cyberjock
https://forums.freenas.org/index.ph...sum-errors-w-win8-1-search.22532/#post-135187
I haven't upgraded to 9.3 yet, but I am on 9.2.1.9 and my controllers are M1015's flashed to IT mode on firmaware P16. As of right now, I do not show a firmware mismatch through the stoplight.

@Bidule0hm I didn't physically touch the disks. It's a 24 bay case with plenty of spare slots that I populated with these two drives (da8 & da9) specifically, months ago. da7 was untouched.

Code:

[root@library] ~# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 14.00.00.00 (2012.07.04)
Copyright (c) 2008-2012 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2008(B1)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B1)     16.00.00.00    10.00.00.06      No Image      00:01:00:00
1  SAS2008(B2)     16.00.00.00    10.00.00.06      No Image      00:02:00:00
2  SAS2008(B1)     16.00.00.00    10.00.00.06      No Image      00:03:00:00

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

 

[cyberjock]

Well, that kills the easy idea. You seem to be left with a few options:

1. Bad PSU, or at least flaky enough that disks are not happy.
2. Cabling.
3. Bug (pretty unlikely IMO)
4. Hardware problem with the SAS controller
5. Hardware problem with the hard drive
6. Other unspecified hardware problem (MB, CPU, RAM, etc.) but this is much less likely than the stuff above.

 

[MDKAOD]

Well, damn. Any tips on where to begin? I have an unused set of sas cables I could swap. It's hard to think the PSU would be a problem, but weirder things have happened I guess, but going back to that previous thread where I was having checksum errors (still crop up infrequently and on random disks) I guess it could be something more system wide than the controllers. The cables are Monoprice 8087 pass through's and I've never had issues with Monoprice's stuff. The Norco backplanes have had problems in the past, I wonder if that's what I'm dealing with here.

 

[Eltimba00]

I've been battling this same issue for about two months now, I even replaced a drive thinking it might be the drive but random drives keep disconnecting and keep having to be resilvered. All my drives are in a HDD enclosure like this. I had 8087's from Monoprice as well like MDKAOD and swapped them for better ones from newegg, still disconnects and checksums seem to happen more often now. My next troubleshooting step will be to remove the drives from the enclosure and hook them up directly to the cables.


Here's a sample of my syslog:

Code:

Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted

 

[SweetAndLow]

I've been battling this same issue for about two months now, I even replaced a drive thinking it might be the drive but random drives keep disconnecting and keep having to be resilvered. All my drives are in a HDD enclosure like this. I had 8087's from Monoprice as well like MDKAOD and swapped them for better ones from newegg, still disconnects and checksums seem to happen more often now. My next troubleshooting step will be to remove the drives from the enclosure and hook them up directly to the cables.


Here's a sample of my syslog:

Code:

Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(16). CDB: 88 00 00 00 00 01 5d 50 a3 87 00 00 00 01 00 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 81 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Retrying command (per sense data)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): READ(10). CDB: 28 00 00 40 00 80 00 00 01 00
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): CAM status: SCSI Status Error
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI status: Check Condition
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun  1 06:28:23 freenas (da3:mpslsi1:0:4:0): Error 5, Retries exhausted

You should have created your own thread. But your problem is 100% the fault of that enclosure and I suspect you are going to get an ear full when it comes to your other hardware selections. Good luck.

 

[Eltimba00]

You should have created your own thread. But your problem is 100% the fault of that enclosure and I suspect you are going to get an ear full when it comes to your other hardware selections. Good luck.

I didn't know you could guess the rest of my hardware choices based on a single particular piece of hardware I have chosen to use. I only used the enclosure because I had them on my previous file server before I delved into the world of FreeNAS. They worked flawlessly for 8 years, so I had no suspicion that they would be a problem with my new hardware combination. Below you'll see the rest of my hardware, please let me know if you think if it's adequate to sustain a FreeNAS install.

Can you elaborate on the enclosure? Is this a known issue with this particular enclosure or should hot swap enclosures be avoided all together?

And I posted a reply to this thread because I hadn't seen this same error posted before and since it's a fairly new thread didn't think creating another thread would be warranted.



FreeNAS-9.3-STABLE-201505130355
Motherboard: Supermicro MBD- X10SL-7-F-O
CPU: Intel(R)Xeon(R) Processor E3-1246v3 3.5GHz
Memory: 16Gb Crucial ECC
HBA: LSI SAS9201-16i
Disks: 4x3TB WD- Green (Disabled Head Park w/ WDIDLE3) 1x3TB WD-Red (the one I replaced when I first started troubleshooting my issue.)
Case: Lian-Li PC-A16

 

[Ericloewe]

I didn't know you could guess the rest of my hardware choices based on a single particular piece of hardware I have chosen to use. I only used the enclosure because I had them on my previous file server before I delved into the world of FreeNAS. They worked flawlessly for 8 years, so I had no suspicion that they would be a problem with my new hardware combination. Below you'll see the rest of my hardware, please let me know if you think if it's adequate to sustain a FreeNAS install.

Can you elaborate on the enclosure? Is this a known issue with this particular enclosure or should hot swap enclosures be avoided all together?

And I posted a reply to this thread because I hadn't seen this same error posted before and since it's a fairly new thread didn't think creating another thread would be warranted.



FreeNAS-9.3-STABLE-201505130355
Motherboard: Supermicro MBD- X10SL-7-F-O
CPU: Intel(R)Xeon(R) Processor E3-1246v3 3.5GHz
Memory: 16Gb Crucial ECC
HBA: LSI SAS9201-16i
Disks: 4x3TB WD- Green (Disabled Head Park w/ WDIDLE3) 1x3TB WD-Red (the one I replaced when I first started troubleshooting my issue.)
Case: Lian-Li PC-A16

Nothing looks wrong (the HDD hotswap bay is also fine, barring any defects). What's the PSU?
I'd go through Cyberjock's suggestions a few posts up, but I'm afraid this will have to be plain ol' troubleshooting.

 

[SweetAndLow]

I didn't know you could guess the rest of my hardware choices based on a single particular piece of hardware I have chosen to use. I only used the enclosure because I had them on my previous file server before I delved into the world of FreeNAS. They worked flawlessly for 8 years, so I had no suspicion that they would be a problem with my new hardware combination. Below you'll see the rest of my hardware, please let me know if you think if it's adequate to sustain a FreeNAS install.

Can you elaborate on the enclosure? Is this a known issue with this particular enclosure or should hot swap enclosures be avoided all together?

And I posted a reply to this thread because I hadn't seen this same error posted before and since it's a fairly new thread didn't think creating another thread would be warranted.



FreeNAS-9.3-STABLE-201505130355
Motherboard: Supermicro MBD- X10SL-7-F-O
CPU: Intel(R)Xeon(R) Processor E3-1246v3 3.5GHz
Memory: 16Gb Crucial ECC
HBA: LSI SAS9201-16i
Disks: 4x3TB WD- Green (Disabled Head Park w/ WDIDLE3) 1x3TB WD-Red (the one I replaced when I first started troubleshooting my issue.)
Case: Lian-Li PC-A16

When you used the word enclosure I thought you meant one of those external USB raid enclosures. Looks kind you are just using a sata backplane which should be fine.

 

[fx24]

MDKAOD and Eltimba00,

Have either of you figured out a solution to this? I have been fighting with this problem for about 6 months now. It seemed to have gone away for a month or two and then reappeared this week. It persisted through the FreeNAS upgrade and firmware updates as well.

I also have the Supermicro MBD- X10SL-7-F-O motherboard and LSI SAS9201-16i HBA but within a Norco 4224 case.

Cyberjock,

Thank you for providing suggestions. I am going to start with new data cables. I already replaced the power leads to my backplanes obviously to no avail. I just recently picked up a spare PSU and now have the luxury of trying that out.

Such randomness is driving me crazy.

 

[MDKAOD]

I'm also using the Norco 4224, a̶n̶d̶ ̶t̶h̶e̶ ̶p̶r̶o̶b̶l̶e̶m̶ ̶d̶o̶e̶s̶ ̶c̶o̶m̶e̶ ̶b̶a̶c̶k̶ ̶o̶c̶c̶a̶s̶i̶o̶n̶a̶l̶l̶y̶.̶ ̶M̶u̶c̶h̶ ̶l̶e̶s̶s̶ ̶f̶r̶e̶q̶u̶e̶n̶t̶l̶y̶ ̶t̶h̶e̶s̶e̶ ̶d̶a̶y̶s̶,̶ ̶a̶n̶d̶ ̶I̶ ̶h̶a̶v̶e̶n̶'̶t̶ ̶s̶e̶e̶n̶ ̶i̶t̶ ̶s̶i̶n̶c̶e̶ ̶I̶ ̶u̶p̶g̶r̶a̶d̶e̶d̶ ̶t̶o̶ ̶9̶.̶3̶ ̶a̶ ̶f̶e̶w̶ ̶w̶e̶e̶k̶s̶ ̶b̶a̶c̶k̶,̶ ̶b̶u̶t̶ ̶i̶t̶ ̶h̶a̶s̶n̶'̶t̶ ̶b̶e̶e̶n̶ ̶l̶o̶n̶g̶ ̶e̶n̶o̶u̶g̶h̶ ̶f̶o̶r̶ ̶m̶e̶ ̶t̶o̶ ̶s̶a̶y̶ ̶i̶t̶'̶s̶ ̶g̶o̶n̶e̶.̶

[Edit] I sat down during lunch to re-read this, this isn't the issue I thought it was. I haven't seen this issue since the one time it happened to me. I still get random checksum errors cropping up randomly, but not this one. Sorry for the misinformation!

 

[fx24]

I'm also using the Norco 4224, and the problem does come back occasionally. Much less frequently these days, and I haven't seen it since I upgraded to 9.3 a few weeks back, but it hasn't been long enough for me to say it's gone.

I see that you are using 3x M1015 cards so I am inclined to believe it unrelated to SAS controllers since me and Eltimba00 are using LSI SAS9201-16i. The SAS backplane doesn't seem likely for me because the problem occurs across about 3-4 different drives on different backplanes, but...

I read somewhere that the firmware for the backplanes can be updated. I wonder if that could be the culprit.

 

[MDKAOD]

See my edit above. Also, I've read there were updated back planes at one time, but I didn't see anything about firmware. That would be interesting to read more about.

 

[fx24]

I'm also using the Norco 4224, a̶n̶d̶ ̶t̶h̶e̶ ̶p̶r̶o̶b̶l̶e̶m̶ ̶d̶o̶e̶s̶ ̶c̶o̶m̶e̶ ̶b̶a̶c̶k̶ ̶o̶c̶c̶a̶s̶i̶o̶n̶a̶l̶l̶y̶.̶ ̶M̶u̶c̶h̶ ̶l̶e̶s̶s̶ ̶f̶r̶e̶q̶u̶e̶n̶t̶l̶y̶ ̶t̶h̶e̶s̶e̶ ̶d̶a̶y̶s̶,̶ ̶a̶n̶d̶ ̶I̶ ̶h̶a̶v̶e̶n̶'̶t̶ ̶s̶e̶e̶n̶ ̶i̶t̶ ̶s̶i̶n̶c̶e̶ ̶I̶ ̶u̶p̶g̶r̶a̶d̶e̶d̶ ̶t̶o̶ ̶9̶.̶3̶ ̶a̶ ̶f̶e̶w̶ ̶w̶e̶e̶k̶s̶ ̶b̶a̶c̶k̶,̶ ̶b̶u̶t̶ ̶i̶t̶ ̶h̶a̶s̶n̶'̶t̶ ̶b̶e̶e̶n̶ ̶l̶o̶n̶g̶ ̶e̶n̶o̶u̶g̶h̶ ̶f̶o̶r̶ ̶m̶e̶ ̶t̶o̶ ̶s̶a̶y̶ ̶i̶t̶'̶s̶ ̶g̶o̶n̶e̶.̶

[Edit] I sat down during lunch to re-read this, this isn't the issue I thought it was. I haven't seen this issue since the one time it happened to me. I still get random checksum errors cropping up randomly, but not this one. Sorry for the misinformation!

My problem started out as just random checksum errors that there were no more than around 200, but sometimes just 1.

Only two times now, I have had error messages that looked like it was related to the controller, driver or firmware issues which scares me much more.

Thanks for clarifying that though. It seems our errors have a lot of common denominators.

 

[MDKAOD]

Sure thing, and I certainly agree our issues are very similar. There's some more information on my checksum errors in this thread, if it helps you out.