SOLVED CAM status: CCB request completed with an error

[Jeremy Hays]

Hey all,

I have a box I put together a while back, and has been flawless. Now I am getting alot of

Code:

(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: SCSI Status Error
(da1:mps0:0:1:0): SCSI status: Check Condition
(da1:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da1:mps0:0:1:0): Retrying command (per sense data)
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 307 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 308 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
0
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: SCSI Status Error
(da0:mps0:0:0:0): SCSI status: Check Condition
(da0:mps0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da0:mps0:0:0:0): Retrying command (per sense data)
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00 length 131072 SMID 344 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 345 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
0
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 355 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da2: 0
mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da2:mps0:0:2:0): Retrying command (per sense data)
(da5:mps0:0:5:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da5:mps0:0:5:0): CAM status: SCSI Status Error
(da5:mps0:0:5:0): SCSI status: Check Condition
(da5:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da5:mps0:0:5:0): Retrying command (per sense data)
        (da3:mps0:0:3:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 603 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
(da3:mps0:0:3:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da3:mps0:0:3:0): CAM status: CCB request completed with an error
(da3:mps0:0:3:0): Retrying command
(da3:mps0:0:3:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da3:mps0:0:3:0): CAM status: SCSI Status Error
(da3:mps0:0:3:0): SCSI status: Check Condition
(da3:mps0:0:3:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da3:mps0:0:3:0): Retrying command (per sense data)
        (da4:mps0:0:4:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 606 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
(da4:mps0:0:4:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da4:mps0:0:4:0): Retrying command (per sense data)
(da6:mps0:0:6:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da6:mps0:0:6:0): CAM status: SCSI Status Error
(da6:mps0:0:6:0): SCSI status: Check Condition
(da6:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da6:mps0:0:6:0): Retrying command (per sense data)
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 539 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 545 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
0
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: SCSI Status Error
(da0:mps0:0:0:0): SCSI status: Check Condition
(da0:mps0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da0:mps0:0:0:0): Retrying command (per sense data)
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00 length 131072 SMID 565 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 576 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
0
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da2:mps0:0:2:0): Retrying command (per sense data)
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00 length 131072 SMID 540 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 552 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da1:mps0:0:1:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
0
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 564 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da1: 0
mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da1:mps0:0:1:0): CAM status: SCSI Status Error
(da1:mps0:0:1:0): SCSI status: Check Condition
(da1:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da1:mps0:0:1:0): Retrying command (per sense data)
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00 length 131072 SMID 778 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 790 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da0:mps0:0:0:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
0
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 802 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da0: 0
mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da0:mps0:0:0:0): CAM status: SCSI Status Error
(da0:mps0:0:0:0): SCSI status: Check Condition
(da0:mps0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da0:mps0:0:0:0): Retrying command (per sense data)
(da6:mps0:0:6:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da6:mps0:0:6:0): CAM status: SCSI Status Error
(da6:mps0:0:6:0): SCSI status: Check Condition
(da6:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da6:mps0:0:6:0): Retrying command (per sense data)
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 803 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 815 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
0
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da1:mps0:0:1:0): CAM status: SCSI Status Error
(da1:mps0:0:1:0): SCSI status: Check Condition
(da1:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da1:mps0:0:1:0): Retrying command (per sense data)
        (da5:mps0:0:5:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 861 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
(da5:mps0:0:5:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da5:mps0:0:5:0): CAM status: CCB request completed with an error
(da5:mps0:0:5:0): Retrying command
(da5:mps0:0:5:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da5:mps0:0:5:0): CAM status: SCSI Status Error
(da5:mps0:0:5:0): SCSI status: Check Condition
(da5:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da5:mps0:0:5:0): Retrying command (per sense data)
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00 length 90112 SMID 806 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er 0
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 838 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
0
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 849 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da2: 0
mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da2:mps0:0:2:0): Retrying command (per sense data)
        (da3:mps0:0:3:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00 length 131072 SMID 850 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
(da3:mps0:0:3:0): READ(10). CDB: 28 00 00 40 03 80 00 01 00 00
(da3:mps0:0:3:0): CAM status: CCB request completed with an error
(da3:mps0:0:3:0): Retrying command
(da3:mps0:0:3:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da3:mps0:0:3:0): CAM status: SCSI Status Error
(da3:mps0:0:3:0): SCSI status: Check Condition
(da3:mps0:0:3:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da3:mps0:0:3:0): Retrying command (per sense data)
(da4:mps0:0:4:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da4:mps0:0:4:0): Retrying command (per sense data)
(da7:mps0:0:7:0): READ(10). CDB: 28 00 00 40 01 d0 00 00 b0 00
(da7:mps0:0:7:0): CAM status: SCSI Status Error
(da7:mps0:0:7:0): SCSI status: Check Condition
(da7:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da7:mps0:0:7:0): Retrying command (per sense data)
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 28 40 29 38 00 00 08 00 length 4096 SMID 973 terminated ioc 804b scsi 0 state 0 xfe                                                                                                           r 0
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00 length 28672 SMID 981 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er (da0:mps0:0:0:0): READ(10). CDB: 28 00 28 40 29 38 00 00 08 00
0
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
        (da0:mps0:0:0:0): READ(10). CDB: 28 00 28 40 22 38 00 00 10 00 length 8192 SMID 953 terminated ioc 804b scsi 0 state 0 xfe                                                                                                           r 0(da0:
mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 28 40 22 38 00 00 10 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 38 00
(da0:mps0:0:0:0): CAM status: SCSI Status Error
(da0:mps0:0:0:0): SCSI status: Check Condition
(da0:mps0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
        (da2:mps0:0:2:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00 length 28672 SMID 979 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er (da0:0
mps0:0:0:0): Retrying command (per sense data)
(da2:mps0:0:2:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 38 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da2:mps0:0:2:0): Retrying command (per sense data)
        (da3:mps0:0:3:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00 length 28672 SMID 978 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er 0
(da3:mps0:0:3:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00
(da3:mps0:0:3:0): CAM status: CCB request completed with an error
(da3:mps0:0:3:0): Retrying command
(da3:mps0:0:3:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 38 00
(da3:mps0:0:3:0): CAM status: SCSI Status Error
(da3:mps0:0:3:0): SCSI status: Check Condition
(da3:mps0:0:3:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da3:mps0:0:3:0): Retrying command (per sense data)
        (da4:mps0:0:4:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00 length 28672 SMID 977 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er 0
(da4:mps0:0:4:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 38 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da4:mps0:0:4:0): Retrying command (per sense data)
        (da7:mps0:0:7:0): READ(10). CDB: 28 00 16 4e d8 d0 00 00 08 00 length 4096 SMID 954 terminated ioc 804b scsi 0 state 0 xfe                                                                                                           r 0
        (da7:mps0:0:7:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 30 00 length 24576 SMID 974 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er (da7:mps0:0:7:0): READ(10). CDB: 28 00 16 4e d8 d0 00 00 08 00
0
(da7:mps0:0:7:0): CAM status: CCB request completed with an error
        (da7:mps0:0:7:0): READ(10). CDB: 28 00 28 40 22 38 00 00 08 00 length 4096 SMID 946 terminated ioc 804b scsi 0 state 0 xfe                                                                                                           r 0(da7:
mps0:0:7:0): Retrying command
(da7:mps0:0:7:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 30 00
(da7:mps0:0:7:0): CAM status: CCB request completed with an error
(da7:mps0:0:7:0): Retrying command
(da7:mps0:0:7:0): READ(10). CDB: 28 00 28 40 22 38 00 00 08 00
(da7:mps0:0:7:0): CAM status: CCB request completed with an error
(da7:mps0:0:7:0): Retrying command
(da7:mps0:0:7:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 30 00
(da7:mps0:0:7:0): CAM status: SCSI Status Error
(da7:mps0:0:7:0): SCSI status: Check Condition
(da7:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da7:mps0:0:7:0): Retrying command (per sense data)
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00 length 28672 SMID 980 terminated ioc 804b scsi 0 state 0 xf                                                                                                           er 0
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 28 4b fb 88 00 00 08 00 length 4096 SMID 944 terminated ioc 804b scsi 0 state 0 xfe                                                                                                           r 0(da1:mps0:0:1:0): READ(10). CDB: 28 00 16 40 84 a0 00 00 38 00

(da5:mps0:0:5:0): CAM status: CCB request completed with an error
(da5:mps0:0:5:0): Retrying command
(da5:mps0:0:5:0): READ(10). CDB: 28 00 16 4e d8 d0 00 00 08 00
(da5:mps0:0:5:0): CAM status: CCB request completed with an error
(da5:mps0:0:5:0): Retrying command
(da5:mps0:0:5:0): READ(10). CDB: 28 00 26 ad 21 08 00 00 30 00
(da5:mps0:0:5:0): CAM status: SCSI Status Error
(da5:mps0:0:5:0): SCSI status: Check Condition
(da5:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da5:mps0:0:5:0): Retrying command (per sense data)

Machine is as follows:
TYAN S5535-HE
Xeon E3-1220 v3
16gb DDR3 ECC
8x WD Reds
LSI (Avago) SAS9207-8i
Random Nvidia graphics card
EVGA 850 watt power supply (80 plus gold)

I have swapped out the SFF8087 breakout cables because I thought that was the problem. All the drives smart tables all look the same, they all have a high attribute 199 (Ultra ATA CRC Error Rate) Also have ran memtest86 overnight, no problems.

This issue just popped up on day and the client got an email from the nas and called me. The nas initially has 9.3 on it, I upgraded to 9.10 and the issue persists. ( I didnt think it would fix it) Machine might be 9 months old or so.

 

[Jeremy Hays]

dmesg

Code:

[root@freenas] ~# dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016
    root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/trueos/sys/FreeNA                                                                                                           S.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (3092.90-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,S                                                                                                           S,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MO                                                                                                           VBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17683185664 (16864 MB)
avail memory = 16496558080 (15732 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  2
cpu2 (AP): APIC ID:  4
cpu3 (AP): APIC ID:  6
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80e73aa0, 0) error 19
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem 0xf6000000-0xf6ffffff,0xe0000000-0xefffffff,0xf4000000-0xf5ffffff irq 16                                                                                                            at device 0.0 on pci1
vgapci0: Boot video device
xhci0: <Intel Lynx Point USB 3.0 controller> mem 0xf7620000-0xf762ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
pci0: <simple comms> at device 22.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xf020-0xf03f mem 0xf7600000-0xf761ffff,0xf7633000-0xf7633fff irq 20 at d                                                                                                           evice 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:81:ef:6c:18
ehci0: <Intel Lynx Point USB 2.0 controller USB-B> mem 0xf7632000-0xf76323ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xd000-0xd01f mem 0xf7200000-0xf72fffff,0xf7300000-0xf7303fff                                                                                                            irq 16 at device 0.0 on pci2
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:e0:81:ef:6c:19
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
mps0: <Avago Technologies (LSI) SAS2308> port 0xc000-0xc0ff mem 0xf7540000-0xf754ffff,0xf7500000-0xf753ffff irq 16 at device 0.0 o                                                                                                           n pci3
mps0: Firmware: 20.00.00.00, Driver: 20.00.00.00-fbsd
mps0: IOCCapabilities: 5285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
ehci1: <Intel Lynx Point USB 2.0 controller USB-A> mem 0xf7631000-0xf76313ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
ichwd0: <Intel Lynx Point watchdog timer> on isa0
orm0: <ISA Option ROMs> at iomem 0xd3000-0xd3fff,0xd4000-0xd4fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
random: unblocking device.
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <0x8086> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
mps0: SAS Address for SATA device = d1695a5c01bfdc8b
mps0: SAS Address for SATA device = d1636355facbe181
mps0: SAS Address for SATA device = d1695a5c01d5da8b
mps0: SAS Address for SATA device = d1655056dbdadd74
mps0: SAS Address for SATA device = d1655056dbd2e089
mps0: SAS Address for SATA device = d1685957fabfe173
mps0: SAS Address for SATA device = d1695a5c01bdc075
mps0: SAS Address for SATA device = d1646555f8debb96
mps0: SAS Address from SATA device = d1695a5c01bfdc8b
mps0: SAS Address from SATA device = d1636355facbe181
mps0: SAS Address from SATA device = d1695a5c01d5da8b
mps0: SAS Address from SATA device = d1655056dbdadd74
mps0: SAS Address from SATA device = d1655056dbd2e089
mps0: SAS Address from SATA device = d1685957fabfe173
mps0: SAS Address from SATA device = d1695a5c01bdc075
mps0: SAS Address from SATA device = d1646555f8debb96
uhub1: 2 ports with 2 removable, self powered
uhub0: 21 ports with 21 removable, self powered
uhub2: 3 ports with 3 removable, self powered
ugen0.2: <Microsoft> at usbus0
ukbd0: <Microsoft Wired Keyboard 600, class 0/0, rev 1.10/1.10, addr 1> on usbus0
kbd2 at ukbd0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1
uhub3: 6 ports with 6 removable, self powered
ugen2.2: <vendor 0x8087> at usbus2
uhub4: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2
uhub4: 8 ports with 8 removable, self powered
ugen0.3: <SanDisk> at usbus0
umass0: <SanDisk Cruzer, class 0/0, rev 2.10/1.00, addr 2> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:2:0:-1: Attached to scbus2
ugen0.4: <Logitech> at usbus0
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number      WD-WCC4M1TRRCZD
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da0: quirks=0x8<4K>
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number      WD-WCC4M7KYYLSN
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da1: quirks=0x8<4K>
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number      WD-WCC4M7KYY6UN
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da2: quirks=0x8<4K>
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number      WD-WCC4M3AS3RV7
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da3: quirks=0x8<4K>
da4 at mps0 bus 0 scbus0 target 4 lun 0
da4: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number      WD-WCC4M3AS3JYL
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
da4: quirks=0x8<4K>
da5 at mps0 bus 0 scbus0 target 5 lun 0
da5: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number      WD-WCC4M2VRPV4Y
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da5: quirks=0x8<4K>
da6 at mps0 bus 0 scbus0 target 6 lun 0
da6: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number      WD-WCC4M6JTR7Z6
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors)
da6: quirks=0x8<4K>
da7 at mps0 bus 0 scbus0 target 7 lun 0
da7: <ATA WDC WD20EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number      WD-WCC4M7KYY498
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 1907729MB (3907029168 512 byte sectors)
da7: quirks=0x8<4K>
da8 at umass-sim0 bus 0 scbus2 target 0 lun 0
da8: <SanDisk Cruzer 1.00> Removable Direct Access SPC-4 SCSI device
da8: Serial Number 4C531001420806114113
da8: 40.000MB/s transfers
da8: 30528MB (62521344 512 byte sectors)
da8: quirks=0x2<NO_6_BYTE>
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1546448357 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00 length 131072 SMID 321 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer 0
        (da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 length 131072 SMID 332 terminated ioc 804b scsi 0 state 0 x                                                                                                           fer(da1:mps0:0:1:0): READ(10). CDB: 28 00 e8 e0 85 80 00 01 00 00

 

[Spearfoot]

Hmmm... did you re-seat the LSI card?

 

[Jeremy Hays]

I have yes, another observation is that if I initiate a scrub, the scrub speed is REALLY slow to me. gettting 8-38M/s.

 

[Spearfoot]

You've re-seated the LSI card and replaced the cables, to no avail... Doesn't seem likely that eight hard drives would fail simultaneously, so I suspect the LSI card.

Do you have another HBA you can swap in to the system, to see if the LSI card has flaked out?

If not, do you have another computer (w/ "known-good" hard drives) in which to test the LSI card?

 

[Jeremy Hays]

I have RMA'd the LSI card. Popped the replacement in and started a scrub, it averaged north of 300M/s which is a huge improvement. I was getting some errors still, but nothing like it was, also I got some errors on 3 of the drives the scrub said. I have pulled the drives and I am currently running WDC data lifeguard diagnostics on them to see if any are bad. Will report back.

 

[Jeremy Hays]

Drives are all ok, WD data lifeguard tools, say all the drives are ok. Not sure how to proceed now.
Have replaced cables, LSI card, drives are ok, ram is fine.

 

[SweetAndLow]

A scrub will not tell you your drives are bad. A scrub tells you when you have corrupt data. SMART test will tell you if a drive is bad.

 

[Jeremy Hays]

I did not use a scrub to tell if the drives are bad, i used Western Digital's diagnostic tool to tell if the drives were bad.

 

[Robert Trevellyan]

Consider also running benchmarks on the drives. Sometimes all tests come up clear, but benchmarks reveal problems.

 

[Jeremy Hays]

so next trouble shooting step i have done, a clean install of Freenas 9.10.1 on a new flash drive. Plugged in 4 brand new disks. Still getting issues.

New HBA, new cables, new drives, same issues. Failing board, powersupply, cpu, ram????

im not sure what to look at next.

Memtest86 says the ram is fine on an overnight run.

 

[Robert Trevellyan]

Failing board, powersupply, cpu, ram????
im not sure what to look at next.
Memtest86 says the ram is fine on an overnight run.

Seems like you're left with PSU and mobo as the most likely culprits.

 

[Jeremy Hays]

So last trouble shooting step I have tried. I plugged the 4 new disks into the motherboard rather than the LSI HBA, drives are not giving errors like they did when plugged into the HBA.

Is it more likely I have a bad replacement from LSI than a bad pcie slot / cpu / motherboard ?

 

[Jeremy Hays]

I spoke to LSI, and they are concerned that the drives which are WD20EFRX have not been tested with this HBA. Does anyone else have these drives on this hba?

 

[Spearfoot]

I spoke to LSI, and they are concerned that the drives which are WD20EFRX have not been tested with this HBA. Does anyone else have these drives on this hba?

The WD20EFRX is Western Digital's Red NAS drive; I'm astonished that LSI would broach the idea that these drives, which are about as common as dirt, might not work with their cards. There are probably dozens of forum members using them with LSI cards.

I wonder if your Tyan mobo is simply incompatible with the LSI cards. I know I asked this earlier upthread... but do you have another system you can use to test the LSI HBA? If it works in another system, but not in the Tyan, that would prove that the two are incompatible.

Also, I would dig more deeply into the PCIe BIOS settings on the Tyan before giving up. Sadly, since Tyan isn't used by many (any?) users here, there isn't much in the way of institutional knowledge for you tap in to.

Good luck!

EDIT: Why don't you ask Tyan if their board is compatible with LSI HBAs? Might save some time...

 

[Jeremy Hays]

I spoke with Tyan, word is that they have not tested it or any other HBA's. But tech said he did not believe it was an incompatibility issue.
I moved the LSI card to two other mainboards, the first did not detect the card, the lsi bios did not show and Windows 10 did not detect the card. The 2nd board showed the LSI bios but did not detect the drives that were attached. Both of these boards are older but work great.

I put the HBA back on the Tyan board and attempted to boot freenas. The LSI card detected the drives and freenas started to boot, then the machine rebooted and no LSI card.... Powered off and back on, same shenanigans.

I am rma'ing the LSI AGAIN.... hopefully good news when the replacement comes in.

As another option, is there another HBA card out there that would suit my needs? I would much prefer something new rather than used as this is a customers machine and not mine. I saw an Adaptec HBA 1000 8i that would work physically with my setup, but am unsure of the freenas side of things. Do they play nice?

 

[Jeremy Hays]

Ok so got the RMA'd LSI 9207 card back and all is well again.

Disappointed that it too 2 rma's to get a good card though.