TrueNAS 12.0-U1 LSI Sas Controller timeouts

N

n0one

Cadet
Joined
Jul 13, 2018
Messages
4
Hey, having a lot of issues with latest release and this controller.

worked ok for few days, then got error msg in ui about 1 disk having SMART issues.
when copying from disks i got timeouts and controller resets from all 8 drives.

reboot back to 12.0-release fixed all issues, so seems its a issue only in -U1 for now

Some of the errors i got in U1 :

error reading device 0x11 sata passthru; iocstatus= 0x47
freenas mps0: reinitializing controller
smid 8 task mgmt 0xfffffe00fdc86ac0 timed out


LSI SAS 9210-8i 8-port 6Gb/s PCIe RAID SATA Controller card, IT mode flash in a
SuperMicro X9DRi-F CSE-846 24x3.5" Storage Server


Driver from 12.0 Release

mps0: <Avago Technologies (LSI) SAS2008> port 0x7000-0x70ff mem 0xdf600000-0xdf6
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
 
K

kspare

Guru
Joined
Feb 19, 2015
Messages
508
I noticed this same error this morning when I was booting a server up and watching the console.....
 
Last edited:
cobrakiller58

cobrakiller58

Guru
Joined
Jan 18, 2017
Messages
525
one of you should post a bug report on Jira about this with the debug file so the Devs can look into it.
 
K

kspare

Guru
Joined
Feb 19, 2015
Messages
508
im still trying to confirm it. I'm seeing errors about "FW is in FAULT State" so I may not be seeing the same thing. but my performance in U1 has dropped.
 
1

17sme

Cadet
Joined
Feb 21, 2021
Messages
2
I experience a similar issue with the latest TrueNAS release. I've been trying to fix the issue for the last couple of weeks, searching for similar issues - and trying the solutions provided, but the errors remain.

HBA info:
Code:
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

        Controller Number              : 0
        Controller                     : SAS2308_2(D1)
        PCI Address                    : 00:09:00:00
        SAS Address                    : 500605b-0-0584-7240
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-4i4e
        BIOS Version                   : 07.27.00.00
        UEFI BSD Version               : 07.23.01.00
        FCODE Version                  : N/A
        Board Name                     : SAS9207-4i4e
        Board Assembly                 : H3-25434-00F
        Board Tracer Number            : SV24300158

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

The firmware has been upgraded to the most recent version I could find.

After a few minutes of uptime, the log is flooded with this:
Code:
(da0:mps0:0:33:0): READ(10). CDB: 28 00 22 42 be 30 00 01 00 00 
(da0:mps0:0:33:0): CAM status: SCSI Status Error
(da0:mps0:0:33:0): SCSI status: Check Condition
(da0:mps0:0:33:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da0:mps0:0:33:0): Info: 0x2242be30
(da0:mps0:0:33:0): Error 5, Unretryable error
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: changing name to 'vnet0.3'
epair2b: changing name to 'epair0b'
lo0: link state changed to UP
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 93 b0 00 00 c0 00 
(da0:mps0:0:33:0): CAM status: SCSI Status Error
(da0:mps0:0:33:0): SCSI status: Check Condition
(da0:mps0:0:33:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da0:mps0:0:33:0): Info: 0x1e6793d8
(da0:mps0:0:33:0): Error 5, Unretryable error
mps0: Controller reported scsi ioc terminated tgt 33 SMID 2083 loginfo 31080000
mps0: (da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 75 20 00 01 00 00 
Controller reported scsi ioc terminated tgt 33 SMID 550 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 360 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 220 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 184 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 315 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 2172 loginfo 31080000
mps0: (da0:mps0:0:33:0): CAM status: CCB request completed with an error
Controller reported scsi ioc terminated tgt 33 SMID 565 loginfo 31080000
mps0: (da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 60 60 00 00 a0 00 
Controller reported scsi ioc terminated tgt 33 SMID 501 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 494 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 2155 loginfo 31080000
mps0: Controller reported scsi ioc terminated tgt 33 SMID 353 loginfo 31080000
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 76 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 77 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 78 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 79 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 7a 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 5d a0 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 5e a0 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 5f a0 00 00 c0 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 7b 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 7c 20 00 01 00 00 
(da0:mps0:0:33:0): CAM status: CCB request completed with an error
(da0:mps0:0:33:0): Retrying command, 3 more tries remain
(da0:mps0:0:33:0): READ(10). CDB: 28 00 1e 67 5c a0 00 01 00 00 
(da0:mps0:0:33:0): CAM status: SCSI Status Error
(da0:mps0:0:33:0): SCSI status: Check Condition
(da0:mps0:0:33:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da0:mps0:0:33:0): Info: 0x1e675d30
(da0:mps0:0:33:0): Error 5, Unretryable error
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
(da0:mps0:0:33:0): Invalidating pack
da0 at mps0 bus 0 scbus0 target 33 lun 0
da0: <ATA WDC> detached
GEOM_MIRROR: Device swap0: provider da0p1 disconnected.
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
(da0:mps0:0:33:0): Periph destroyed
da0 at mps0 bus 0 scbus0 target 33 lun 0
da0: <ATA WDC> Fixed Direct Access SPC-4 SCSI device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 5723166MB (11721045168 512 byte sectors)
da0: quirks=0x8<4K>
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
(da6:mps0:0:52:0): Invalidating pack
da6 at mps0 bus 0 scbus0 target 52 lun 0
da6: <ATA> detached
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
Solaris: WARNING: mps0: Reinitializing controller
Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xd SATA PASSTHRU; iocstatus= 0x47


The last section
Code:
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x47
mps0: Error reading device 0xc SATA PASSTHRU; iocstatus= 0x804b
mps0: Sleeping 3 seconds after SATA ID error to wait for spinup
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: Error reading device 0xb SATA PASSTHRU; iocstatus= 0x47
mps0: IOC Fault 0x40000d04, Resetting
mps0: Reinitializing controller

is on repeat until the system hangs itself or until I reset/powercycle the machine.
 
1

17sme

Cadet
Joined
Feb 21, 2021
Messages
2
On a side note: Booting the installer (12.0U1) IIRC from USB allows me to resilver and scrub the pools without any errors from the controller, or the disks.
 
You must log in or register to reply here.

Similar threads

Z
Constant Errors from LSI HBA. IOC Fault.
Replies
2
Views
1K
zbrown90
Z
L
Help Diagnosing Hardware Errors
Replies
2
Views
464
lostinak
L
blckhm
System hangs at step: "mpr0: Reinitializing controller" during system boot on 12.0-U1
Replies
0
Views
776
blckhm
blckhm
Mystik
Problems with LSI SAS 9207-8i after 12.0-U8 -> 13.0-U1.1 upgrade
Replies
8
Views
2K
mardel
M
beralt
Problem with LSI HBA after upgrade from FreeNAS 11.3U2 to TrueNAS 12.0
Replies
7
Views
2K
beralt
beralt
Top