kernel: mptscsih: ioc0: task abort: FAILED

[g1ngerninja]

Hi All,

I've been testing my server (Supermicro X8DTE-F Dual Socket with IPMI motherboard) running virtual box running a guest Linux Fedora 22 distro for a while now with no issues. I recently swapped my LSI 9266-4i RAID for a LSI 9211-8i in IT mode as per the recommendations on here. But now for some unknown reason my VM guest is exhibiting the above errors! So much so that it remounts the drive as read only when the drive is being written to a lot. I'm running Oracle 12c on it.

Jul 24 18:06:29 dbciFID kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807d9fbca80)
Jul 24 18:06:32 dbciFID kernel: sd 2:0:2:0: [sdc] tag#2 CDB: Read(10) 28 00 00 f8 c2 90 00 00 10 00
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807d9fbca80)
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807d9fbc180)
Jul 24 18:06:32 dbciFID kernel: sd 2:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 08 6c 3b 20 00 00 08 00
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807d9fbc180)
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8807d9fbc180)
Jul 24 18:06:32 dbciFID kernel: sd 2:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 08 6c 3b 20 00 00 08 00
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8807d9fbc180)
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8807d9fbca80)
Jul 24 18:06:32 dbciFID kernel: sd 2:0:2:0: [sdc] tag#2 CDB: Read(10) 28 00 00 f8 c2 90 00 00 10 00
Jul 24 18:06:32 dbciFID kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8807d9fbca80)

I've found lots of mention of the above error but from really old kernels.

Any ideas what might be causing this and how I can fix it without resorting to putting my 9266-4i back in?
I should probably mention that the drives mounted to the VM are iSCSI targets, but these have remained the same. I should also point out that changing the controller from SAS LSI to SCSI LSI results in the same problem, presumably cause it uses the same driver. If I switch to using the SATA controller I have no problems.

Also I have had to disable the boot support on the 9211 as it wouldn't boot anything from the USB ports without me pressing the control c option to get into the configuration utility of the card, but it never reaches the config tool and then proceeds to boot the USB thumb drive! Very strange.

Thanks

 

[g1ngerninja]

OK so I have updated the distro to Fedora 24 but same errors persist!

 

[dlavigne]

Installed software won't make a difference as it looks like a hardware or driver issue.

Were you able to resolve it?

 

[g1ngerninja]

I've not touched it for a while, but no I couldn't resolve. Will look at it again when I have more time.

 

[g1ngerninja]

OK so I finally got time to check this again. Same issue happening. But I don't see how it's a hardware issue as the underlying OS in this case FreeNAS is perfectly fine, no errors exhibited. And as I mentioned if I switch the controller in the VM from SAS to SATA no issues exist!

Not sure what might be causing this.

 

[Mirfster]

Not to sound rude, but what does any of this have to do with FreeNAS?

 

[g1ngerninja]

Not to sound rude, but what does any of this have to do with FreeNAS?

You are correct cause I'm a muppet and mentioned the wrong one, too many different things going with different OS's. My apologies, I've updated the post accordingly.

 

[g1ngerninja]

Oh I came across this post http://unix.stackexchange.com/quest...ut-be-disabled-in-linux-attempting-task-abort and so I tried it. Unfortunately that setting is 24hrs and my VM was stuck in an oracle process and had to be restarted. I've changed it now to 120secs from the default 30 and now I've only seen 1 instance of the error thus far. Not sure its a solution but something weird is going on.

 

[g1ngerninja]

J

Oh I came across this post http://unix.stackexchange.com/quest...ut-be-disabled-in-linux-attempting-task-abort and so I tried it. Unfortunately that setting is 24hrs and my VM was stuck in an oracle process and had to be restarted. I've changed it now to 120secs from the default 30 and now I've only seen 1 instance of the error thus far. Not sure its a solution but something weird is going on.

Just when I thought everything was working I get this :-


Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807214a5980)
Sep 04 19:11:15 dbABC kernel: sd 2:0:2:0: [sdc] tag#48 CDB: Write(10) 2a 00 06 8a 54 d0 00 00 10 00
Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807214a5980)
Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff880721485680)
Sep 04 19:11:15 dbABC kernel: sd 2:0:2:0: [sdc] tag#35 CDB: Write(10) 2a 00 06 50 f1 60 00 00 10 00
Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff880721485680)
Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8807214a5980)
Sep 04 19:11:15 dbABC kernel: sd 2:0:2:0: [sdc] tag#48 CDB: Write(10) 2a 00 06 8a 54 d0 00 00 10 00
Sep 04 19:11:15 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8807214a5980)
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88059a201080)
Sep 04 19:14:11 dbABC kernel: sd 2:0:2:0: [sdc] tag#29 CDB: Write(10) 2a 00 01 0a 0e 00 00 02 00 00
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff88059a201080)
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88059a200180)
Sep 04 19:14:11 dbABC kernel: sd 2:0:2:0: [sdc] tag#15 CDB: Write(10) 2a 00 06 50 ac 00 00 00 10 00
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff88059a200180)
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff88059a201080)
Sep 04 19:14:11 dbABC kernel: sd 2:0:2:0: [sdc] tag#29 CDB: Write(10) 2a 00 01 0a 0e 00 00 02 00 00
Sep 04 19:14:11 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff88059a201080)
Sep 04 19:16:16 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8805b4359e00)
Sep 04 19:16:16 dbABC kernel: sd 2:0:2:0: [sdc] tag#41 CDB: Write(10) 2a 00 05 84 54 30 00 00 10 00
Sep 04 19:16:16 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8805b4359e00)
Sep 04 19:16:16 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8805b4359e00)
Sep 04 19:16:16 dbABC kernel: sd 2:0:2:0: [sdc] tag#41 CDB: Write(10) 2a 00 05 84 54 30 00 00 10 00
Sep 04 19:16:16 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8805b4359e00)
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8805abbc1e00)
Sep 04 19:19:30 dbABC kernel: sd 2:0:2:0: [sdc] tag#28 CDB: Write(10) 2a 00 05 9b 28 60 00 00 10 00
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8805abbc1e00)
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8800d7a76d80)
Sep 04 19:19:30 dbABC kernel: sd 2:0:2:0: [sdc] tag#23 CDB: Write(10) 2a 00 05 84 6c 50 00 00 10 00
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8800d7a76d80)
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8805abbc1e00)
Sep 04 19:19:30 dbABC kernel: sd 2:0:2:0: [sdc] tag#28 CDB: Write(10) 2a 00 05 9b 28 60 00 00 10 00
Sep 04 19:19:30 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8805abbc1e00)
Sep 04 19:21:34 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807f95bd500)
Sep 04 19:22:06 dbABC kernel: sd 2:0:2:0: [sdc] tag#31 CDB: Write(10) 2a 00 05 dc 27 70 00 00 10 00
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807f95bd500)
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807fec48480)
Sep 04 19:22:06 dbABC kernel: sd 2:0:2:0: [sdc] tag#12 CDB: Write(10) 2a 00 05 08 28 e0 00 00 10 00
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807fec48480)
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8805c7805080)
Sep 04 19:22:06 dbABC kernel: sd 2:0:2:0: [sdc] tag#3 CDB: Read(10) 28 00 04 b2 f6 90 00 00 10 00
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8805c7805080)
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8807f95bd500)
Sep 04 19:22:06 dbABC kernel: sd 2:0:2:0: [sdc] tag#31 CDB: Write(10) 2a 00 05 dc 27 70 00 00 10 00
Sep 04 19:22:06 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8807f95bd500)
Sep 04 19:25:20 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8805c0c6f980)
Sep 04 19:25:20 dbABC kernel: sd 2:0:2:0: [sdc] tag#29 CDB: Write(10) 2a 00 04 bf 6f 70 00 00 10 00
Sep 04 19:25:20 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8805c0c6f980)
Sep 04 19:25:20 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8805c0c6f980)
Sep 04 19:25:20 dbABC kernel: sd 2:0:2:0: [sdc] tag#29 CDB: Write(10) 2a 00 04 bf 6f 70 00 00 10 00
Sep 04 19:25:20 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8805c0c6f980)
Sep 04 19:32:11 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8807fa95fb00)
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#5 CDB: Write(10) 2a 00 05 15 c4 d0 00 00 10 00
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8807fa95fb00)
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88059a200600)
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#19 CDB: Write(10) 2a 00 05 35 9f 10 00 00 10 00
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff88059a200600)
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8807fa95fb00)
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#5 CDB: Write(10) 2a 00 05 15 c4 d0 00 00 10 00
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8807fa95fb00)
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#28 Medium access timeout failure. Offlining disk!
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: attempting host reset! (sc=ffff8800d7a77b00)
Sep 04 19:32:26 dbABC kernel: mptscsih: ioc0: host reset: SUCCESS (sc=ffff8800d7a77b00)
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] tag#5 CDB: Write(10) 2a 00 05 15 c4 d0 00 00 10 00
Sep 04 19:32:26 dbABC kernel: blk_update_request: I/O error, dev sdc, sector 85312720
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] killing request
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] killing request
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] CDB: Write(10) 2a 00 05 5c 86 c0 00 00 10 00
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: [sdc] CDB: Write(10) 2a 00 05 35 9f 10 00 00 10 00
Sep 04 19:32:26 dbABC kernel: blk_update_request: I/O error, dev sdc, sector 89949888
Sep 04 19:32:26 dbABC kernel: blk_update_request: I/O error, dev sdc, sector 87400208
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: EXT4-fs warning (device sdc): ext4_end_bio:315: I/O error -5 writing to inode 7471465 (offset 0 size 0 starting block 15161984)
Sep 04 19:32:26 dbABC kernel: Buffer I/O error on device sdc, logical block 15161983
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: EXT4-fs warning (device sdc): ext4_end_bio:315: I/O error -5 writing to inode 7471466 (offset 0 size 0 starting block 29918089)
Sep 04 19:32:26 dbABC kernel: Buffer I/O error on device sdc, logical block 29918088
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: Aborting journal on device sdc-8.
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: JBD2: Error -5 detected when updating journal superblock for sdc-8.
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: EXT4-fs error (device sdc): ext4_journal_check_start:56: Detected aborted journal
Sep 04 19:32:26 dbABC kernel: EXT4-fs (sdc): Remounting filesystem read-only
Sep 04 19:32:26 dbABC kernel: EXT4-fs (sdc): previous I/O error to superblock detected
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device
Sep 04 19:32:26 dbABC kernel: sd 2:0:2:0: rejecting I/O to offline device


So I've given up, I've reverted to using the SATA controller for the database as that is solid as a rock.