SAS2008 controller timeouts whenever performance is pushed (Sending abort to target X)

braiam

braiam

Dabbler
Joined
Nov 28, 2021
Messages
10
Everytime I try to push my LSI SAS2008 card for performance, it starts timing out. I already flashed the controller to v20.0.07.00 in IT mode. These are the kind of messages I'm seeing:

Code:
(da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 c2 cf 97 a0 00 00 00 08 00 00 length 4096 SMID 1656 Command timeout on target 2(0x000a) 90000 set, 90.135617296 elapsed
mps0: Sending abort to target 2 for SMID 1656
    (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 c2 cf 97 a0 00 00 00 08 00 00 length 4096 SMID 1656 Aborting command 0xfffffe00c511b140
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1440 Command timeout on target 2(0x000a) 90000 set, 90.207423994 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1886 Command timeout on target 3(0x0009) 90000 set, 90.269591921 elapsed
mps0: Sending abort to target 3 for SMID 1886
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1886 Aborting command 0xfffffe00c512e650
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e ff d0 00 00 30 00 length 24576 SMID 1480 Command timeout on target 4(0x000c) 90000 set, 90.398612076 elapsed
mps0: Sending abort to target 4 for SMID 1480
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e ff d0 00 00 30 00 length 24576 SMID 1480 Aborting command 0xfffffe00c510c4c0
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e fe d0 00 01 00 00 length 131072 SMID 1435 Command timeout on target 4(0x000c) 90000 set, 90.526706772 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e fd d0 00 01 00 00 length 131072 SMID 126 Command timeout on target 4(0x000c) 90000 set, 90.587853050 elapsed
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 f0 44 9f 00 00 01 00 length 512 SMID 1140 Command timeout on target 0(0x000b) 90000 set, 90.121922308 elapsed
mps0: Sending abort to target 0 for SMID 1140
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 f0 44 9f 00 00 01 00 length 512 SMID 1140 Aborting command 0xfffffe00c50efbe0
    (xpt0:mps0:0:2:0): SMID 13 task mgmt 0xfffffe00c5091178 timed out
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>


Code:
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 2083 Command timeout on target 3(0x0009) 90000 set, 90.246071505 elapsed
mps0: Sending abort to target 3 for SMID 2083
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 2083 Aborting command 0xfffffe00c513ef08
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 750 Command timeout on target 3(0x0009) 90000 set, 90.374417806 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 aa 00 00 01 00 00 length 131072 SMID 1523 Command timeout on target 3(0x0009) 90000 set, 90.435550608 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a9 00 00 01 00 00 length 131072 SMID 1189 Command timeout on target 3(0x0009) 90000 set, 90.496875977 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 01 00 00 length 131072 SMID 1548 Command timeout on target 3(0x0009) 90000 set, 90.558443568 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 177 Command timeout on target 3(0x0009) 90000 set, 90.620658793 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 941 Command timeout on target 3(0x0009) 90000 set, 90.681728418 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a5 00 00 01 00 00 length 131072 SMID 1951 Command timeout on target 3(0x0009) 90000 set, 90.742576826 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a4 00 00 01 00 00 length 131072 SMID 98 Command timeout on target 3(0x0009) 90000 set, 90.804019079 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a3 00 00 01 00 00 length 131072 SMID 1820 Command timeout on target 3(0x0009) 90000 set, 90.864894103 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 324 Command timeout on target 2(0x000a) 90000 set, 90.926398938 elapsed
mps0: Sending abort to target 2 for SMID 324
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 324 Aborting command 0xfffffe00c50ab360
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 aa 00 00 01 00 00 length 131072 SMID 1889 Command timeout on target 2(0x000a) 90000 set, 90.1053828598 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a2 00 00 01 00 00 length 131072 SMID 1600 Command timeout on target 3(0x0009) 90000 set, 90.1115597697 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a1 00 00 01 00 00 length 131072 SMID 150 Command timeout on target 3(0x0009) 90000 set, 90.1177655506 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a0 00 00 01 00 00 length 131072 SMID 1563 Command timeout on target 3(0x0009) 90000 set, 90.1239169550 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 aa 80 00 01 00 00 length 131072 SMID 1842 Command timeout on target 4(0x000c) 90000 set, 90.1301496434 elapsed
mps0: Sending abort to target 4 for SMID 1842
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 aa 80 00 01 00 00 length 131072 SMID 1842 Aborting command 0xfffffe00c512ab30
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a9 80 00 01 00 00 length 131072 SMID 1736 Command timeout on target 4(0x000c) 90000 set, 90.1430829791 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a8 80 00 01 00 00 length 131072 SMID 1748 Command timeout on target 4(0x000c) 90000 set, 90.1492591283 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a9 00 00 01 00 00 length 131072 SMID 976 Command timeout on target 2(0x000a) 90000 set, 90.1555067219 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 01 00 00 length 131072 SMID 1199 Command timeout on target 2(0x000a) 90000 set, 90.1616485481 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 2007 Command timeout on target 2(0x000a) 90000 set, 90.1678256014 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 1793 Command timeout on target 2(0x000a) 90000 set, 90.1740030896 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 00 80 00 length 65536 SMID 103 Command timeout on target 4(0x000c) 90000 set, 90.1802795652 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 236 Command timeout on target 4(0x000c) 90000 set, 90.1863678186 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 130 Command timeout on target 4(0x000c) 90000 set, 90.1925108355 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a5 00 00 01 00 00 length 131072 SMID 1569 Command timeout on target 4(0x000c) 90000 set, 90.1986720831 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a4 00 00 01 00 00 length 131072 SMID 1571 Command timeout on target 4(0x000c) 90000 set, 90.2048685759 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9f 80 00 00 80 00 length 65536 SMID 146 Command timeout on target 3(0x0009) 90000 set, 90.2114220373 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9e 80 00 01 00 00 length 131072 SMID 192 Command timeout on target 3(0x0009) 90000 set, 90.-2119645584 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9e 00 00 00 80 00 length 65536 SMID 1309 Command timeout on target 3(0x0009) 90000 set, 90.-2057277883 elapsed
    (da4:mps0:0:4:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 2075 Command timeout on target 4(0x000c) 90000 set, 90.-2015517865 elapsed
    (da2:mps0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 1559 Command timeout on target 2(0x000a) 90000 set, 90.-1950535916 elapsed
    (da3:mps0:0:3:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 1141 Command timeout on target 3(0x0009) 90000 set, 90.-1885794373 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ad 80 00 00 80 00 length 65536 SMID 2098 Command timeout on target 4(0x000c) 90000 set, 90.-1810982818 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ac 80 00 01 00 00 length 131072 SMID 1712 Command timeout on target 4(0x000c) 90000 set, 90.-1749184747 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ab 80 00 01 00 00 length 131072 SMID 1784 Command timeout on target 4(0x000c) 90000 set, 90.-1687088677 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ad 00 00 01 00 00 length 131072 SMID 1879 Command timeout on target 2(0x000a) 90000 set, 90.-1624601350 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 1917 Command timeout on target 2(0x000a) 90000 set, 90.-1562668760 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 f3 98 3a 60 00 00 40 00 length 32768 SMID 147 Command timeout on target 3(0x0009) 90000 set, 90.-1523446109 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 f3 98 3a 58 00 00 48 00 length 36864 SMID 1851 Command timeout on target 2(0x000a) 90000 set, 90.-1461970659 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 f3 98 3a 58 00 00 48 00 length 36864 SMID 1530 Command timeout on target 4(0x000c) 90000 set, 90.-1399871977 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 b3 b0 00 00 80 00 length 65536 SMID 1279 Command timeout on target 4(0x000c) 90000 set, 90.-1364945410 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 b3 b0 00 00 80 00 length 65536 SMID 1785 Command timeout on target 2(0x000a) 90000 set, 90.-1302906753 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 b2 78 00 00 80 00 length 65536 SMID 300 Command timeout on target 2(0x000a) 90000 set, 90.-1240767054 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 b2 78 00 00 80 00 length 65536 SMID 1832 Command timeout on target 3(0x0009) 90000 set, 90.-1179232102 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 b2 70 00 00 80 00 length 65536 SMID 1868 Command timeout on target 4(0x000c) 90000 set, 90.-1117157298 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1816 Command timeout on target 4(0x000c) 90000 set, 90.-1055225799 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1316 Command timeout on target 3(0x0009) 90000 set, 90.-993350472 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1654 Command timeout on target 2(0x000a) 90000 set, 90.-931994975 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1639 Command timeout on target 4(0x000c) 90000 set, 90.-869895079 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1171 Command timeout on target 3(0x0009) 90000 set, 90.-808341213 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1438 Command timeout on target 2(0x000a) 90000 set, 90.-746927912 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 1983 Command timeout on target 4(0x000c) 90000 set, 90.-685211235 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 331 Command timeout on target 3(0x0009) 90000 set, 90.-623481053 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 1829 Command timeout on target 2(0x000a) 90000 set, 90.-562415021 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 2092 Command timeout on target 4(0x000c) 90000 set, 90.-501021227 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 1459 Command timeout on target 3(0x0009) 90000 set, 90.-439434170 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 1819 Command timeout on target 2(0x000a) 90000 set, 90.-377839029 elapsed
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 da f2 8a 00 00 10 00 length 8192 SMID 1437 Command timeout on target 0(0x000b) 90000 set, 90.-1050544238 elapsed
mps0: Sending abort to target 0 for SMID 1437
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 da f2 8a 00 00 10 00 length 8192 SMID 1437 Aborting command 0xfffffe00c5108af8
mps0: Calling Reinit from mps_wait_command, timeout=30, elapsed=30
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: Reinit success
mps0: mps_user_pass_thru: invalid request: error 60
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1618 Command timeout on target 4(0x000a) 90000 set, 90.238810847 elapsed
mps0: Sending abort to target 4 for SMID 1618
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1618 Aborting command 0xfffffe00c5117e30
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 975 Command timeout on target 4(0x000a) 90000 set, 90.366451062 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1182 Command timeout on target 4(0x000a) 90000 set, 90.427712162 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 2061 Command timeout on target 4(0x000a) 90000 set, 90.489198138 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 674 Command timeout on target 4(0x000a) 90000 set, 90.550512085 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 1597 Command timeout on target 4(0x000a) 90000 set, 90.611718379 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1190 Command timeout on target 4(0x000a) 90000 set, 90.673476846 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1805 Command timeout on target 4(0x000a) 90000 set, 90.735009195 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1125 Command timeout on target 2(0x000b) 90000 set, 90.797470002 elapsed
mps0: Sending abort to target 2 for SMID 1125
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1125 Aborting command 0xfffffe00c50ee7b8
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 1149 Command timeout on target 2(0x000b) 90000 set, 90.925468019 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1238 Command timeout on target 2(0x000b) 90000 set, 90.986876845 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 1469 Command timeout on target 2(0x000b) 90000 set, 90.1048365154 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 1088 Command timeout on target 2(0x000b) 90000 set, 90.1110251467 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 191 Command timeout on target 2(0x000b) 90000 set, 90.1172172441 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1053 Command timeout on target 2(0x000b) 90000 set, 90.1233532050 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1586 Command timeout on target 2(0x000b) 90000 set, 90.1295478910 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1047 Command timeout on target 3(0x0009) 90000 set, 90.1358649757 elapsed
mps0: Sending abort to target 3 for SMID 1047
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1047 Aborting command 0xfffffe00c50e7ee8
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1339 Command timeout on target 3(0x0009) 90000 set, 90.1488324637 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 2068 Command timeout on target 3(0x0009) 90000 set, 90.1549818199 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1341 Command timeout on target 3(0x0009) 90000 set, 90.1611778453 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 278 Command timeout on target 3(0x0009) 90000 set, 90.1673807672 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 1335 Command timeout on target 3(0x0009) 90000 set, 90.1735217121 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 1674 Command timeout on target 4(0x000a) 90000 set, 90.1797431970 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 88 Command timeout on target 4(0x000a) 90000 set, 90.1859284314 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6e 80 00 01 00 00 length 131072 SMID 1566 Command timeout on target 4(0x000a) 90000 set, 90.1920340995 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 1096 Command timeout on target 3(0x0009) 90000 set, 90.1982289706 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1684 Command timeout on target 3(0x0009) 90000 set, 90.2044322129 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1703 Command timeout on target 3(0x0009) 90000 set, 90.2106329308 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 116 Command timeout on target 2(0x000b) 90000 set, 90.-2125949142 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 1533 Command timeout on target 2(0x000b) 90000 set, 90.-2064194289 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 6e 80 00 01 00 00 length 131072 SMID 1020 Command timeout on target 2(0x000b) 90000 set, 90.-2001672444 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 1791 Command timeout on target 3(0x0009) 90000 set, 90.-1939288718 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 1016 Command timeout on target 3(0x0009) 90000 set, 90.-1877002065 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 66 80 00 00 80 00 length 65536 SMID 375 Command timeout on target 4(0x000a) 90000 set, 90.-1810509498 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6a 80 00 00 80 00 length 65536 SMID 1245 Command timeout on target 4(0x000a) 90000 set, 90.-1748772634 elapsed
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 e1 2c c4 00 00 08 00 length 4096 SMID 1078 Command timeout on target 0(0x000c) 90000 set, 90.-1683978868 elapsed
mps0: Sending abort to target 0 for SMID 1078
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 e1 2c c4 00 00 08 00 length 4096 SMID 1078 Aborting command 0xfffffe00c50ea890
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 672 Command timeout on target 4(0x000a) 90000 set, 90.-1572716408 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 2019 Command timeout on target 4(0x000a) 90000 set, 90.-1510530563 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 92 Command timeout on target 4(0x000a) 90000 set, 90.-1448234527 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1187 Command timeout on target 4(0x000a) 90000 set, 90.-1386584359 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 1909 Command timeout on target 2(0x000b) 90000 set, 90.-1323044348 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 1673 Command timeout on target 2(0x000b) 90000 set, 90.-1260991057 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 163 Command timeout on target 2(0x000b) 90000 set, 90.-1198906067 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1079 Command timeout on target 2(0x000b) 90000 set, 90.-1136901177 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 1350 Command timeout on target 3(0x0009) 90000 set, 90.-1074021622 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 1812 Command timeout on target 3(0x0009) 90000 set, 90.-1011637399 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 1423 Command timeout on target 3(0x0009) 90000 set, 90.-949366878 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 1751 Command timeout on target 4(0x000a) 90000 set, 90.-1905680586 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 1711 Command timeout on target 2(0x000b) 90000 set, 90.-1843789859 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 961 Command timeout on target 3(0x0009) 90000 set, 90.-1782010647 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 1235 Command timeout on target 4(0x000a) 90000 set, 90.-1720243415 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 63 Command timeout on target 2(0x000b) 90000 set, 90.-1658334681 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 66 Command timeout on target 3(0x0009) 90000 set, 90.-1597355036 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8f 80 00 00 80 00 length 65536 SMID 1353 Command timeout on target 2(0x000b) 90000 set, 90.-1537330062 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 8f 00 00 00 80 00 length 65536 SMID 71 Command timeout on target 3(0x0009) 90000 set, 90.-1475278184 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8f 00 00 00 80 00 length 65536 SMID 326 Command timeout on target 2(0x000b) 90000 set, 90.-1414218269 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 779 Command timeout on target 4(0x000a) 90000 set, 90.-1352373423 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1541 Command timeout on target 4(0x000a) 90000 set, 90.-1290934493 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1236 Command timeout on target 2(0x000b) 90000 set, 90.-1228937511 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 1961 Command timeout on target 3(0x0009) 90000 set, 90.-1166956931 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 1768 Command timeout on target 2(0x000b) 90000 set, 90.-1105114076 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1464 Command timeout on target 3(0x0009) 90000 set, 90.-1043317587 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 1830 Command timeout on target 4(0x000a) 90000 set, 90.-980900407 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 1650 Command timeout on target 2(0x000b) 90000 set, 90.-919421545 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 733 Command timeout on target 3(0x0009) 90000 set, 90.-857973306 elapsed
    (xpt0:mps0:0:4:0): SMID 1 task mgmt 0xfffffe00c5090158 timed out
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
    (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 c6 fc 06 58 00 00 00 08 00 00 length 4096 SMID 1698 Command timeout on target 4(0x000a) 90000 set, 90.7424792 elapsed
mps0: Sending abort to target 4 for SMID 1698
    (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 c6 fc 06 58 00 00 00 08 00 00 length 4096 SMID 1698 Aborting command 0xfffffe00c511e9b0
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 eb a2 88 00 01 00 00 length 131072 SMID 2100 Command timeout on target 0(0x000b) 90000 set, 90.66333841 elapsed
mps0: Sending abort to target 0 for SMID 2100
    (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 eb a2 88 00 01 00 00 length 131072 SMID 2100 Aborting command 0xfffffe00c51405e0
    (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 08 00 00 00 80 00 00 length 65536 SMID 387 Command timeout on target 4(0x000a) 90000 set, 90.212281186 elapsed
    (da3:mps0:0:3:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 10 00 00 00 80 00 00 length 65536 SMID 728 Command timeout on target 3(0x0009) 90000 set, 90.279061399 elapsed
mps0: Sending abort to target 3 for SMID 728
    (da3:mps0:0:3:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 10 00 00 00 80 00 00 length 65536 SMID 728 Aborting command 0xfffffe00c50cd240
    (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 587 Command timeout on target 2(0x000c) 90000 set, 90.419441905 elapsed
mps0: Sending abort to target 2 for SMID 587
    (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 587 Aborting command 0xfffffe00c50c14c8
    (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 1214 Command timeout on target 4(0x000a) 90000 set, 90.559639039 elapsed
    (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 26 46 d8 b0 00 00 00 80 00 00 length 65536 SMID 390 Command timeout on target 4(0x000a) 90000 set, 90.116655936 elapsed
    (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 26 46 d8 b8 00 00 00 80 00 00 length 65536 SMID 485 Command timeout on target 2(0x000c) 90000 set, 90.183680379 elapsed
    (pass3:mps0:0:3:0): ATA COMMAND PASS THROUGH(16). CDB: 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 length 0 SMID 763 Command timeout on target 3(0x0009) 60000 set, 60.110738282 elapsed
    (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 87 2b 9b 68 00 00 08 00 length 4096 SMID 1161 Command timeout on target 2(0x000c) 90000 set, 90.286155133 elapsed
    (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 87 2b 9b 60 00 00 08 00 length 4096 SMID 522 Command timeout on target 3(0x0009) 90000 set, 90.346322166 elapsed
    (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 87 2b 9b 60 00 00 08 00 length 4096 SMID 493 Command timeout on target 4(0x000a) 90000 set, 90.406735410 elapsed
    (xpt0:mps0:0:4:0): SMID 4 task mgmt 0xfffffe00c5090560 timed out
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: Calling Reinit from mps_wait_command, timeout=60, elapsed=60
mps0: Reinitializing controller
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: Reinit success
mps_config_get_bios_pg3: request for header completed with error 60
mps_user_get_adapter_data: Error while retrieving BIOS Version


Usually, it recovers, but it leaves my applications hanging for IO. I've seen similar issues here https://www.truenas.com/community/threads/timeout-errors-when-mving-files-to-truenas-server.89820/ but nothing that can point to a specific issue that can be investigated.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
How hot is the card?

What are the attached disks? (if they are SSDs, maybe consider a 3008 series replacement card... usually $50 on Ebay).
 
braiam

braiam

Dabbler
Joined
Nov 28, 2021
Messages
10
Sadly, the card doesn't seem to have onboard temp monitoring (at least not where I looked). This is an array of RAIDZ1 with 3 HDD and a single SSD for read cache.
 
braiam

braiam

Dabbler
Joined
Nov 28, 2021
Messages
10
Forgot to mention, this is a TYAN S5512 platform, so replacing it means not using the integrated controller at all.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Try removing the L2ARC.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
So it's a proprietary form factor?

In any case, it's probably a good idea to replace the thermal paste on these things, given their age. They run hot and don't deal well with the heat, and since you'd hope that Tyan did their homework, the paste seems like low/hanging fruit.
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Ericloewe said:
So it's a proprietary form factor?
Click to expand...
Embedded onto the motherboard.

@braiam what case is this motherboard in, and does it have a good amount of airflow over the SAS chipset's heatsink?

A "touch test" or IR spot thermometer may also be misleading, as if the thermal interface material has degraded as @Ericloewe mentioned, then the chip will be hot but the heatsink won't be.
 
braiam

braiam

Dabbler
Joined
Nov 28, 2021
Messages
10
I reapplied the thermal compound to an old one that this was replacing. It seems to have not produced the same problem as before and apparently didn't need reflashing to behave that way. I flashed it anyways, just to touch on wood, but it has behaved as expected with the workloads I've thrown at.
 
You must log in or register to reply here.

Similar threads

D0peX
Upgrading to 13 results in controller reinitialization
Replies
7
Views
819
mardel
M
beralt
Problem with LSI HBA after upgrade from FreeNAS 11.3U2 to TrueNAS 12.0
Replies
7
Views
2K
beralt
beralt
K
Timeout errors when mving files to TrueNAS Server
Replies
9
Views
2K
Vlad_Chudo
V
K
SAS array is significantly slower than SATA on same hardware
Replies
2
Views
751
kristin
K
A
Noticed these errors in the log, is this disk related?
Replies
3
Views
1K
sretalla
sretalla
Top