System has started crashing repeatedly. Pls help with some troubleshooting.

[Goobz]

Hi guys,

Thanks in advance for any help you can give. I have been running my FreeNAS box quite successfully for the last 18 months or there abouts, with minor hiccups along the way, but nothing I haven't been able to figure out with a bit of googling. The crashes I have been experiencing on and off last few days has got me stumped though. As of this morning the system now won't stay up for more than 10 or 15 minutes without crashing. The system consists of a Supermicro X10SLH-F with a Xeon 1230v3 and 32GB of Supermicro certified Samsung ECC RAM. The storage was until last week 2 striped raidz2 vdevs of 6x3TB and 6x2TB drives. Over the last week I have one at a time replaced the 2TB drives with 3TB's to expand the storage which all seemed to go smoothly although maybe this was a catalyst for the crashes? The drives are attached to a M1015 (x8) card and the motherboard (x4) itself:

Anyway this is a sample of what comes up in the console footer just prior to a crash:

Code:

Dec 21 08:18:28 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 69 27 3b e8 00 00 08 00 length 4096 SMID 263 command timeout cm 0xfffffe0000b090f8 ccb 0xfffff80038203800
Dec 21 08:18:28 freenas     (noperiph:mps0:0:4294967295:0): SMID 1 Aborting command 0xfffffe0000b090f8
Dec 21 08:18:28 freenas mps0: Sending reset from mpssas_send_abort for target ID 6
Dec 21 08:18:28 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 69 27 4d 90 00 00 08 00 length 4096 SMID 324 command timeout cm 0xfffffe0000b0df20 ccb 0xfffff8000bb28800
Dec 21 08:18:28 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 69 27 5f 60 00 00 08 00 length 4096 SMID 484 command timeout cm 0xfffffe0000b1ac20 ccb 0xfffff800344aa000
Dec 21 08:18:28 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 69 26 6b 90 00 00 40 00 length 32768 SMID 229 command timeout cm 0xfffffe0000b06568 ccb 0xfffff80034537000
Dec 21 08:18:28 freenas     (noperiph:mps0:0:4294967295:0): SMID 2 Aborting command 0xfffffe0000b06568
Dec 21 08:18:28 freenas mps0: Sending reset from mpssas_send_abort for target ID 0
Dec 21 08:18:28 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 69 26 75 80 00 01 00 00 length 131072 SMID 473 command timeout cm 0xfffffe0000b19e08 ccb 0xfffff8021da6f000
Dec 21 08:18:28 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 69 26 76 80 00 01 00 00 length 131072 SMID 950 command timeout cm 0xfffffe0000b40130 ccb 0xfffff8021da6c800
Dec 21 08:18:28 freenas     (da3:mps0:0:3:0): READ(10). CDB: 28 00 69 26 78 c0 00 01 00 00 length 131072 SMID 483 command timeout cm 0xfffffe0000b1aad8 ccb 0xfffff80038999000
Dec 21 08:18:28 freenas     (noperiph:mps0:0:4294967295:0): SMID 3 Aborting command 0xfffffe0000b1aad8
Dec 21 08:18:28 freenas mps0: Sending reset from mpssas_send_abort for target ID 3
Dec 21 08:18:28 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 69 26 77 c0 00 01 00 00 length 131072 SMID 490 command timeout cm 0xfffffe0000b1b3d0 ccb 0xfffff80038221000
Dec 21 08:18:28 freenas     (noperiph:mps0:0:4294967295:0): SMID 4 Aborting command 0xfffffe0000b1b3d0
Dec 21 08:18:28 freenas mps0: Sending reset from mpssas_send_abort for target ID 7
Dec 21 08:18:28 freenas     (da3:mps0:0:3:0): READ(10). CDB: 28 00 69 26 79 c0 00 01 00 00 length 131072 SMID 455 command timeout cm 0xfffffe0000b186f8 ccb 0xfffff8000bbde000
Dec 21 08:18:28 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 69 26 78 c0 00 01 00 00 length 131072 SMID 344 command timeout cm 0xfffffe0000b0f8c0 ccb 0xfffff8003899b800
Dec 21 08:18:29 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ec 44 6a 48 00 00 18 00 length 12288 SMID 745 command timeout cm 0xfffffe0000b2fa88 ccb 0xfffff80034413000
Dec 21 08:18:29 freenas     (noperiph:mps0:0:4294967295:0): SMID 5 Aborting command 0xfffffe0000b2fa88
Dec 21 08:18:29 freenas mps0: Sending reset from mpssas_send_abort for target ID 8
Dec 21 08:18:29 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ec 44 6a 48 00 00 18 00 length 12288 SMID 446 command timeout cm 0xfffffe0000b17b70 ccb 0xfffff8003818c000
Dec 21 08:18:29 freenas     (noperiph:mps0:0:4294967295:0): SMID 6 Aborting command 0xfffffe0000b17b70
Dec 21 08:18:29 freenas mps0: Sending reset from mpssas_send_abort for target ID 9 

Trying to rule out the recent update I booted into the previous build from the bootloader but here's what I got after 13mins:

Code:

Dec 21 08:46:30 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 e0 40 fc 88 00 00 08 00 length 4096 SMID 262 command timeout cm 0xfffffe0000b0cfb0 ccb 0xfffff8003d9f3800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 1 Aborting command 0xfffffe0000b0cfb0
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 6
Dec 21 08:46:30 freenas     (da3:mps0:0:3:0): READ(10). CDB: 28 00 e0 40 fc 88 00 00 08 00 length 4096 SMID 213 command timeout cm 0xfffffe0000b090e8 ccb 0xfffff8003a587800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 2 Aborting command 0xfffffe0000b090e8
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 3
Dec 21 08:46:30 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 03 f6 b0 78 00 00 08 00 length 4096 SMID 200 command timeout cm 0xfffffe0000b08040 ccb 0xfffff8003a61c800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 3 Aborting command 0xfffffe0000b08040
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 7
Dec 21 08:46:30 freenas     (da7:mps0:0:9:0): READ(10). CDB: 28 00 03 ea d7 f8 00 00 08 00 length 4096 SMID 818 command timeout cm 0xfffffe0000b39810 ccb 0xfffff802f7654800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 4 Aborting command 0xfffffe0000b39810
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 9
Dec 21 08:46:30 freenas     (da1:mps0:0:1:0): READ(10). CDB: 28 00 03 f7 47 40 00 00 08 00 length 4096 SMID 612 command timeout cm 0xfffffe0000b29020 ccb 0xfffff80322156000
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 5 Aborting command 0xfffffe0000b29020
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 1
Dec 21 08:46:30 freenas     (da2:mps0:0:2:0): READ(10). CDB: 28 00 03 f7 6f b8 00 00 08 00 length 4096 SMID 988 command timeout cm 0xfffffe0000b471e0 ccb 0xfffff803871d4000
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 6 Aborting command 0xfffffe0000b471e0
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 2
Dec 21 08:46:30 freenas     (da6:mps0:0:8:0): READ(10). CDB: 28 00 03 ec 59 d0 00 00 08 00 length 4096 SMID 477 command timeout cm 0xfffffe0000b1e328 ccb 0xfffff8000ba36800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 7 Aborting command 0xfffffe0000b1e328
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 8
Dec 21 08:46:30 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 e0 40 fc 88 00 00 08 00 length 4096 SMID 89 command timeout cm 0xfffffe0000aff208 ccb 0xfffff8003d9ea800
Dec 21 08:46:30 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 03 f6 83 f0 00 01 00 00 length 131072 SMID 123 command timeout cm 0xfffffe0000b01d98 ccb 0xfffff8003a58e800
Dec 21 08:46:30 freenas     (noperiph:mps0:0:4294967295:0): SMID 8 Aborting command 0xfffffe0000b01d98
Dec 21 08:46:30 freenas mps0: Sending reset from mpssas_send_abort for target ID 0
Dec 21 08:46:30 freenas     (da1:mps0:0:1:0): READ(10). CDB: 28 00 03 f6 8f 60 00 00 c0 00 length 98304 SMID 763 command timeout cm 0xfffffe0000b35198 ccb 0xfffff802f7875800
Dec 21 08:46:30 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 03 f7 17 60 00 00 08 00 length 4096 SMID 794 command timeout cm 0xfffffe0000b37950 ccb 0xfffff802f7875000
Dec 21 08:46:30 freenas     (da6:mps0:0:8:0): READ(10). CDB: 28 00 03 ec 69 d8 00 00 08 00 length 4096 SMID 654 command timeout cm 0xfffffe0000b2c5f0 ccb 0xfffff80167c5a800
Dec 21 08:46:30 freenas     (da6:mps0:0:8:0): READ(10). CDB: 28 00 03 ec 9c 88 00 00 08 00 length 4096 SMID 875 command timeout cm 0xfffffe0000b3e118 ccb 0xfffff8003a4f3000
Dec 21 08:46:30 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 03 f7 c9 80 00 00 08 00 length 4096 SMID 226 command timeout cm 0xfffffe0000b0a190 ccb 0xfffff8003d1af000
Dec 21 08:46:30 freenas     (da7:mps0:0:9:0): READ(10). CDB: 28 00 03 ec 69 d8 00 00 08 00 length 4096 SMID 963 command timeout cm 0xfffffe0000b451d8 ccb 0xfffff8003a61c000
Dec 21 08:46:30 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 03 f6 8e a0 00 01 00 00 length 131072 SMID 907 command timeout cm 0xfffffe0000b40a18 ccb 0xfffff8003d9f0800
Dec 21 08:46:30 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 03 f7 17 60 00 00 08 00 length 4096 SMID 523 command timeout cm 0xfffffe0000b21e18 ccb 0xfffff8058a295800
Dec 21 08:46:30 freenas     (da7:mps0:0:9:0): READ(10). CDB: 28 00 03 ec 9c 88 00 00 08 00 length 4096 SMID 883 command timeout cm 0xfffffe0000b3eb58 ccb 0xfffff8058a296800
Dec 21 08:46:30 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 03 f8 03 a0 00 00 08 00 length 4096 SMID 771 command timeout cm 0xfffffe0000b35bd8 ccb 0xfffff8003a593800
Dec 21 08:46:30 freenas     (da5:mps0:0:7:0): READ(10). CDB: 28 00 03 f7 c9 80 00 00 08 00 length 4096 SMID 848 command timeout cm 0xfffffe0000b3be80 ccb 0xfffff8003a4fa000
Dec 21 08:46:30 freenas     (da1:mps0:0:1:0): READ(10). CDB: 28 00 03 f6 90 70 00 00 40 00 length 32768 SMID 574 command timeout cm 0xfffffe0000b25f70 ccb 0xfffff803221a9000
Dec 21 08:46:30 freenas     (da2:mps0:0:2:0): READ(10). CDB: 28 00 03 f6 91 f8 00 00 80 00 length 65536 SMID 460 command timeout cm 0xfffffe0000b1cd60 ccb 0xfffff8003a619800
Dec 21 08:46:30 freenas     (da3:mps0:0:3:0): READ(10). CDB: 28 00 03 f6 90 30 00 01 00 00 length 131072 SMID 96 command timeout cm 0xfffffe0000affb00 ccb 0xfffff8003d1b0800
Dec 21 08:46:30 freenas     (da4:mps0:0:6:0): READ(10). CDB: 28 00 03 f6 90 30 00 01 00 00 length 131072 SMID 350 command timeout cm 0xfffffe0000b14070 ccb 0xfffff8003a6b5800
Dec 21 08:46:30 freenas     (da2:mps0:0:2:0): READ(10). CDB: 28 00 03 f6 92 f8 00 00 40 00 length 32768 SMID 634 command timeout cm 0xfffffe0000b2ac50 ccb 0xfffff8000bbdf000
Dec 21 08:46:32 freenas     (da2:mps0:0:2:0): READ(10). CDB: 28 00 b2 f7 56 78 00 00 08 00 length 4096 SMID 498 command timeout cm 0xfffffe0000b1fe10 ccb 0xfffff802a2134800
Dec 21 08:46:32 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 b3 04 06 10 00 00 40 00 length 32768 SMID 981 command timeout cm 0xfffffe0000b468e8 ccb 0xfffff8003a2a4800
Dec 21 08:46:32 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 b3 04 06 18 00 00 40 00 length 32768 SMID 690 command timeout cm 0xfffffe0000b2f410 ccb 0xfffff80322155800
Dec 21 08:46:32 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 65 f0 00 00 40 00 length 32768 SMID 368 command timeout cm 0xfffffe0000b15780 ccb 0xfffff8000bbd5000
Dec 21 08:46:32 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 65 f0 00 00 40 00 length 32768 SMID 978 command timeout cm 0xfffffe0000b46510 ccb 0xfffff8003d0ef800
Dec 21 08:46:32 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 b3 04 06 18 00 00 40 00 length 32768 SMID 522 command timeout cm 0xfffffe0000b21cd0 ccb 0xfffff8000bbd0000
Dec 21 08:46:32 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 b3 04 06 18 00 00 40 00 length 32768 SMID 277 command timeout cm 0xfffffe0000b0e2e8 ccb 0xfffff8003a4ff000
Dec 21 08:46:32 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 b3 04 06 18 00 00 40 00 length 32768 SMID 184 command timeout cm 0xfffffe0000b06bc0 ccb 0xfffff802a21ef000
Dec 21 08:46:32 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 b3 04 06 18 00 00 40 00 length 32768 SMID 715 command timeout cm 0xfffffe0000b31418 ccb 0xfffff8003d9f0000
Dec 21 08:46:33 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 81 08 00 00 18 00
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): Retrying command
Dec 21 08:46:33 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 81 10 00 00 10 00
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): Retrying command
Dec 21 08:46:33 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 81 08 00 00 18 00
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:33 freenas (da7:mps0:0:9:0): Retrying command
Dec 21 08:46:33 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 81 10 00 00 10 00
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:33 freenas (da6:mps0:0:8:0): Retrying command
Dec 21 08:46:34 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 81 08 00 00 18 00
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): Retrying command
Dec 21 08:46:34 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 81 10 00 00 10 00
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): Retrying command
Dec 21 08:46:34 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 81 08 00 00 18 00
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:34 freenas (da7:mps0:0:9:0): Retrying command
Dec 21 08:46:34 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 81 10 00 00 10 00
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:34 freenas (da6:mps0:0:8:0): Retrying command
Dec 21 08:46:35 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 ea f2 81 08 00 00 18 00
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): Error 5, Retries exhausted
Dec 21 08:46:35 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 9
Dec 21 08:46:35 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): CAM status: CAM subsystem is busy
Dec 21 08:46:35 freenas (da7:mps0:0:9:0): Retrying command
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 ea f2 81 10 00 00 10 00
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): Error 5, Retries exhausted
Dec 21 08:46:35 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 8
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): CAM status: CAM subsystem is busy
Dec 21 08:46:35 freenas (da6:mps0:0:8:0): Retrying command 

Once these messages pop up the system freezes for a few minutes and then the IPMI console starts to spit out tons of random text so fast it's unreadable. I managed a quick screen grab though here :

Also from time to time the reboot fails halfway, the system locks up and tries again but usually comes good after that, until the next time. Lol.

It appears it is all the drives attached to the M1015 that are crashing(da0 - da7) so could this be the cause? Is there anything I can do to confirm this before I try to get another one? If you need any more info please don't hesitate to ask.

Thanks again for your assistance.

 

[Goobz]

Here's one more crash in case there's anything relevant:

Code:

Dec 21 09:09:53 freenas     (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 16 c0 00 00 01 00 00 00 length 131072 SMID 303 command timeout cm 0xfffffe0000b0c438 ccb 0xfffff801c2353000
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 1 Aborting command 0xfffffe0000b0c438
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 2
Dec 21 09:09:53 freenas     (da1:mps0:0:1:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 20 78 00 00 01 00 00 00 length 131072 SMID 953 command timeout cm 0xfffffe0000b40508 ccb 0xfffff801c2353800
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 2 Aborting command 0xfffffe0000b40508
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 1
Dec 21 09:09:53 freenas     (da5:mps0:0:7:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 25 80 00 00 00 40 00 00 length 32768 SMID 737 command timeout cm 0xfffffe0000b2f048 ccb 0xfffff8003e581000
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 3 Aborting command 0xfffffe0000b2f048
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 7
Dec 21 09:09:53 freenas     (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 17 c0 00 00 01 00 00 00 length 131072 SMID 95 command timeout cm 0xfffffe0000afb9b8 ccb 0xfffff8003e61d000
Dec 21 09:09:53 freenas     (da5:mps0:0:7:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 26 00 00 00 00 40 00 00 length 32768 SMID 897 command timeout cm 0xfffffe0000b3bd48 ccb 0xfffff800431a7000
Dec 21 09:09:53 freenas     (da3:mps0:0:3:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 26 80 00 00 00 40 00 00 length 32768 SMID 138 command timeout cm 0xfffffe0000aff0d0 ccb 0xfffff800431b4800
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 4 Aborting command 0xfffffe0000aff0d0
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 3
Dec 21 09:09:53 freenas     (da1:mps0:0:1:0): READ(16). CDB: 88 00 00 00 00 01 24 6c 21 78 00 00 01 00 00 00 length 131072 SMID 64 command timeout cm 0xfffffe0000af9200 ccb 0xfffff8003e60e000
Dec 21 09:09:53 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a 8b a8 00 00 18 00 length 12288 SMID 307 command timeout cm 0xfffffe0000b0c958 ccb 0xfffff801c2346800
Dec 21 09:09:53 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a 8b b0 00 00 18 00 length 12288 SMID 117 command timeout cm 0xfffffe0000afd5e8 ccb 0xfffff801c2035800
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 5 Aborting command 0xfffffe0000afd5e8
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 0
Dec 21 09:09:53 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a 8b b0 00 00 18 00 length 12288 SMID 769 command timeout cm 0xfffffe0000b31948 ccb 0xfffff801c2347000
Dec 21 09:09:53 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a 8b b0 00 00 10 00 length 8192 SMID 896 command timeout cm 0xfffffe0000b3bc00 ccb 0xfffff804b9173800
Dec 21 09:09:53 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a 8b b0 00 00 10 00 length 8192 SMID 928 command timeout cm 0xfffffe0000b3e500 ccb 0xfffff80043986000
Dec 21 09:09:53 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a 8b b0 00 00 10 00 length 8192 SMID 329 command timeout cm 0xfffffe0000b0e588 ccb 0xfffff801c2345000
Dec 21 09:09:53 freenas     (noperiph:mps0:0:4294967295:0): SMID 6 Aborting command 0xfffffe0000b0e588
Dec 21 09:09:53 freenas mps0: Sending reset from mpssas_send_abort for target ID 6
Dec 21 09:09:54 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 29 70 00 00 40 00 length 32768 SMID 214 command timeout cm 0xfffffe0000b05230 ccb 0xfffff8003e4ee800
Dec 21 09:09:54 freenas     (noperiph:mps0:0:4294967295:0): SMID 7 Aborting command 0xfffffe0000b05230
Dec 21 09:09:54 freenas mps0: Sending reset from mpssas_send_abort for target ID 8
Dec 21 09:09:54 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 29 70 00 00 40 00 length 32768 SMID 772 command timeout cm 0xfffffe0000b31d20 ccb 0xfffff801c234d800
Dec 21 09:09:54 freenas     (noperiph:mps0:0:4294967295:0): SMID 8 Aborting command 0xfffffe0000b31d20
Dec 21 09:09:54 freenas mps0: Sending reset from mpssas_send_abort for target ID 9
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2a 48 00 00 40 00 length 32768 SMID 162 command timeout cm 0xfffffe0000b00f90 ccb 0xfffff801c2363800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 29 c8 00 00 40 00 length 32768 SMID 567 command timeout cm 0xfffffe0000b21678 ccb 0xfffff8003e57d000
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 29 c8 00 00 40 00 length 32768 SMID 700 command timeout cm 0xfffffe0000b2c0e0 ccb 0xfffff801c2033800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2a 48 00 00 40 00 length 32768 SMID 318 command timeout cm 0xfffffe0000b0d770 ccb 0xfffff800431ae800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2a 08 00 00 40 00 length 32768 SMID 719 command timeout cm 0xfffffe0000b2d938 ccb 0xfffff8003e45b800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2a 08 00 00 40 00 length 32768 SMID 269 command timeout cm 0xfffffe0000b098a8 ccb 0xfffff804b916d000
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2a 88 00 00 40 00 length 32768 SMID 550 command timeout cm 0xfffffe0000b200b0 ccb 0xfffff8003e4e2800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2a 88 00 00 40 00 length 32768 SMID 113 command timeout cm 0xfffffe0000afd0c8 ccb 0xfffff8003e577000
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2a c8 00 00 40 00 length 32768 SMID 478 command timeout cm 0xfffffe0000b1a470 ccb 0xfffff8015e802000
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2a c8 00 00 40 00 length 32768 SMID 827 command timeout cm 0xfffffe0000b36398 ccb 0xfffff8003e581800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2b 08 00 00 40 00 length 32768 SMID 941 command timeout cm 0xfffffe0000b3f5a8 ccb 0xfffff8000ba36800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2b 08 00 00 40 00 length 32768 SMID 226 command timeout cm 0xfffffe0000b06190 ccb 0xfffff800431a7800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2b 60 00 00 40 00 length 32768 SMID 88 command timeout cm 0xfffffe0000afb0c0 ccb 0xfffff800431aa800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2b 60 00 00 40 00 length 32768 SMID 746 command timeout cm 0xfffffe0000b2fbd0 ccb 0xfffff800431dd800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2b a0 00 00 40 00 length 32768 SMID 420 command timeout cm 0xfffffe0000b15a20 ccb 0xfffff8003e576800
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2b a0 00 00 40 00 length 32768 SMID 176 command timeout cm 0xfffffe0000b02180 ccb 0xfffff8003e618800
Dec 21 09:09:57 freenas     (da6:mps0:0:8:0): WRITE(10). CDB: 2a 00 e9 9c 2b e0 00 00 40 00 length 32768 SMID 1015 command timeout cm 0xfffffe0000b45478 ccb 0xfffff801c2035000
Dec 21 09:09:57 freenas     (da7:mps0:0:9:0): WRITE(10). CDB: 2a 00 e9 9c 2b e0 00 00 40 00 length 32768 SMID 467 command timeout cm 0xfffffe0000b19658 ccb 0xfffff801c2368000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a ca f8 00 00 40 00 length 32768 SMID 367 command timeout cm 0xfffffe0000b11638 ccb 0xfffff8003e6a9800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a ca f8 00 00 40 00 length 32768 SMID 911 command timeout cm 0xfffffe0000b3cf38 ccb 0xfffff8004f3b8000
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a ca f8 00 00 40 00 length 32768 SMID 413 command timeout cm 0xfffffe0000b15128 ccb 0xfffff8003e618000
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a ca f8 00 00 40 00 length 32768 SMID 989 command timeout cm 0xfffffe0000b43328 ccb 0xfffff8004392e800
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cb 00 00 00 40 00 length 32768 SMID 964 command timeout cm 0xfffffe0000b41320 ccb 0xfffff801c2363000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cb 00 00 00 40 00 length 32768 SMID 874 command timeout cm 0xfffffe0000b39fd0 ccb 0xfffff801c2366800
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cb 38 00 00 40 00 length 32768 SMID 974 command timeout cm 0xfffffe0000b41ff0 ccb 0xfffff8000bbe2000
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cb 38 00 00 40 00 length 32768 SMID 547 command timeout cm 0xfffffe0000b1fcd8 ccb 0xfffff80043251800
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cb 38 00 00 40 00 length 32768 SMID 580 command timeout cm 0xfffffe0000b22720 ccb 0xfffff8003e60e800
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cb 38 00 00 40 00 length 32768 SMID 416 command timeout cm 0xfffffe0000b15500 ccb 0xfffff801c234e000
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cb 40 00 00 40 00 length 32768 SMID 643 command timeout cm 0xfffffe0000b277d8 ccb 0xfffff801c234d000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cb 40 00 00 40 00 length 32768 SMID 1013 command timeout cm 0xfffffe0000b451e8 ccb 0xfffff8004392f800
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cb 78 00 00 40 00 length 32768 SMID 970 command timeout cm 0xfffffe0000b41ad0 ccb 0xfffff800431db800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cb 78 00 00 40 00 length 32768 SMID 860 command timeout cm 0xfffffe0000b38de0 ccb 0xfffff8000bbd8000
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cb 78 00 00 40 00 length 32768 SMID 542 command timeout cm 0xfffffe0000b1f670 ccb 0xfffff8003e4e5000
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cb 78 00 00 40 00 length 32768 SMID 616 command timeout cm 0xfffffe0000b25540 ccb 0xfffff800431bb800
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cb 80 00 00 40 00 length 32768 SMID 169 command timeout cm 0xfffffe0000b01888 ccb 0xfffff801c2348000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cb 80 00 00 40 00 length 32768 SMID 220 command timeout cm 0xfffffe0000b059e0 ccb 0xfffff800431bb000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cb b8 00 00 40 00 length 32768 SMID 246 command timeout cm 0xfffffe0000b07b30 ccb 0xfffff800431db000
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cb b8 00 00 40 00 length 32768 SMID 237 command timeout cm 0xfffffe0000b06fa8 ccb 0xfffff8004f87d800
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cb b8 00 00 40 00 length 32768 SMID 397 command timeout cm 0xfffffe0000b13ca8 ccb 0xfffff801c2355000
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cb b8 00 00 40 00 length 32768 SMID 111 command timeout cm 0xfffffe0000afce38 ccb 0xfffff8003e6a9000
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cb c0 00 00 40 00 length 32768 SMID 765 command timeout cm 0xfffffe0000b31428 ccb 0xfffff801c2355800
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cb c0 00 00 40 00 length 32768 SMID 665 command timeout cm 0xfffffe0000b29408 ccb 0xfffff8016d54c000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cb f8 00 00 40 00 length 32768 SMID 682 command timeout cm 0xfffffe0000b2a9d0 ccb 0xfffff801c2345800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cb f8 00 00 40 00 length 32768 SMID 410 command timeout cm 0xfffffe0000b14d50 ccb 0xfffff8003e4ea800
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cb f8 00 00 40 00 length 32768 SMID 415 command timeout cm 0xfffffe0000b153b8 ccb 0xfffff8000bb23000
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cb f8 00 00 40 00 length 32768 SMID 544 command timeout cm 0xfffffe0000b1f900 ccb 0xfffff8003e4de800
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cc 00 00 00 40 00 length 32768 SMID 356 command timeout cm 0xfffffe0000b10820 ccb 0xfffff8003e4ea000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cc 00 00 00 40 00 length 32768 SMID 598 command timeout cm 0xfffffe0000b23e30 ccb 0xfffff8004322a000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cc 78 00 00 40 00 length 32768 SMID 432 command timeout cm 0xfffffe0000b16980 ccb 0xfffff801c2347800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cc 78 00 00 40 00 length 32768 SMID 89 command timeout cm 0xfffffe0000afb208 ccb 0xfffff8000bbe7000
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cc 78 00 00 40 00 length 32768 SMID 198 command timeout cm 0xfffffe0000b03db0 ccb 0xfffff8003e4de000
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cc 78 00 00 40 00 length 32768 SMID 767 command timeout cm 0xfffffe0000b316b8 ccb 0xfffff8004f2cf000
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cc 80 00 00 40 00 length 32768 SMID 937 command timeout cm 0xfffffe0000b3f088 ccb 0xfffff80043930800
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cc 80 00 00 40 00 length 32768 SMID 740 command timeout cm 0xfffffe0000b2f420 ccb 0xfffff801c2368800
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cc 38 00 00 40 00 length 32768 SMID 885 command timeout cm 0xfffffe0000b3ade8 ccb 0xfffff80043229000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cc b8 00 00 40 00 length 32768 SMID 947 command timeout cm 0xfffffe0000b3fd58 ccb 0xfffff8015e814000
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cc 38 00 00 40 00 length 32768 SMID 892 command timeout cm 0xfffffe0000b3b6e0 ccb 0xfffff8003e61d800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cc b8 00 00 40 00 length 32768 SMID 493 command timeout cm 0xfffffe0000b1b7a8 ccb 0xfffff801c2033000
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cc 38 00 00 40 00 length 32768 SMID 94 command timeout cm 0xfffffe0000afb870 ccb 0xfffff801c234f800
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cc b8 00 00 40 00 length 32768 SMID 448 command timeout cm 0xfffffe0000b17e00 ccb 0xfffff8001ad97800
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cc 38 00 00 40 00 length 32768 SMID 414 command timeout cm 0xfffffe0000b15270 ccb 0xfffff801c234e800
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cc b8 00 00 40 00 length 32768 SMID 951 command timeout cm 0xfffffe0000b40278 ccb 0xfffff801c2034800
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cc 40 00 00 40 00 length 32768 SMID 763 command timeout cm 0xfffffe0000b31198 ccb 0xfffff8003e577800
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cc c0 00 00 40 00 length 32768 SMID 985 command timeout cm 0xfffffe0000b42e08 ccb 0xfffff8000bbea000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cc 40 00 00 40 00 length 32768 SMID 680 command timeout cm 0xfffffe0000b2a740 ccb 0xfffff8003e445000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cc c0 00 00 40 00 length 32768 SMID 995 command timeout cm 0xfffffe0000b43ad8 ccb 0xfffff8003e45b000
Dec 21 09:09:57 freenas     (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 2b 5a cc f8 00 00 40 00 length 32768 SMID 129 command timeout cm 0xfffffe0000afe548 ccb 0xfffff8004324f800
Dec 21 09:09:57 freenas     (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 2b 5a cc f8 00 00 40 00 length 32768 SMID 832 command timeout cm 0xfffffe0000b36a00 ccb 0xfffff801c2367000
Dec 21 09:09:57 freenas     (da4:mps0:0:6:0): WRITE(10). CDB: 2a 00 2b 5a cc f8 00 00 40 00 length 32768 SMID 915 command timeout cm 0xfffffe0000b3d458 ccb 0xfffff8000bb2b800
Dec 21 09:09:57 freenas     (da5:mps0:0:7:0): WRITE(10). CDB: 2a 00 2b 5a cc f8 00 00 40 00 length 32768 SMID 921 command timeout cm 0xfffffe0000b3dc08 ccb 0xfffff8003e4ee000
Dec 21 09:09:57 freenas     (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 2b 5a cd 00 00 00 40 00 length 32768 SMID 538 command timeout cm 0xfffffe0000b1f150 ccb 0xfffff8000bbd3000
Dec 21 09:09:57 freenas     (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 2b 5a cd 00 00 00 40 00 length 32768 SMID 838 command timeout cm 0xfffffe0000b371b0 ccb 0xfffff80043932800 

Thanks again

 

[tvsjr]

These are I/O errors. I'd be looking at drives (perhaps one of the new ones you added has something squirrely with it), cables, backplane, controller. A flaky PSU could also be the culprit.

 

[Goobz]

These are I/O errors. I'd be looking at drives (perhaps one of the new ones you added has something squirrely with it), cables, backplane, controller. A flaky PSU could also be the culprit.

Ahh ok thanks for your reply. Can you suggest maybe a first point of troubleshooting?

If I were to say remove the hdd's one at a time and then reboot that should rule that out if the fault persists correct? I bit the bullet and ordered another M1015 card off ebay so I'll try that once it arrives. I'll try to source a couple of new SAS breakout cables as well to rule that out. I read on here somewhere someone had a screwy SAS cable causing some major issues. Not sure if I have another PSU floating around but I can rob one from another PC temporarily if need be.

I'll update here once I have tried a few things. Thanks again for you input!

 

[cyberjock]

Did you figure this out? If not, can you provide a debug file (system -> Advanced -> Save Debug).

 

[Goobz]

Did you figure this out? If not, can you provide a debug file (system -> Advanced -> Save Debug).

Hi sorry for not getting back sooner. The last week or 2 have been a bit full on and I've been doing what I can where I can and it looks to be ok now.

I tried new cables and unplugging the drives one by one with no luck. I then replaced the M1015 card once it arrived and everything seemed to be ok after that. It's been up for about 3 days now with no issue so I guess it I can rule the M1015 as the culprit.

Just a quick question, is there some way to tell if it is a hardware fault on the card or just a corrupted firmware, or do I just reflash the old card and throw it in and see what happens? It would be a shame to trash the old card if it is salvageable.

Thanks again for the help!

 

[cyberjock]

There really isn't a way to check for a hardware fault. For the most part, if there is a hardware fault, the card should fail its intialization routine that begins during POST testing. So in your case I'd probably try reflashing it all and try to use it. If it still misbehaves simply trash it.

It is at least possible that the PCIe slot in your motherboard wasn't making good contact with the card. If you replaced the M1015 and used a different slot it's still possible. In fact, it's still possible to be the cause even if you used the same slot since you reseated a new card in the system. ;)