What error is this?

[Phase]

The server worked well for 6 months, but now it freezes (network connection drops and console seems unresponsive) 15-30 mins after it boots. I removed the NIC and the behavior is the same. I replaced the SATA interface card and still the same.

I took a picture of the frozen server screen. The picture is attached. The bellow is an OCR

(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted hcich0: Timeout on slot 3 port 0 hcich0: is 00000000 cs 00000038 ss 00000038 rs 00080038 tfd 80 serr 00000000 cmd 0015d817 (ada0:ahcich0:0:0:0): HEAD FPDNA -QUEUED. ACB: 60 28 c0 99 02 40 02 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAH status: Command timeout (ada0:ahcich0:0:0:0): Error 5, Periph was invalidated (ada0:ahcich0:0:0:0): MHITE_FPDNA_QUEUED. ACB: 61 18 20 41 70 40 02 00 00 00 08 10 (ada0:ahcich0:0:0:0): CAN status: Unconditionally He-queue Request (ada0:ahcich0:0:0:0): Error 5, Periph was invalidated (ada0:ahcich0:0:0:0): HEAD_FPDNA_QUEUED. ACB: 60 38 f8 c3 24 40 02 00 00 00 00 0 (ada0:ahcich0:0:0:0): CAN status: Unconditionally Re-queue Request (ada0:ahcich0:0:0:0): Error 5, Periph was invalidated (ada0:ahcich0:0:0:0): Periph destroyed ahcich0: AHCI reset: device not ready after 31000s (tfd = 00000080) ahcich0: Poll timeout on slot 6 port 0 ahcich0: is 00000000 cs 00000040 ss 00000000 rs 00000040 tfd 80 serr 00000800 cmd 0015d817 (aprobe0:ahcicl10:0:0:0): SOFT_HESET, ACB: 00 00 00 00 00 00 00 00 00 00 00 08 (aprobe0:ahcich0:0:0:0): CAN status: Command timeout (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted

I'm thinking of moving it to a new server (keeping the same NIC and drives), but before I start chasing my tail on that I wanted another opinion.

Thanks!!

 

[Phase]

FYI, seems to have been cables/connectors getting loose. Upgraded to v12 and in the process I exported all pools so that I could disconnect all data drives and make sure I was installing it in the right drive. After the upgrade and reconnecting the drives the error seems to have gone away.

 

[Phase]

Nope, it came back... LOL Replacing the card and cabling.

 

[Phase]

Issue was with the boot pool. Changed controller to an LSI one and the boot SSD was no longer recognized. Replacing that pool and trying again tomorrow, when I get the new SSDs.

 

[Phase]

Yeap, that was it. Stable now for a few days.