CAM status: CCB request completed with error

[SaltyCoffee]

Last week I started getting spammed by this error out of the blue. Oddly it doesn't show up in any of the logs. It only prints to the console:

Code:

(da0:unass-sim0:0:0:0): WRITE(10). CDB: 2a 08 83 48 41 3d 88 88 20 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 03 48 54 57 08 08 01 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 08 83 48 54 57 00 00 01 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 03 48 54 57 08 00 01 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 80 83 48 54 57 08 08 81 08
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 0 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 88 03 48 54 57 88 88 81 80
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 88 82 88 4c 39 88 88 87 88
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 08 82 88 4c 39 88 88 87 88
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 88 02 88 4c 39 88 89
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10), CDB: 2a 88 02 88 4c 39 89 88 87 88
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:unass-sim0:0:0:0): Retrying command, 0 more tries remain
(da0:umass-sim0:0:0:0): WRITE(10), CDB: 2a 80 82 88 4c 39 88 88 87 88
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended
.
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.

It takes an arbitrary amount of time after boot for this to happen. It's taken as little as 15 minutes and as long as several hours. But, once this happens, services start dropping off one by one until nothing is reachable. The only thing you can do is hard reset at this point (can't reboot gracefully).

You'll notice that the disk is a usb and that's because when this first started happening, it killed (I think) my boot drive.

• But it happens with any drive, on any SATA port. And on any USB. I've installed TN about 10 times by now on various disks and sticks and nothing seems to make it stop.
• I've even downgraded TN to 12.0 since this started happening shortly after upgrading to 13.
• I've reseated all the components physically since that seemed to work for some who got this error.
• I read somewhere to scrub the boot pool to see if any error comes up, but the boot pool doesn't come up as an option. Also, it happens on any disk so...

I fear the worst, that the SATA bus is toast (but shouldn't USB work then?). The data pool drives appear to be unaffected though, also on the SATA bus, further complicating troubleshooting. But I don't know what else it could be or what I can try? I thought maybe a PCI M.2 expansion? But wanted to probe the brains here first.

 

[SaltyCoffee]

Just to update this for any poor soul who happens upon this thread on the same unfortunate journey I've been on (I know there will be given the lonely posts I've seen till now on this topic). I finally think I solved it with a SSD. I've gone all day without any catastrophic unscheduled reboots. There were a few in the first 20 minutes which also are still unexplained (graceful reboot completely on it's own with nothing in the log to explain why, so seemingly unrelated). But since then it's been humming along, doing file transfers the whole time with no issue like nothing ever happened. I still have no idea what the cause of the issue is, only that installing TN on an SSD as the boot drive (not HDD or USB) seems to have made the problem go away. Seems completely random, but what do I know. Anyway, fwiw.