CAM status: CCB request completed with an error

[iamcorvin]

I've recently upgraded my freenas server. I added a second 8x3TB Z2 vdev to my pool. Everything went fine, I had a couple of weeks of glorious free space. Then I started getting emails, the pool was degraded, it was restored and being reslivered, over and over. I checked the console and I found these errors.

https://imgur.com/a/MAcgr

I also found a security log email that it managed to send out before I shut it down.

https://pastebin.com/mu9Ng8Fm

I upgraded from using the onboard SATA ports to using an HBA with an expander card connecting to the backplane on a Norco RPC-4224. I've already replaced the HBA and have reseated all the SAS cables, still getting the same errors. I've shut down the system for now while I try and figure out what to test/replace next.

Any thoughts?

Hardware:
ASRock H87WS-DL ATX Server Motherboard LGA 1150 Intel H87 DDR3 1600/1333
Intel Celeron G1840 Dual Core Socket LGA1150, 2.80 GHz, 2MB L3 Cache, 22nm (Retail Boxed) (BX80646G1840)
16 GB of ECC RAM
Case Norco RPC-4224 (24 bay hotswap bays with SAS connectors)
Dell Kit 487738-001 HP 24 Port PCI-e 3Gb SAS Expander
Dell PERC H200 in IT mode
2x 8x3TB Raidz2 Vdevs in a single pool.

 

[Chris Moore]

Did you do burn-in testing on the drives before putting them in the pool?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[iamcorvin]

Did you do burn-in testing on the drives before putting them in the pool?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

No, I didn't burn in the drives, don't think that's the problem though.

 

[Chris Moore]

What kind of airflow do you have over the cooler on the SAS expander?
If you already replaced the SAS controller and the drives are good, that is the only thing that I would suggest as the failed item.
It would have been better to directly connect the drives and test them to ensure that they are healthy.
What kind of drives are you using?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[droeders]

How big is your power supply? How are you connecting to the 6 molex connectors on that backplane (i.e. all direct connects with no splitters)?

 

[Chris Moore]

How big is your power supply? How are you connecting to the 6 molex connectors on that backplane (i.e. all direct connects with no splitters)?

Good point. If the power supply was borderline too small, and now is failing, it might be putting out unstable power.

 

[iamcorvin]

Turns out it was the expander overheating. My fan wall was not running for some reason, after reseating all the fans they are now spinning up and the errors are gone. All is well again. Thanks for the assistance.