LSI-9500 HBA resets and backplanes/drives disappear during high load/scrub

[sretalla]

figured it out, for anyone else: download from https://docs.broadcom.com/docs/1232743397 - unzip then dpkg -i Ubuntu/storcli_007.2705.0000.0000_all.deb and add either /opt/MegaRAID/storcli or /opt/MegaRAID/storcli/storcli64 to your path

That's one way.

The other would be to use ubernerd/nerdctl (https://github.com/Jip-Hop/ubernerd) to launch a privileged container and install/run storcli and my script from within that.

 

[kingtool]

Not sure if it's interesting, but I have a resource available to log temperatures (disks and LSI HBA) to an influxdb (then paired with Grafana to chart)...

InfluxDB Disk Temperature Logging by Serial Number (or chosen substitution)

It was brought to my attention that it may be useful for some of the forum members to log disk temperatures periodically to an influxDB (and then chart that with Grafana like the below example) to help with trends for individual disks or...

www.truenas.com

That may help you to keep an more consistent eye on the temps.

A note on this: you are using "storcli show all" when you can just use "storcli show temperature" to get the ROC temp. "show all" can stress the disks if you have a lot of them on the controller (due to the detailed data it pulls for every disk in a seemingly synchronous way), and if your command is running frequently enough it can hit the goldilocks moment of "drives were in flight on a read/write, then this flurry hit, and now a timeout has occurred, and now mpt3sas is resetting the controller". Your util doesn't seem to need anything besides "show temperature" so might be an easy patch and speed it up too.

As I mentioned in a previous post in this thread, I went through *hell* troubleshooting this because it seemed like the very act of measuring the temps on my controllers was causing random resets.

 

[sretalla]

If you read the post directly before yours, I'm mentioning how to get the version of storcli that supports that "show temperature" switch, but knowing that the included one in TrueNAS doesn't support it, hence my script using what it has access to... it's easy enough for somebody who wants to to change that line of the script and use the right version if they have worked that out.

The script allows (and actually defaults to) an opt-out of taking those temps.