SOLVED Bad controller? (No - ASRock mb shipped with ancient BIOS)

[jonl]

Hi folks - I built out a new system and as I move data onto it I'm seeing a couple things that seem to indicate controller issues. If you could help confirm that, and assess if these are replace-it-worthy that'd be awesome.

Controller: LSI 6Gbps SAS HBA LSI 9201-8i
The 8 drives are split between two SFF-8087 ports/cables, so if it's a cable problem it's either both uniformly, or the controller.

First, I'm getting these notifications (scale) periodically (typically after boot, but here and there during runtime also): Warning: Critical Interrupt #0xfe Asserted Bus Degraded.

It's an asrock rack mb, and from the IPMI I can see the source event:

ID	Record Type	Time Stamp	GenID(Low)	GenID(High)	EvMRev	Sensor Type	Sensor #	EvtDir Type	Event Data1	Event Data2	Event Data3
025fh	02h	645a802fh	01h	00h	04h	13h	feh	6fh	abh	01h	01h

Event Data1 Bus Degraded
Event Data2 Bus number: 0x01
Event Data3 Link speed degraded

Second, I'm also seeing checksum errors that manifest uniformly across all 8 drives on that controller.

Code:

❯ sudo zpool status -v data
  pool: data
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 14:27:35 with 1 errors on Tue May  9 22:35:13 2023
config:

    NAME                                      STATE     READ WRITE CKSUM
    data                                      ONLINE       0     0     0
      raidz2-0                                ONLINE       0     0     0
        ab020c34-c835-4400-8600-7602eb70daab  ONLINE       0     0     2
        19d49ffb-60d8-4613-b34a-574d2666d603  ONLINE       0     0     2
        339c8b3b-01ea-467e-bdb6-25086b937e1e  ONLINE       0     0     2
        6a7e094f-9fb7-42ba-9c16-1bf3d5768a39  ONLINE       0     0     2
        f92c134b-0a3d-430e-aa7e-f89a34afa9c6  ONLINE       0     0     2
        eccd1f92-f822-4da3-ac44-3ed73af98e71  ONLINE       0     0     2
        8c4a1def-4d8f-4834-b102-417807f8e469  ONLINE       0     0     2
        26ee378e-1893-467a-9951-cd27d199e3fb  ONLINE       0     0     2

I had 10 the last time, but cleared them to see if they'd recur.

Here's the smartctl output for a couple of drives in the array in case it's useful:

Code:

❯ sudo smartctl --all /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST18000NM000J-2TV103
Serial Number:    ZR5BKMDK
LU WWN Device Id: 5 000c50 0e52bad24
Firmware Version: SN02
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed May 10 07:19:05 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  567) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (1537) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x70bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       168337720
  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       20
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   085   060   045    Pre-fail  Always       -       319507513
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       5501
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       20
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   055   000    Old_age   Always       -       33 (Min/Max 31/40)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       9
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1057
194 Temperature_Celsius     0x0022   033   045   000    Old_age   Always       -       33 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       5451 (212 132 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       121194232748
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       97375998482082

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      5430         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

❯ sudo smartctl --all /dev/sdg
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 EVO 2TB
Serial Number:    S6PNNS0T808464E
LU WWN Device Id: 5 002538 f3280338d
Firmware Version: SVT02B6Q
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed May 10 07:19:08 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x53) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 160) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       5500
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       20
177 Wear_Leveling_Count     0x0013   093   093   000    Pre-fail  Always       -       152
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   079   053   000    Old_age   Always       -       21
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       7
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       164059180752
252 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
  256        0    65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[WN1X]

Have you tried cleaning the contacts and reseating the HBA board?

 

[jonl]

Have you tried cleaning the contacts and reseating the HBA board?

Thanks - done. Saw the same bus errors on the subsequent boot (also while I hadn't cleaned it until just now, I've reseated it a few times for various reasons with errors before/after)

 

[jgreco]

First, I'm getting these notifications (scale) periodically (typically after boot, but here and there during runtime also): Warning: Critical Interrupt #0xfe Asserted Bus Degraded.

Check the temperature of the heatsink on the HBA. LSI's controllers are known for throwing random errors if they are running too hot. See point #8 in the following article --

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

and note that front to back airflow is very necessary for these controllers.

 

[jonl]

Check the temperature of the heatsink on the HBA. ... front to back airflow is very necessary for these controllers.

Thanks -- This asrock rack board needs some amount of manual fan configuration, and I was just thinking about how I was putting a lot of attention into making sure my CPU "fan" was correct but I never configured anything for the other fans. The controller heatsink was certainly warm to the touch but not hot hot, though perhaps enough be a problem. I'll see about configuring temp sensors/case fan mappings to improve the cooling.

Current temps (via IPMI) are 26-34 (deg C) across CPU/card side/mb.

 

[jonl]

I'm still working on the fan config, but I'm getting the bus asserts on boot even after doing some other checks (that allowed everything to cool down if >30 is problematic.)

Art of Server was gracious enough to consider the problem and suggested I check the pins in the slot and processor socket for anything out of place, and to ensure the cooler is tightened evenly when I reseated the processor. They also suggested I could set the link speed for slot 4/6 to "gen 2" or equivalent to skip any link speed negotiation problems, all of which I did and still the bus asserts remain.

 

[jonl]

Update - Art of Server sent me another controller to try, but that didn't do it, so I contacted ASRock support who provided a BIOS update (upgraded from 0.02 to 2.11) which resolved the issue.

After I asked, there is also a BMC update (0.01 to 1.02.) Why these aren't on their website I have no idea.

 

[WN1X]

Glad to hear you were able to resolve the problems.