Hi folks - I built out a new system and as I move data onto it I'm seeing a couple things that seem to indicate controller issues. If you could help confirm that, and assess if these are replace-it-worthy that'd be awesome.
Controller: LSI 6Gbps SAS HBA LSI 9201-8i
The 8 drives are split between two SFF-8087 ports/cables, so if it's a cable problem it's either both uniformly, or the controller.
First, I'm getting these notifications (scale) periodically (typically after boot, but here and there during runtime also): Warning: Critical Interrupt #0xfe Asserted Bus Degraded.
It's an asrock rack mb, and from the IPMI I can see the source event:
Event Data1 Bus Degraded
Event Data2 Bus number: 0x01
Event Data3 Link speed degraded
Second, I'm also seeing checksum errors that manifest uniformly across all 8 drives on that controller.
I had 10 the last time, but cleared them to see if they'd recur.
Here's the smartctl output for a couple of drives in the array in case it's useful:
Controller: LSI 6Gbps SAS HBA LSI 9201-8i
The 8 drives are split between two SFF-8087 ports/cables, so if it's a cable problem it's either both uniformly, or the controller.
First, I'm getting these notifications (scale) periodically (typically after boot, but here and there during runtime also): Warning: Critical Interrupt #0xfe Asserted Bus Degraded.
It's an asrock rack mb, and from the IPMI I can see the source event:
| ID | Record Type | Time Stamp | GenID(Low) | GenID(High) | EvMRev | Sensor Type | Sensor # | EvtDir Type | Event Data1 | Event Data2 | Event Data3 |
| 025fh | 02h | 645a802fh | 01h | 00h | 04h | 13h | feh | 6fh | abh | 01h | 01h |
Event Data2 Bus number: 0x01
Event Data3 Link speed degraded
Second, I'm also seeing checksum errors that manifest uniformly across all 8 drives on that controller.
Code:
❯ sudo zpool status -v data
pool: data
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 14:27:35 with 1 errors on Tue May 9 22:35:13 2023
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ab020c34-c835-4400-8600-7602eb70daab ONLINE 0 0 2
19d49ffb-60d8-4613-b34a-574d2666d603 ONLINE 0 0 2
339c8b3b-01ea-467e-bdb6-25086b937e1e ONLINE 0 0 2
6a7e094f-9fb7-42ba-9c16-1bf3d5768a39 ONLINE 0 0 2
f92c134b-0a3d-430e-aa7e-f89a34afa9c6 ONLINE 0 0 2
eccd1f92-f822-4da3-ac44-3ed73af98e71 ONLINE 0 0 2
8c4a1def-4d8f-4834-b102-417807f8e469 ONLINE 0 0 2
26ee378e-1893-467a-9951-cd27d199e3fb ONLINE 0 0 2I had 10 the last time, but cleared them to see if they'd recur.
Here's the smartctl output for a couple of drives in the array in case it's useful:
Code:
❯ sudo smartctl --all /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST18000NM000J-2TV103
Serial Number: ZR5BKMDK
LU WWN Device Id: 5 000c50 0e52bad24
Firmware Version: SN02
User Capacity: 18,000,207,937,536 bytes [18.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed May 10 07:19:05 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1537) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 082 064 044 Pre-fail Always - 168337720
3 Spin_Up_Time 0x0003 091 090 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 20
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 085 060 045 Pre-fail Always - 319507513
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5501
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 20
18 Unknown_Attribute 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 067 055 000 Old_age Always - 33 (Min/Max 31/40)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 9
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1057
194 Temperature_Celsius 0x0022 033 045 000 Old_age Always - 33 (0 21 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5451 (212 132 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 121194232748
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 97375998482082
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 5430 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
❯ sudo smartctl --all /dev/sdg
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 EVO 2TB
Serial Number: S6PNNS0T808464E
LU WWN Device Id: 5 002538 f3280338d
Firmware Version: SVT02B6Q
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed May 10 07:19:08 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 160) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 5500
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 20
177 Wear_Leveling_Count 0x0013 093 093 000 Pre-fail Always - 152
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0032 079 053 000 Old_age Always - 21
195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 7
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 164059180752
252 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Last edited:
