SAS Log Messages - not sure what they mean

[ZFSLover]

Hi All,

I have a system with 12x SAS HDD's, all is working fine, but I keep getting the below message in my log file for 2 HDD's. They have both passed long SMART tests.

Code:

[sdf] tag#418 BRCM Debug mfi stat 0x2d, data len requested/completed 0x100000/0x0

Any idea what these mean?

 

[HoneyBadger]

mfi

Are you using a hardware RAID controller with your HDDs? They are known to interfere with reading full SMART data, and are generally considered a "Bad Idea" for several other reasons.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[ZFSLover]

No, it is supposed to be HBA. Certainly no HW raid is configured and no raid options in the bios on the machine.

Code:

01:00.0 Serial Attached SCSI controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
    Subsystem: QUANTA Computer Inc MegaRAID SAS-3 3108 [Invader]
    Physical Slot: 2
    Flags: bus master, fast devsel, latency 0, IRQ 24, NUMA node 0, IOMMU group 25
    I/O ports at e000
    Memory at fbb00000 (64-bit, non-prefetchable) [size=64K]
    Memory at fba00000 (64-bit, non-prefetchable) [size=1M]
    Expansion ROM at fb900000 [disabled] [size=1M]
    Capabilities: [50] Power Management version 3
    Capabilities: [68] Express Endpoint, MSI 00
    Capabilities: [d0] Vital Product Data
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [c0] MSI-X: Enable+ Count=97 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [1e0] Secondary PCI Express
    Capabilities: [1c0] Power Budgeting <?>
    Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
    Kernel driver in use: megaraid_sas
    Kernel modules: megaraid_sas

Also, it is just reporting on the same 2 HDD's, none of the rest.

 

[HoneyBadger]

The SAS3108 is an IR/RAID card by default, I believe. What does the output of sas3flash -list show for Firmware Product ID - (IT) or (IR)?

Are all of the HDD's the same model? It could be power management spinning them down and they're delaying on spinup.

 

[ZFSLover]

Code:

vago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    No Avago SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -list
    Exiting Program.

All HDD's are identical 8TB HGST SAS3, power/spin down is disabled on all of em.

 

[HoneyBadger]

There should definitely be at least one LSI/Avago SAS adaptor detected there. I have to assume at this point that it isn't an HBA and has some manner of RAID logic or is in "JBOD Mode."

The underlying error seems to relate to device resets or timeouts, so the next step would be the look at the hardware itself. Is it possible to swap two drives in the chassis to see if the fault follows the slot vs drive?