Advice on Critical Interrupt Error

[MikeyG]

Every month or so, I get these messages from TrueNAS (12-U2.1):

Code:

Apr 15 05:30:48 nas t5iov0: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas t5iov0: status 0x00000041 mask 0x00002000
Apr 15 05:30:48 nas t5iov0: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas t5iov0: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas t5iov1: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas t5iov1: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas t5iov1: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas t5iov1: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas t5iov2: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas t5iov2: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas t5iov2: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas t5iov2: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas t5iov3: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas t5iov3: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas t5iov3: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas t5iov3: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas t5nex0: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas t5nex0: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas t5nex0: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas t5nex0: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas unknown: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas unknown: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas unknown: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas unknown: Clearing PCIe errors 0x0009
Apr 15 05:30:48 nas unknown: Correctable PCIe error reported by APEI
Apr 15 05:30:48 nas unknown: status 0x00002041 mask 0x00002000
Apr 15 05:30:48 nas unknown: Clearing COR AER errors 0x00002000
Apr 15 05:30:48 nas unknown: Clearing PCIe errors 0x0009

At the same time in IPMI I see this:

Code:

    79        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func0) - Assertion
    80        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func1) - Assertion
    81        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func2) - Assertion
    82        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func3) - Assertion
    83        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func4) - Assertion
    84        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func5) - Assertion
    85        2021/04/15 05:30:22    Critical Interrupt    PCI PERR @Bus66 (Dev0, Func6) - Assertion

In dmidecode, it appears that this is slot 7, which is where the Chelsio card is plugged in:

Code:

System Slot Information
    Designation: CPU SLOT7 PCI-E 3.0 x 8
    Type: x8 PCI Express 3 x8
    Current Usage: In Use
    Length: Long
    ID: 7
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:66:00.0

Hardware:
Supermicro X11SRM-F
2x LSI 9400-16is
Chelsio T620-SO-CR
Memory: MEM-DR464L-CL02-LR26
Seasonic Focus 650W 80 Plus Gold

I was getting the same errors in IPMI in Freenas 11.3, but now that I'm on 12 I'm getting this additional information in the logs. It happens at all different times, and does not appear correlated to a specific load or activity.

As far as I know, this has never actually caused a visible problem. No freezes, crashes, reboots, etc.

Any ideas on what this could be?