BNX2X Errors

[clifford64]

I recently upgraded to a new MOBO and TrueNAS Scale. After the upgrade, I have been experiencing an issue with a 10gb card that I have been using. It has happened a few times now. Every couple of weeks, all traffic stops and the NIC becomes unresponsive. Rebooting the system fixes it, but it appears to be happening sporadically.

I recently upgraded the NIC firmware because I saw a few other threads about this being a firmware issue, but I am still experiencing the issue with what I believe is the latest firmware I received from QLogic's website. The nic is a Broadcom 57810 SFP. I have this same card with the same updated firmware running in two other ESXi hosts (different mobo and setup) and they don't have this issue. It just started happening after I upgraded the motherboard to a Supermicro X10DAi and upgraded to TrueNAS scale. I have been running on this setup for a few months now without any other issues other than this.

Any thoughts as to what might be causing and is there a way to fix it without replacing the NIC?

Code:

Sep  4 19:39:28 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f1)]MFW seems hanged: drv_pulse (0x6536) != mcp_pulse (0x7fff)
Sep  4 19:39:28 truenas kernel: bnx2x: [bnx2x_hw_stats_update:870(enp129s0f1)]NIG timer max (4294967295)
Sep  4 19:39:28 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f1)]lock_status 0xffffffff  resource_bit 0x1
Sep  4 19:39:28 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:28 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:28 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: Error:  Power fault on Port 0 has been detected and the power to that SFP+ module has been removed to prevent failure of the card. Please remove the SFP+ module and restart the
system to clear this error.
Sep  4 19:39:29 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f0)]MFW seems hanged: drv_pulse (0x653a) != mcp_pulse (0x7fff)
Sep  4 19:39:29 truenas kernel: bnx2x: [bnx2x_hw_stats_update:870(enp129s0f0)]NIG timer max (4294967295)
Sep  4 19:39:29 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f0)]lock_status 0xffffffff  resource_bit 0x1
Sep  4 19:39:29 truenas kernel: bnx2x 0000:81:00.0 enp129s0f0: MDC/MDIO access timeout
Sep  4 19:39:29 truenas kernel: bnx2x 0000:81:00.0 enp129s0f0: MDC/MDIO access timeout
Sep  4 19:39:29 truenas kernel: bnx2x 0000:81:00.0 enp129s0f0: Error:  Power fault on Port 0 has been detected and the power to that SFP+ module has been removed to prevent failure of the card. Please remove the SFP+ module and restart the
system to clear this error.
Sep  4 19:39:29 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f1)]MFW seems hanged: drv_pulse (0x6537) != mcp_pulse (0x7fff)
Sep  4 19:39:29 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f1)]lock_status 0xffffffff  resource_bit 0x1
Sep  4 19:39:29 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:29 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:30 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f0)]MFW seems hanged: drv_pulse (0x653b) != mcp_pulse (0x7fff)
Sep  4 19:39:30 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f0)]lock_status 0xffffffff  resource_bit 0x1
Sep  4 19:39:30 truenas kernel: bnx2x 0000:81:00.0 enp129s0f0: MDC/MDIO access timeout
Sep  4 19:39:30 truenas kernel: bnx2x 0000:81:00.0 enp129s0f0: MDC/MDIO access timeout
Sep  4 19:39:30 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f1)]MFW seems hanged: drv_pulse (0x6538) != mcp_pulse (0x7fff)
Sep  4 19:39:30 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f1)]lock_status 0xffffffff  resource_bit 0x1
Sep  4 19:39:30 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:30 truenas kernel: bnx2x 0000:81:00.1 enp129s0f1: MDC/MDIO access timeout
Sep  4 19:39:31 truenas kernel: bnx2x: [bnx2x_timer:5811(enp129s0f0)]MFW seems hanged: drv_pulse (0x653c) != mcp_pulse (0x7fff)
Sep  4 19:39:31 truenas kernel: bnx2x: [bnx2x_acquire_hw_lock:2023(enp129s0f0)]lock_status 0xffffffff  resource_bit 0x1