Truenas Compatibility Issues With HPE BCM57414?

[Izaak_Newton]

Hello Guys,

We have been pleasantly using Truenas 12.4/FreeBSD for over a year now.
We recently upgraded to a set of Supermicro 5049P-E1CTR36L 4U and upgraded them with a HPE Broadcom BCM57414 25G network card.
We thought everything was working fine untill some of them randomly gave "TX completion errors" and are unable to forward any traffic.

Right now we are running the onboard 10Gbit without any issues, but we need the 25Gbit working as soon as possible. Does anyone has a clue what to try?



Some logs below:

bnxt0@pci0:179:0:0: class=0x020000 card=0x159714e4 chip=0x16d714e4 rev=0x01 hdr=0x00
vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0x38fffff10000, size 65536, enabled
bar [18] = type Prefetchable Memory, range 64, base 0x38ffffe00000, size 1048576, enabled
bar [20] = type Prefetchable Memory, range 64, base 0x38fffff22000, size 8192, enabled
cap 01[48] = powerspec 3 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
cap 11[a0] = MSI-X supports 74 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
link x8(x8) speed 8.0(8.0) ClockPM disabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[13c] = Serial 1 e43d1afffe0deca0
ecap 0004[150] = Power Budgeting 1
ecap 0002[160] = VC 1 max VC0
ecap 000b[180] = Vendor 1 ID 0
ecap 0018[1b0] = LTR 1
ecap 000e[1b8] = ARI 1
ecap 0017[230] = TPH Requester 1
ecap 0019[300] = PCIe Sec 1 lane errors 0
ecap 001f[200] = Precision Time Measurement 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = '10/25Gb 2-port SFP28 BCM57414 Adapter'
VPD ro PN = 'P26264-001'
VPD ro MN = '14E4'
VPD ro V0 = '214.0.194.0'
VPD ro V1 = '214.0.305.0'
VPD ro V3 = '214.4.91.1'
VPD ro V6 = '214.0.241.0'
VPD ro V7 = '216.0.52.1'
VPD ro V8 = '214.4.42.1'
VPD ro VA = '216.0.333.11'
VPD ro SN = 'VNE111024Y'
VPD ro VB = 'REV005DEV000'
VPD ro V2 = '5111'
VPD ro V4 = 'E43D1A0DECA0'
bnxt1@pci0:179:0:1: class=0x020000 card=0x159714e4 chip=0x16d714e4 rev=0x01 hdr=0x00
vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0x38fffff00000, size 65536, enabled
bar [18] = type Prefetchable Memory, range 64, base 0x38ffffd00000, size 1048576, enabled
bar [20] = type Prefetchable Memory, range 64, base 0x38fffff20000, size 8192, enabled
cap 01[48] = powerspec 3 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
cap 11[a0] = MSI-X supports 74 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
link x8(x8) speed 8.0(8.0) ClockPM disabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[13c] = Serial 1 e43d1afffe0deca0
ecap 0004[150] = Power Budgeting 1
ecap 000b[180] = Vendor 1 ID 0
ecap 000e[1b8] = ARI 1
ecap 0017[230] = TPH Requester 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = '10/25Gb 2-port SFP28 BCM57414 Adapter'
VPD ro PN = 'P26264-001'
VPD ro MN = '14E4'
VPD ro V0 = '214.0.194.0'
VPD ro V1 = '214.0.305.0'
VPD ro V3 = '214.4.91.1'
VPD ro V6 = '214.0.241.0'
VPD ro V7 = '216.0.52.1'
VPD ro V8 = '214.4.42.1'
VPD ro VA = '216.0.333.11'
VPD ro SN = 'VNE111024Y'
VPD ro VB = 'REV005DEV000'
VPD ro V2 = '5111'
VPD ro V4 = 'E43D1A0DECA0'
bnxt0: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem 0x38fffff10000-0x38fffff1ffff,0x38ffffe00000-0x38ffffefffff,0x38fffff22000-0x38fffff23fff irq 50 at device 0.0 numa-domain 0 on pci11
bnxt0: Using 256 TX descriptors and 256 RX descriptors
bnxt0: Using 8 RX queues 8 TX queues
bnxt0: Using MSI-X interrupts with 9 vectors
bnxt0: Ethernet address: e4:3d:1a:0d:ec:a0
bnxt1: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem 0x38fffff00000-0x38fffff0ffff,0x38ffffd00000-0x38ffffdfffff,0x38fffff20000-0x38fffff21fff irq 53 at device 0.1 numa-domain 0 on pci11
bnxt1: Using 256 TX descriptors and 256 RX descriptors
bnxt1: Using 8 RX queues 8 TX queues
bnxt1: Using MSI-X interrupts with 9 vectors
bnxt1: Ethernet address: e4:3d:1a:0d:ec:a1
bnxt1: Unknown phy type

 

[jgreco]

Yup. A NAS is extremely intensive on network traffic, and many devices are really not quite up to snuff.

The strategy of buying a rando card and then expecting FreeBSD to work perfectly with it isn't a great strategy, I'm sorry to say. Unlike buying a system from a vendor, where they will have confirmed the compatibility between the cards and the OS they're installing on it for you, build-your-own-servers require you to do this research and validation yourself. It's a hazard, I agree.

Broadcom's historically not done a fantastic job on FreeBSD, so what you really need to do is to research what cards are known to work well with FreeBSD and then buy one of those. There is some discussion of this in the 10 Gig Networking Primer, I believe. I would expect the Intel XXV710-DA2 to be a likely candidate, though I have not tried one. Intel's driver team has generally done really well for FreeBSD. There may be Chelsio and Mellanox options as well.

 

[Izaak_Newton]

Hi Jgreco, thanks for your quick answer.
I'll be ordering one of the Intel XXV710-DA2 25G cards and test them.

Will give you guys the results here!

 

[jgreco]

Please do, and if you would be so kind, please report successes and failures in the 10 Gig Networking Primer thread as well, to help out others on the forum struggling with the same issues.

 

[Ericloewe]

Yes, I'm curious about how the 25G stuff works in practice. I just got two new servers at work with BCM57414 NICs (Dell wouldn't sell anything other than Broadcom even though it's just an OCP 2.0 slot) after their 10G counterpart worked well enough on Linux. The next major upgrade to the network will likely involve 25GbE to the NAS servers and the XXV710 is at the top of the shortlist, pending real-world data.