SOLVED Problem with Broadcom NetXtreme II 10 Gigabit Ethernet (BCM57840)

[sirszabo]

Dear All,

I have a problem with FreeNAS-11.1-U4 (bare metal) installation with Broadcom NetXtreme II 10 Gigabit Ethernet (BCM57840) card.
The correct model/type is -> https://www.asus.com/us/Commercial-Servers-Workstations/PEB10G578402T/
The NAS connected directly in two VMware ESXi 6.5 host, where also these cards has been installed.
When I configured the communication between NAS and the VMware hosts, the connection dropping in every minute.
First I think I have a problem with VMware and card compatibility. Therefore I made temporary a direct communication between the two VMware host via the 10Gigabit port, but in this way the communication was stable.

Here is the NAS errors:

pciconf -lv
bxe0@pci0:2:0:0: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet
bxe1@pci0:2:0:1: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet

dmesg
bxe0: link_vars phy_flags : 1
bxe1: link_vars phy_flags : 1
bxe1:
bxe1: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link_vars phy_flags : 1
bxe0:
bxe0: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe1: link_vars phy_flags : 5
bxe1: link state changed to DOWN
bxe0: link state changed to DOWN

VMware log:

2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3702: [vmnic4] Pausing TX Queues On Link down
2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3717: [vmnic4] NIC Link is Down
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3737: [vmnic4] Resuming TX Queues On Link up
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3777: [vmnic4] NIC Link is Up, 10000 Mb/s full duplex, Flow control: ON - receive & transmit
2018-04-18T10:42:54.664Z cpu6:2097292)NetqueueBal: 5030: vmnic4: device Up notification, reset logical space needed
2018-04-18T10:42:54.664Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.664Z cpu16:2107634)NetSched: 654: vmnic4-0-tx: worldID = 2107634 exits
2018-04-18T10:42:54.664Z cpu3:2107635)NetSched: 654: vmnic4-1-tx: worldID = 2107635 exits
2018-04-18T10:42:54.664Z cpu14:2107636)NetSched: 654: vmnic4-2-tx: worldID = 2107636 exits
2018-04-18T10:42:54.664Z cpu7:2107638)NetSched: 654: vmnic4-4-tx: worldID = 2107638 exits
2018-04-18T10:42:54.664Z cpu31:2107639)NetSched: 654: vmnic4-5-tx: worldID = 2107639 exits
2018-04-18T10:42:54.664Z cpu16:2107637)NetSched: 654: vmnic4-3-tx: worldID = 2107637 exits
2018-04-18T10:42:54.664Z cpu3:2107640)NetSched: 654: vmnic4-6-tx: worldID = 2107640 exits
2018-04-18T10:42:54.664Z cpu4:2107641)NetSched: 654: vmnic4-7-tx: worldID = 2107641 exits
2018-04-18T10:42:54.665Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.707Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 1 in kernel
2018-04-18T10:42:54.710Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 2 in kernel
2018-04-18T10:42:54.713Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 3 in kernel
2018-04-18T10:42:54.716Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 4 in kernel
2018-04-18T10:42:54.719Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 5 in kernel
2018-04-18T10:42:54.722Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 6 in kernel
2018-04-18T10:42:54.725Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 7 in kernel
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 537: Driver claims supporting 7 RX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 533: Driver claims supporting 7 TX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.728Z cpu22:2107668)NetSched: 654: vmnic4-0-tx: worldID = 2107668 exits
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.728Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25024
2018-04-18T10:42:54.729Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25056
2018-04-18T10:42:54.731Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25088
2018-04-18T10:42:54.732Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25120
2018-04-18T10:42:54.734Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25152
2018-04-18T10:42:54.735Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25184
2018-04-18T10:42:54.737Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25216

Could you please somebody can help for me to debug or solve this issue?
Maybe the FreeNAS not supporting this device?

Many thanks!

 

[Chris Moore]

In my signature is a button called " useful links " and in there is a primer on 10G network for FreeNAS. The model recommend is (if I recall correctly) an Intel or Chelsio. I can't remember the model number. Broadcom is not a recommendation because of driver support.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[sirszabo]

Thanks for your answer Chris!
Previously I don't know about the Broadcom "gap" ...
Because I bought 3pcs. of this card I would like to use it. (I hope it is possible)
The FreeBSD supporting the bxe drivers (as I know) since the version 10. In this way the FreeNAS 11 need to support also.

Right now I checked deeper a bit the FreeNAS regarding the driver and firmware version.

Port1:
dev.bxe.0.pci_link: 5.0GT/s x8
dev.bxe.0.mac_addr: 2c:4d:54:45:99:27
dev.bxe.0.mf_mode: Single
dev.bxe.0.fw_version: 7.13.1.0
dev.bxe.0.bc_version: 7.13.0
dev.bxe.0.mf_vnics: 1
dev.bxe.0.version: 1.78.90
dev.bxe.0.%parent: pci2
dev.bxe.0.%pnpinfo: vendor=0x14e4 device=0x16a1 subvendor=0x1043 subdevice=0x866e class=0x020000
dev.bxe.0.%location: slot=0 function=0 dbsf=pci0:2:0:0
dev.bxe.0.%driver: bxe
dev.bxe.0.%desc: QLogic NetXtreme II BCM57840 4x10GbE (B1) BXE v:1.78.90

Port2:
dev.bxe.1.pci_link: 5.0GT/s x8
dev.bxe.1.mac_addr: 2c:4d:54:45:99:29
dev.bxe.1.mf_mode: Single
dev.bxe.1.fw_version: 7.13.1.0
dev.bxe.1.bc_version: 7.13.0
dev.bxe.1.mf_vnics: 1
dev.bxe.1.version: 1.78.90
dev.bxe.1.%parent: pci2
dev.bxe.1.%pnpinfo: vendor=0x14e4 device=0x16a1 subvendor=0x1043 subdevice=0x866e class=0x020000
dev.bxe.1.%location: slot=0 function=1 dbsf=pci0:2:0:1
dev.bxe.1.%driver: bxe
dev.bxe.1.%desc: QLogic NetXtreme II BCM57840 4x10GbE (B1) BXE v:1.78.90

On the VMware side also checked the driver:

Driver Info:
Bus Info: 0000:08:00:0
Driver: qfle3
Firmware Version: FW: 7.13.3.0 BC: 7.13.0
Version: 1.0.60.0

The all three cards has been arrived with firmware version 7.12.81.
I think the driver miss match cause this communication error.

 

[sirszabo]

The problem caused by a hardware error. The card has been replaced to a new one, and everything works now!

 

[jdbrant]

The problem caused by a hardware error. The card has been replaced to a new one, and everything works now!

Curious if you replaced it with the same model adapter? or did you change to an adapter with a different chip?

 

[sirszabo]

Hey jdbrant,

It is replaced to the same model! Since ~1 year ago I don't have any issues with this card!
If you need any evidence, or need any help from my side pls. let me know!

p.s.: NAS server is a Supermicro one, and the two VMware 6.5 server is installed on the HPE DL380 Gen9.

 

[jdbrant]

Hey jdbrant,

It is replaced to the same model! Since ~1 year ago I don't have any issues with this card!
If you need any evidence, or need any help from my side pls. let me know!

p.s.: NAS server is a Supermicro one, and the two VMware 6.5 server is installed on the HPE DL380 Gen9.

I was curious because we have been having the same kind of issue however we have the 10gb broadcom card installed in a freenas unit running on the DL380 Gen9.

For the mean time we have moved the connectivity to the build in adapters in lagg format using two of the nics. I think we may just bite the bullet and order a couple of chelsio 10gb cards as we have 2 nas units one primary and one secondary at a dr site. The primary one would drop the network connection and then any files written during the drop would just be gone. Either way it caused me a lot of grief. Thanks for getting back to me so quickly though its much appreciated.

 

[jdbrant]

Here's one for yah. Turns out it isnt even the network adapter. Server is randomly resetting itself so it just reboots at random. Trying a firmware update this evening to see if it resolves the issue from there is a support call to hp.