Dear All,
I have a problem with FreeNAS-11.1-U4 (bare metal) installation with Broadcom NetXtreme II 10 Gigabit Ethernet (BCM57840) card.
The correct model/type is -> https://www.asus.com/us/Commercial-Servers-Workstations/PEB10G578402T/
The NAS connected directly in two VMware ESXi 6.5 host, where also these cards has been installed.
When I configured the communication between NAS and the VMware hosts, the connection dropping in every minute.
First I think I have a problem with VMware and card compatibility. Therefore I made temporary a direct communication between the two VMware host via the 10Gigabit port, but in this way the communication was stable.
Here is the NAS errors:
pciconf -lv
bxe0@pci0:2:0:0: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet
bxe1@pci0:2:0:1: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet
dmesg
bxe0: link_vars phy_flags : 1
bxe1: link_vars phy_flags : 1
bxe1:
bxe1: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link_vars phy_flags : 1
bxe0:
bxe0: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe1: link_vars phy_flags : 5
bxe1: link state changed to DOWN
bxe0: link state changed to DOWN
VMware log:
2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3702: [vmnic4] Pausing TX Queues On Link down
2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3717: [vmnic4] NIC Link is Down
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3737: [vmnic4] Resuming TX Queues On Link up
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3777: [vmnic4] NIC Link is Up, 10000 Mb/s full duplex, Flow control: ON - receive & transmit
2018-04-18T10:42:54.664Z cpu6:2097292)NetqueueBal: 5030: vmnic4: device Up notification, reset logical space needed
2018-04-18T10:42:54.664Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.664Z cpu16:2107634)NetSched: 654: vmnic4-0-tx: worldID = 2107634 exits
2018-04-18T10:42:54.664Z cpu3:2107635)NetSched: 654: vmnic4-1-tx: worldID = 2107635 exits
2018-04-18T10:42:54.664Z cpu14:2107636)NetSched: 654: vmnic4-2-tx: worldID = 2107636 exits
2018-04-18T10:42:54.664Z cpu7:2107638)NetSched: 654: vmnic4-4-tx: worldID = 2107638 exits
2018-04-18T10:42:54.664Z cpu31:2107639)NetSched: 654: vmnic4-5-tx: worldID = 2107639 exits
2018-04-18T10:42:54.664Z cpu16:2107637)NetSched: 654: vmnic4-3-tx: worldID = 2107637 exits
2018-04-18T10:42:54.664Z cpu3:2107640)NetSched: 654: vmnic4-6-tx: worldID = 2107640 exits
2018-04-18T10:42:54.664Z cpu4:2107641)NetSched: 654: vmnic4-7-tx: worldID = 2107641 exits
2018-04-18T10:42:54.665Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.707Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 1 in kernel
2018-04-18T10:42:54.710Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 2 in kernel
2018-04-18T10:42:54.713Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 3 in kernel
2018-04-18T10:42:54.716Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 4 in kernel
2018-04-18T10:42:54.719Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 5 in kernel
2018-04-18T10:42:54.722Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 6 in kernel
2018-04-18T10:42:54.725Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 7 in kernel
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 537: Driver claims supporting 7 RX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 533: Driver claims supporting 7 TX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.728Z cpu22:2107668)NetSched: 654: vmnic4-0-tx: worldID = 2107668 exits
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.728Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25024
2018-04-18T10:42:54.729Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25056
2018-04-18T10:42:54.731Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25088
2018-04-18T10:42:54.732Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25120
2018-04-18T10:42:54.734Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25152
2018-04-18T10:42:54.735Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25184
2018-04-18T10:42:54.737Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25216
Could you please somebody can help for me to debug or solve this issue?
Maybe the FreeNAS not supporting this device?
Many thanks!
I have a problem with FreeNAS-11.1-U4 (bare metal) installation with Broadcom NetXtreme II 10 Gigabit Ethernet (BCM57840) card.
The correct model/type is -> https://www.asus.com/us/Commercial-Servers-Workstations/PEB10G578402T/
The NAS connected directly in two VMware ESXi 6.5 host, where also these cards has been installed.
When I configured the communication between NAS and the VMware hosts, the connection dropping in every minute.
First I think I have a problem with VMware and card compatibility. Therefore I made temporary a direct communication between the two VMware host via the 10Gigabit port, but in this way the communication was stable.
Here is the NAS errors:
pciconf -lv
bxe0@pci0:2:0:0: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet
bxe1@pci0:2:0:1: class=0x020000 card=0x866e1043 chip=0x16a114e4 rev=0x11 hdr=0x00
vendor = 'Broadcom Limited'
device = 'BCM57840 NetXtreme II 10 Gigabit Ethernet'
class = network
subclass = ethernet
dmesg
bxe0: link_vars phy_flags : 1
bxe1: link_vars phy_flags : 1
bxe1:
bxe1: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link_vars phy_flags : 1
bxe0:
bxe0: link state changed to UP
NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe1: link_vars phy_flags : 5
bxe1: link state changed to DOWN
bxe0: link state changed to DOWN
VMware log:
2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3702: [vmnic4] Pausing TX Queues On Link down
2018-04-18T10:42:41.433Z cpu24:2097869)qfle3: qfle3_report_link:3717: [vmnic4] NIC Link is Down
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3737: [vmnic4] Resuming TX Queues On Link up
2018-04-18T10:42:51.732Z cpu24:2097869)qfle3: qfle3_report_link:3777: [vmnic4] NIC Link is Up, 10000 Mb/s full duplex, Flow control: ON - receive & transmit
2018-04-18T10:42:54.664Z cpu6:2097292)NetqueueBal: 5030: vmnic4: device Up notification, reset logical space needed
2018-04-18T10:42:54.664Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.664Z cpu16:2107634)NetSched: 654: vmnic4-0-tx: worldID = 2107634 exits
2018-04-18T10:42:54.664Z cpu3:2107635)NetSched: 654: vmnic4-1-tx: worldID = 2107635 exits
2018-04-18T10:42:54.664Z cpu14:2107636)NetSched: 654: vmnic4-2-tx: worldID = 2107636 exits
2018-04-18T10:42:54.664Z cpu7:2107638)NetSched: 654: vmnic4-4-tx: worldID = 2107638 exits
2018-04-18T10:42:54.664Z cpu31:2107639)NetSched: 654: vmnic4-5-tx: worldID = 2107639 exits
2018-04-18T10:42:54.664Z cpu16:2107637)NetSched: 654: vmnic4-3-tx: worldID = 2107637 exits
2018-04-18T10:42:54.664Z cpu3:2107640)NetSched: 654: vmnic4-6-tx: worldID = 2107640 exits
2018-04-18T10:42:54.664Z cpu4:2107641)NetSched: 654: vmnic4-7-tx: worldID = 2107641 exits
2018-04-18T10:42:54.665Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.707Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 1 in kernel
2018-04-18T10:42:54.710Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 2 in kernel
2018-04-18T10:42:54.713Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 3 in kernel
2018-04-18T10:42:54.716Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 4 in kernel
2018-04-18T10:42:54.719Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 5 in kernel
2018-04-18T10:42:54.722Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 6 in kernel
2018-04-18T10:42:54.725Z cpu6:2097292)WARNING: qfle3: qfle3_tq_stop:921: [vmnic4] Stopping TX queue 7 in kernel
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 537: Driver claims supporting 7 RX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 533: Driver claims supporting 7 TX queues, and 7 queues are accepted.
2018-04-18T10:42:54.728Z cpu6:2097292)NetPort: 1580: disabled port 0x6000002
2018-04-18T10:42:54.728Z cpu22:2107668)NetSched: 654: vmnic4-0-tx: worldID = 2107668 exits
2018-04-18T10:42:54.728Z cpu6:2097292)Uplink: 11680: enabled port 0x6000002 with mac 2c:4d:54:45:9a:8f
2018-04-18T10:42:54.728Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25024
2018-04-18T10:42:54.729Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25056
2018-04-18T10:42:54.731Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25088
2018-04-18T10:42:54.732Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25120
2018-04-18T10:42:54.734Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25152
2018-04-18T10:42:54.735Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25184
2018-04-18T10:42:54.737Z cpu6:2097292)WARNING: qfle3: qfle3_rx_ustorm_prods_offset:54: [vmnic4] rx prods offset 25216
Could you please somebody can help for me to debug or solve this issue?
Maybe the FreeNAS not supporting this device?
Many thanks!