Link state changed to UP / DOWN continuously

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
I continuously getting the the message that the Lin state has changed to UP and then DOWN and so forth.

Code:
Apr 21 18:34:23 nas kernel: bxe0: link state changed to UP
Apr 21 19:07:23 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:07:23 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:07:28 nas kernel: bxe0: link state changed to UP
Apr 21 19:07:28 nas kernel: bxe0: link state changed to UP
Apr 21 19:22:04 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:22:04 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:22:04 nas kernel: bxe0: link state changed to UP
Apr 21 19:22:04 nas kernel: bxe0: link state changed to UP
Apr 21 19:26:51 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:26:51 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:26:51 nas kernel: bxe0: link state changed to UP
Apr 21 19:26:51 nas kernel: bxe0: link state changed to UP
Apr 21 19:28:27 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:28:27 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:28:27 nas kernel: bxe0: link state changed to UP
Apr 21 19:28:27 nas kernel: bxe0: link state changed to UP
Apr 21 19:52:04 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:52:04 nas kernel: bxe0: link state changed to DOWN
Apr 21 19:52:04 nas kernel: bxe0: link state changed to UP
Apr 21 19:52:04 nas kernel: bxe0: link state changed to UP
Apr 21 20:05:32 nas kernel: bxe0: link state changed to DOWN
Apr 21 20:05:32 nas kernel: bxe0: link state changed to DO


I have tried to change the cable but that doesn't make any difference.

I'm running FreeNAS-11.3-U2 on the bare metal of a Dell r720XD (no virtualisation).

Here is my network setup:
Code:
nas% ifconfig -a
bxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: 10G1
  options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 90:b1:1c:0a:93:fc
    hwaddr 90:b1:1c:0a:93:fc
    inet 192.168.66.21 netmask 0xffffff00 broadcast 192.168.66.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
bxe1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 90:b1:1c:0a:93:fe
    hwaddr 90:b1:1c:0a:93:fe
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (none)
    status: no carrier
bxe2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 90:b1:1c:0a:94:00
    hwaddr 90:b1:1c:0a:94:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (none)
    status: no carrier
bxe3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 90:b1:1c:0a:94:02
    hwaddr 90:b1:1c:0a:94:02
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (none)
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
nas%


I only have one of my network cards (bxe0) enabled and no Jumbo frames.

I can see there are others that have had similar issues but I don't see that they got the problem fixed. Any help or directions to a solution is highly appreciated.
 

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
Just to add more detail to the thread. Now the output in the console looks like this:
Code:
Apr 27 04:51:53 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 04:51:53 nas kernel: bxe0: link state changed to UP
Apr 27 04:51:53 nas kernel: bxe0: link state changed to UP
Apr 27 05:02:09 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:02:09 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:02:09 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 05:02:09 nas kernel: bxe0: link state changed to UP
Apr 27 05:02:09 nas kernel: bxe0: link state changed to UP
Apr 27 05:30:03 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:30:03 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:30:03 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 05:30:03 nas kernel: bxe0: link state changed to UP
Apr 27 05:30:03 nas kernel: bxe0: link state changed to UP
Apr 27 05:44:59 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:44:59 nas kernel: bxe0: link state changed to DOWN
Apr 27 05:44:59 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 05:44:59 nas kernel: bxe0: link state changed to UP
Apr 27 05:44:59 nas kernel: bxe0: link state changed to UP
Apr 27 06:11:37 nas kernel: bxe0: link state changed to DOWN
Apr 27 06:11:37 nas kernel: bxe0: link state changed to DOWN
Apr 27 06:11:37 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 06:11:37 nas kernel: bxe0: link state changed to UP
Apr 27 06:11:37 nas kernel: bxe0: link state changed to UP
Apr 27 07:01:19 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:01:19 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:01:19 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 07:01:19 nas kernel: bxe0: link state changed to UP
Apr 27 07:01:19 nas kernel: bxe0: link state changed to UP
Apr 27 07:25:34 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:25:34 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:25:34 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 07:25:34 nas kernel: bxe0: link state changed to UP
Apr 27 07:25:34 nas kernel: bxe0: link state changed to UP
Apr 27 07:28:46 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:28:46 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:28:46 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 07:28:46 nas kernel: bxe0: link state changed to UP
Apr 27 07:28:46 nas kernel: bxe0: link state changed to UP
Apr 27 07:33:41 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:33:41 nas kernel: bxe0: link state changed to DOWN
Apr 27 07:33:41 nas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Apr 27 07:33:41 nas kernel: bxe0: link state changed to UP
Apr 27 07:33:41 nas kernel: bxe0: link state changed to UP


If you need more information please let me know. I will do my best to provide it to you.

All help is highly appreciated. Thanks in advance.
 
Joined
Jan 4, 2014
Messages
1,644
H/w details of the NIC in the Dell r720XD?
 

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
Ah sorry I forgot.
It is the Dell Broadcom 57800 SFP+ (network daughter card).
 
Joined
Jan 4, 2014
Messages
1,644
Joined
Jan 4, 2014
Messages
1,644

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
Yes, I just ran Dell's firmware check and it has the most recent firmwar released on 17th January 2020. But Dell marks the firmware I have as version 15.05.12.
 
Joined
Jan 4, 2014
Messages
1,644
And you've eliminated the cable and switch port as possible causes?
 
Last edited:

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
And you've eliminated the cable and switch port as possible causes?
Yes, I have changed the cable with another one. I can try a third cable.

I have furthermore, just updated to the lastest version of FreeNAS-11.3-U2.1. I don't know if that will fix it.
 
Joined
Jan 4, 2014
Messages
1,644
Other things to consider maybe:
  1. Bad NIC?
  2. Matching switch port and NIC characteristics rather than auto-negotiating e.g. speed (try a lower speed as well), duplex setting, etc.
 
Joined
Dec 29, 2014
Messages
1,135
If you force duplex on one side, you have to force it on both sides. Otherwise you end up with a duplex mismatch which is about as bad scenario as you can get in a LAN environment.
 

runevn

Explorer
Joined
Apr 4, 2019
Messages
63
Just to follow up.

I think I found the problem. I tried a third cable that seems to be rock stabil. However, the difference between this third cable and the two previous that I tried is that the new one is a fiber cable and the other two are DAC cables.

It seems that my Broadcom network card doesn't like my DAC cables. But what is a bit strange is that both the DAC and fiber cables uses Intel coded transceivers. Why there is a difference - I do not know, but that seems to be the problem/issue.

So the solution for me seems to either buy new DAC cables that are dual-brand coded (Broadcom - Intel) or buy a refurbished intel network card for my Dell server. Are there any good reason to use around 30$ extra to get the Intel card instead of buying new DAC cables? Is Intel cards in general more compatible with FreeBSD/Freenas?

Again, thanks for all your answers and comments - they are highly appreciated.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Avoid DAC cables they have way too many issues. Vender compatibility is never the same so they fail. Stick with getting vender specific transceivers and using normal single mode fiber.
 
Top