SOLVED NIC issues

[dffvb]

Hi there, after my SCLAE went for a couple of weeks without any hick ups, it now frequently disconnects the NIC (Intel X550) , I am not fully sure, but I could be caused by the latest update.... var/log/messages gives the following

May 23 08:34:36 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:34:41 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:35:04 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:35:09 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:35:35 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:35:45 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:36:53 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:37:07 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:37:21 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:37:47 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 1 Gbps, Flow Control: RX/TX
May 23 08:39:10 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:39:14 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:40:08 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:40:13 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:41:09 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:41:13 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:41:39 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:41:44 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:42:53 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:42:58 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:43:28 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:43:33 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:44:33 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:44:37 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:46:17 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:46:22 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:49:13 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:49:18 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 08:49:42 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 08:49:47 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 09:00:17 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 09:00:26 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 09:08:24 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 09:08:28 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 09:10:25 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 09:10:30 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 09:15:20 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 09:15:24 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 09:21:43 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
May 23 09:21:46 truenas kernel: ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 23 10:03:48 truenas kernel: show_signal_msg: 5 callbacks suppressed


Unfortunately thats not the only problem, NIC also sometimes decides to go down to 100 Mbit speeds... I tried using a cron job and setting speeds manually to 10 Gbit with ethtool, but this made things worse...

Any ideas?

 

[NugentS]

That looks like a hardware issue to me. Check / replace the cabling and check the port its plugged into on the other end

 

[dffvb]

To my great surprise it was indeed the (quite expensive) cabling. Wouldnt have suspected these little boys.... Thanks for the help

 

[NugentS]

 

[RX9K]

The server didnt move, nothing to my knowledge bumped it... and yet this was the issue for me as well... somewhere in-between enraged and relieved... thanks for reminding me of what I should have checked first.