SOLVED Network status: no carrier after reboot in X11SPH-nCTF

[ilmarmors]

Version: FreeNAS-11.3-U5 (upgrade ot TrueNAS-12.0-U1 didn't help)
Chassis: SuperStorage 5029P-E1CTR12L
Motherboard: X11SPH-nCTF
Dual LAN with 10GBase-T with Intel® X722 + X557

Server has 3 network ports on motherboard - (two 10GBase-T and one 1000Base-T for IPMI, all ports connected to the same switch with 1 Gigabit Ethernet ports.

Initial setup of FreeNAS-11.3-U4 was successfully running since july. After upgrading to FreeNAS-11.3-U5 everything was fine. There were more than one reboot before, but then after one reboot, one port started to have status: no carrier. I thought that it is coincidence (didn't visited place in person yet).

ixl0 that went dark was actively used (~1TB per day upload).

ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ixl0
options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 3c:ec:ef:0c:f6:38
hwaddr 3c:ec:ef:0c:f6:38
inet 192.168.103.101 netmask 0xffffff00 broadcast 192.168.103.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier

ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ixl1
options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 3c:ec:ef:0c:f6:39
hwaddr 3c:ec:ef:0c:f6:39
inet 192.168.103.100 netmask 0xffffff00 broadcast 192.168.103.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

ixl1 was reconfigured to overtake ixl0 traffic. Then after another 1.5 month I did another server reboot. And now second port started to have status: no carrier.

IPMI port is still fine. I upgraded via mounted CD to TrueNAS 12.0-U1, but it didn't help. BIOS upgraded to latest 3.4.

Anybody has any ideas why ports went into status "no carrier" after reboots and what I could try remotely as I might not be able to visit hosting faciliy in person due to Christmas holidays? Were reboots and Coincidence? With coming Christmas I might not be able to visit hosting facility in person. Is reboots just coincidence? Anybody has experienced such unexpected things? Could it be that server knocked out switch ports or switch knocked out server ports?

 

[ilmarmors]

It turned out to be some esotheric issue most probably with autonegotiation between server with 10GBase-T and router/switch with 1000Base-T SPF+ module. Pulling out network cables from server and plugin into 1000Base-T port on laptop brought lights back to the switch ports. When plugging in those network cables back to the server 10GBase-T ports, everything worked again. Ended up replacing SPF+ modules in switch with 10GBase-T ones.