Intel NICs, Network Dropping/Unresponsive

[JayG30]

I'm experiencing what I believe to be network connectivity issues with a build.

Hardware

• Intel R2312GL4GS barebones server
• 128GB of RAM
• 2 x E5-2670 CPU's
• 6 x 4TB HDD on an LSI 2308 (IT firmware)
• Intel S3710 SLOG on SATA port
• 32GB USB 3.0 Sandisk Ultra Fit

Software

• FreeNAS-9.10.1-U2 (f045a8b).

This server has a Intel quad i350 NIC onboard. I've just booted it up and I the web GUI is really unresponsive and fails to load at times. Thought about it a while and decided to test the connectivity so I ran a constant ping to the server. It is frequently "timing out". I've tried each of the 4 NICS and they all have the issue. I'm going to try booting into it on another exact replica of this server to rule out a hardware issue.

I thought perhaps a driver issue for the i350, but researching online brought up that FreeBSD and FreeNAS support this NIC fine in newer releases.

Thoughts?

 

[Unlisted Person]

I'm experiencing what I believe to be network connectivity issues with a build.

Thoughts?

What does ifconfig return?

 

[JayG30]

What does ifconfig return?

It returns exactly what I expect.
The machine has 4 x 1G Intel NICs, but only 1 plugged in currently (igb3). That is set to a static IP address (have also tried setting to for DHCP). There is also a 10G SFP+ Mellanox ConnectX2 card in the server, but I'm not currently using it.

Code:

[admin@freenas] /% ifconfig
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
		ether 00:1e:67:d5:0a:3e
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
		ether 00:1e:67:d5:0a:3f
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
		ether 00:1e:67:d5:0a:40
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
		ether 00:1e:67:d5:0a:41
		inet 192.168.0.220 netmask 0xffffff00 broadcast 192.168.0.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:02:c9:4e:88:5c
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier

Static config is;
IP: 192.168.0.220
MASK: 255.255.255.0
GATEWAY: 192.168.0.2
NAMESERVER: 192.168.0.5

192.168.0.2 is a L3 Cisco switch. It serves as the gateway (for VLAN routing) with default route to router.
192.168.0.1 is a EdgeRouter Lite. It is the router and connects out to WAN. It also handles DHCP and ipsec VPN.
192.168.0.5 is DNS/AD.

Boot the machine from a live Ubuntu ISO (over KVM) and everything works fine.

 

[JayG30]

This is strange. I shut down the server last night. Come in this morning and turn it on. Everything has been running fine since.

 

[JayG30]

The issue came back and I figured out what the issue was. It didn't have anything to do with freenas.

The server was running VMWare ESXi. I move that installation (USB key) to another machine. Turns out that the management network (vmk0) didn't update the MAC address, thus causing a MAC conflict. I figured this out after looking through my network equipment trying to figure out what was wrong. The ARP table on the switch gave me the hint and then I started thinking why the MAC would have multiple IP addresses assigned to it, which made me look up issues with ESXi. The knowledge article from VMware is HERE.

Cleared out vmk0 and defaulted the management network. Now everything is working again.

The reason I didn't see it happening all the time was that sometimes I had the server I moved ESXi to off.