10G NIC loses connectivity after reboot

[xyzzy]

I have an 11.0-U3 system that I just finished setting up. The motherboard is a Supermicro X9SRE-F with two onboard 1G NICs (one is connected to the LAN, the other is not connected). The system also has an Intel X710-DA4 which I've connected via an SFP+ twinax cable directly to an ESXi box with another X710-DA4.

Everything including iSCSI was working great. Then I shut down both boxes (ESXi then FreeNAS) and powered them both back up (FreeNAS then ESXi). Since then, I can't get the FreeNAS box to send or receive traffic on the previously working X710-DA4 port. The onboard 1G NIC still works fine. I've used a Windows box with an X710-DA4 (temporarily replacing the other 2 boxes one at a time) to verify that the ESXi box can send/receive pings but the FreeNAS box cannot.

On the FreeNAS box, I've verified the interface link is up. Just for yucks, I ran tcpdump -i ixl0 on the FreeNAS box while it was attempting to ping one of the other boxes. I noticed that the box is sending ARP requests to get the destination system's MAC address but never receiving a response.

I haven't explicitly turned on any firewalls on the FreeNAS box and don't recall FreeNAS having one on by default.

Any ideas what's going on here?

 

[Stux]

Hardware fault?

 

[xyzzy]

Hardware fault?

I guess that's certainly possible but I'd first like to rule out any software or config related causes.

 

[xyzzy]

I think I've narrowed it down a bit.

If I go into FreeNAS > Network > Interfaces and delete the interface and then re-add it, it works fine. If I reboot the FreeNAS box, it stops working again ... until I delete and re-add the interface in FreeNAS.

In case it helps, the X710-DA4 is using firmware 5.05.

 

[Stux]

I guess that's certainly possible but I'd first like to rule out any software or config related causes.

I only mention it because I've just received a replacement for a new X550-T2 that I had to RMA because one of the ports was faulty.

 

[xyzzy]

I only mention it because I've just received a replacement for a new X550-T2 that I had to RMA because one of the ports was faulty.

Ah, good to know. Did the faulty port completely fail (i.e., no link light, no "up" interface) or something more subtle?

 

[xyzzy]

For what its worth, here's what the interface and routing table look like after a reboot:

Code:

root@freenas:~ # ifconfig ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
		options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 3c:fd:fe:a0:22:00
		inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
		status: active

root@freenas:~ # netstat -r -4
Routing tables

Internet:
Destination		Gateway			Flags	 Netif Expire
default			192.168.10.1	   UGS		igb0
localhost		  link#7			 UH		  lo0
192.168.10.0/24	link#5			 U		  igb0
192.168.10.22	  link#5			 UHS		 lo0
192.168.20.0/24	link#1			 U		  ixl0
192.168.20.2	   link#1			 UHS		 lo0
192.168.21.0/24	link#2			 U		  ixl1
192.168.21.2	   link#2			 UHS		 lo0

root@freenas:~ # ping 192.168.20.4
PING 192.168.20.4 (192.168.20.4): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down

And here are the same commands after I delete and re-create the interface in the FreeNAS GUI:

Code:

root@freenas:~ # ifconfig ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
		options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 3c:fd:fe:a0:22:00
		inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
		status: active

root@freenas:~ # netstat -r -4
Routing tables

Internet:
Destination		Gateway			Flags	 Netif Expire
default			192.168.10.1	   UGS		igb0
localhost		  link#7			 UH		  lo0
192.168.10.0/24	link#5			 U		  igb0
192.168.10.22	  link#5			 UHS		 lo0
192.168.20.0/24	link#1			 U		  ixl0
192.168.20.2	   link#1			 UHS		 lo0
192.168.21.0/24	link#2			 U		  ixl1
192.168.21.2	   link#2			 UHS		 lo0

root@freenas:~ # ping 192.168.20.4
PING 192.168.20.4 (192.168.20.4): 56 data bytes
64 bytes from 192.168.20.4: icmp_seq=0 ttl=128 time=0.155 ms
64 bytes from 192.168.20.4: icmp_seq=1 ttl=128 time=0.086 ms
^C

Note that the ifconfig and netstat output are identical both before and after. This tells me that either there's another software component involved or there's some weird interaction between the device & driver at boot time that's reset by doing the delete-then-recreate.

 

[Stux]

It wouldn't train at 10G and took 3 minutes to get link at 1G.

It seemed to be deteriorating. On inspection after pulling, there was noticeable 'corrosion' or something on some smt components near the affected port.

 

[xyzzy]

It wouldn't train at 10G and took 3 minutes to get link at 1G.

It seemed to be deteriorating. On inspection after pulling, there was noticeable 'corrosion' or something on some smt components near the affected port.

Yikes. That doesn't sound like fun.

 

[bigphil]

What driver is loaded for the X710 in FreeNAS?

sysctl -a | grep "dev.ixl.0.%desc"

 

[xyzzy]

What driver is loaded for the X710 in FreeNAS?

sysctl -a | grep "dev.ixl.0.%desc"

Code:

root@freenas:~ # sysctl -a | grep "dev.ixl.0.%desc"
dev.ixl.0.%desc: Intel(R) Ethernet Connection XL710/X722 Driver, Version - 1.6.6-k

 

[bigphil]

Pretty old driver. The latest is 1.7.12.1 and NVM image is now 6.01. I've uploaded the latest driver here. It's built against FreeNAS 9.10, so not sure if it will load for 11 but its easy to try. If it doesn't, I can build for 11.

 

[xyzzy]

Pretty old driver. The latest is 1.7.12.1 and NVM image is now 6.01. I've uploaded the latest driver here. It's built against FreeNAS 9.10, so not sure if it will load for 11 but its easy to try. If it doesn't, I can build for 11.

Thanks! I'm trying to update the NVM image but trying to use the Intel EFI tool to do it. (I'd like to update all of my X710's the same way without having to move them to a test rig.)

For some reason, the Intel zip file splits into two folders - EFIx64 and EFI2x64. They have identical contents except the actual prog (nvmupdate64e.efi) that gets run to do the update. Any idea what the difference is? I'd hate to pick the wrong one.

 

[bigphil]

I have no idea what the difference is. I don't think it's gonna brick anything though. I'd try the EFIx64 version and see if it works. You could also use the FreeBSD version for the FreeNAS box and the ESXi version for the VMware box.

 

[xyzzy]

I have no idea what the difference is. I don't think it's gonna brick anything though. I'd try the EFIx64 version and see if it works. You could also use the FreeBSD version for the FreeNAS box and the ESXi version for the VMware box.

I think I'll play it safe and use the FreeBSD version. I'm going to try installing your driver first and will let you know how it goes. Thanks again for your help!

 

[xyzzy]

@bigphil -- It looks like it didn't work. I saw this at the beginning of the boot:

Code:

KLD file if_ixl_17121.ko is missing dependencies

And after the system booted, I saw it was still using the baked-in driver:

Code:

root@ganymede:~ # sysctl -a | grep "dev.ixl.0.%desc"
dev.ixl.0.%desc: Intel(R) Ethernet Connection XL710/X722 Driver, Version - 1.6.6-k

If the offer still holds, I'd like to take you up on your offer of building a FreeNAS 11 version.

 

[bigphil]

Yeah, give me a few and I'll build one up.

 

[xyzzy]

Yeah, give me a few and I'll build one up.

Thanks!!!

 

[bigphil]

Ok, here is the driver for 11. Let me know if it loads up properly or not.

 

[xyzzy]

Ok, here is the driver for 11. Let me know if it loads up properly or not.

Thanks!!! I just loaded it and it looks like it worked.

Code:

root@freenas:~ # sysctl -a | grep "dev.ixl.0.%desc"
dev.ixl.0.%desc: Intel(R) Ethernet Connection XL710/X722 Driver, Version - 1.7.12.1

What's even better is that after the reboot, the interface came up normally. I didn't need to remove-then-recreate to be able to ping the box on the other end of the X710 connection.

I assume I should still proceed with updating the NVM to match the driver, right?