Link state changes frequently

[Jaesii]

I've been seeing the following quite a lot show up in my logs and in my email. Any ideas what would cause it? I don't see any issues accessing the iSCSI disk in vmware.

Code:

Nov 19 09:50:35 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:35 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:35 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:35 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:35 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:35 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:35 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:35 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:37 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:37 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:37 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:37 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:38 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:38 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:38 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:38 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:43 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:43 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:43 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:43 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:44 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:44 freenas kernel: ix0: link state changed to DOWN
Nov 19 09:50:44 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:44 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:44 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:44 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:44 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:44 freenas kernel: ix0: link state changed to UP
Nov 19 09:50:44 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:44 freenas kernel: ix1: link state changed to DOWN
Nov 19 09:50:45 freenas kernel: ix1: link state changed to UP
Nov 19 09:50:45 freenas kernel: ix1: link state changed to UP
Nov 19 10:04:57 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:04:57 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:04:57 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:04:57 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:04:58 freenas kernel: ix0: link state changed to UP
Nov 19 10:04:58 freenas kernel: ix0: link state changed to UP
Nov 19 10:04:58 freenas kernel: ix1: link state changed to UP
Nov 19 10:04:58 freenas kernel: ix1: link state changed to UP
Nov 19 10:05:00 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:05:00 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:05:00 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:05:00 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:05:00 freenas kernel: ix0: link state changed to UP
Nov 19 10:05:00 freenas kernel: ix0: link state changed to UP
Nov 19 10:05:00 freenas kernel: ix1: link state changed to UP
Nov 19 10:05:00 freenas kernel: ix1: link state changed to UP
Nov 19 10:05:01 freenas WARNING: 172.16.2.2 (iqn.1998-01.com.vmware:57b61850-f648-97d7-adb1-90e2ba3a7270-72d9de9e): no ping reply (NOP-Out) after 5 seconds; dropping connection
Nov 19 10:05:02 freenas WARNING: 172.16.1.2 (iqn.1998-01.com.vmware:57b61850-f648-97d7-adb1-90e2ba3a7270-72d9de9e): no ping reply (NOP-Out) after 5 seconds; dropping connection
Nov 19 10:05:29 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:05:29 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:05:29 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:05:29 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:05:29 freenas kernel: ix0: link state changed to UP
Nov 19 10:05:29 freenas kernel: ix0: link state changed to UP
Nov 19 10:05:29 freenas kernel: ix1: link state changed to UP
Nov 19 10:05:29 freenas kernel: ix1: link state changed to UP
Nov 19 10:13:54 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:13:54 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:13:54 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:13:54 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:13:54 freenas kernel: ix0: link state changed to UP
Nov 19 10:13:54 freenas kernel: ix0: link state changed to UP
Nov 19 10:13:54 freenas kernel: ix1: link state changed to UP
Nov 19 10:13:54 freenas kernel: ix1: link state changed to UP
Nov 19 10:14:23 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:14:23 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:14:23 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:14:23 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:14:23 freenas kernel: ix1: link state changed to UP
Nov 19 10:14:23 freenas kernel: ix1: link state changed to UP
Nov 19 10:14:24 freenas kernel: ix0: link state changed to UP
Nov 19 10:14:24 freenas kernel: ix0: link state changed to UP
Nov 19 10:18:54 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:18:54 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:18:54 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:18:54 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:18:54 freenas kernel: ix0: link state changed to UP
Nov 19 10:18:54 freenas kernel: ix0: link state changed to UP
Nov 19 10:18:54 freenas kernel: ix1: link state changed to UP
Nov 19 10:18:54 freenas kernel: ix1: link state changed to UP
Nov 19 10:18:56 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:18:56 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:18:56 freenas kernel: ix0: link state changed to UP
Nov 19 10:18:56 freenas kernel: ix0: link state changed to UP
Nov 19 10:18:56 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:18:56 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:18:56 freenas kernel: ix1: link state changed to UP
Nov 19 10:18:56 freenas kernel: ix1: link state changed to UP
Nov 19 10:19:02 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:19:02 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:19:02 freenas kernel: ix0: link state changed to UP
Nov 19 10:19:02 freenas kernel: ix0: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix0: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix0: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix0: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix1: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix1: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix1: link state changed to DOWN
Nov 19 10:19:03 freenas kernel: ix1: link state changed to UP
Nov 19 10:19:03 freenas kernel: ix1: link state changed to UP

 

[joeschmuck]

I know this sounds like a stupid question but what version of FreeNAS are you running and are you running it in VMWare? Also, did this recently start up after a software update or any hardware changes? I see you have 2 NICs, maybe if you could explain a bit about how those are configured to work?

I'm sure you have done this already but have you Powered Off, waiting 5 seconds, Powered On? Not a reboot but a complete power off.

 

[Jaesii]

im running 9.10.1-u4, and it is installed on bare metal.

I actually have 5 nics in this system.

two onboard 1GB
Two 10GB nics on a single card and a 3rd 10GB nic on another card

the two onboard are not plugged in. the dual port 10GB cards are connected via twinax cable to my HP esxi server. The 3rd 10GB card is connected via twinax to my switch. ix0 and ix1 are nics from the dual port card.

I've never noticed a drop on vmware side.

I've rebooted the FreeNAS server several times today for other reasons.

I dont always see the interfaces flap, its pretty random, but when it happens its usually in batches like the above.

The only recent hardware change is the addition of the 3rd 10GB card, but that shows up to the system as mlxen0 since it is a different brand of adapter. I installed that card maybe early this month.

 

[pirateghost]

How are the 2 10gb NICs configured?

 

[Jaesii]

 

[pirateghost]

Have you tried without jumbo frames? Those can cause issues