10gbe Intel X540-T2 stopping all transmission intermittently?

[Jonathan Hughes]

I'm having an issue where i recently installed a x540-t2 into a new freenas box. I have both ports of that connected to a matching x540-t2 on the hyper-v server (no switch). I've assigned static IP's for each interface and there is connectivity between the two machines using ms iscsi. the problem arises every so often where the NIC's just stop transmitting and receiving any data. I notice this on the reporting page of the freenas box - the interfaces traffic levels just drop to 0.

The errors i get on the windows side are mostly these:

Event ID 153: The IO operation at logical block address 0x40918 for Disk 10 (PDO name: \Device\0000006a) was retried.
Event ID 51: An error was detected on device \Device\Harddisk12\DR12 during a paging operation.
Event ID 140: The system failed to flush data to the transaction log. Corruption may occur in VolumeId: M:, DeviceName: \Device\HarddiskVolume27.
(A device which does not exist was specified.)
Event ID 1: Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data.
Event ID 63: Can not Reset the Target or LUN. Will attempt session recovery.

There are more but i'm hoping this is enough to help someone help me get down the road of troubleshooting this. I'm kind of out of ideas of things to try at this point. Any additional questions or help would be much appreciated!

Freenas Box:
Build FreeNAS-9.10.2-U1 (86c7ef5)
Platform Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
Motherboard ASRock Z97 EXTREME4 ATX LGA1150
Memory 32429MB
Single X540-T2 for 10gbe to Hyper-v
ix0: 100.100.100.2/24
ix1: 100.100.200.2/24

Hyper-V box:
HP DL-360 Gen 9
Windows Server 2012
Single X540-T2 for 10gbe to NAS
Port 1: 100.100.100.1/24
Port 2: 100.100.200.1/24


Logs from messages (changed machine name to hyper-vServer.omitted.com):

Jan 30 08:05:16 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:05:46 hv4nas1 (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 34 73 00 00 02 28 00 00
Jan 30 08:05:46 hv4nas1 (0:3:0/0): Tag: 0x43e9c302, type 1
Jan 30 08:05:46 hv4nas1 (0:3:0/0): ctl_process_done: 420 seconds
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:29 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:25 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 73 tasks
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 164 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:44 hv4nas1 connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:07 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:35 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:57 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:03 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:31 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks

 

[SweetAndLow]

Freenas version, hardware specs, client specs, client os version. Please read the forum rules and provide some more information.

Sent from my Nexus 5X using Tapatalk

 

[bigphil]

Using CAT6 or higher cables? Tried different cables? Tried using only one interface? Latest firmware on the NIC's? Do you have the ability to test 1Gbe interfaces to narrow it down to issues with the 10Gbe cards?

 

[Jonathan Hughes]

Using CAT6 or higher cables? Tried different cables? Tried using only one interface? Latest firmware on the NIC's? Do you have the ability to test 1Gbe interfaces to narrow it down to issues with the 10Gbe cards?

These cables are brand new and cat6. It's happening on both ports so i suppose it 'could' be cables but i'm thinking that's highly unlikely.
I have not tried migrating to a single port on the NIC yet. I'm currently migrating my VM's storage off of this new NAS before i do that. Will report back if that helps.
I have a quad port intel gbe coming in this week i could use for testing to see if the cards are bad.

 

[c32767a]

I'm having an issue where i recently installed a x540-t2 into a new freenas box. I have both ports of that connected to a matching x540-t2 on the hyper-v server (no switch). I've assigned static IP's for each interface and there is connectivity between the two machines using ms iscsi. the problem arises every so often where the NIC's just stop transmitting and receiving any data. I notice this on the reporting page of the freenas box - the interfaces traffic levels just drop to 0.

It would be helpful to know the IP addresses involved and specific interface configurations, as per the "getting help" doc/forum rules.
Both on the hyper V side and the FreeNAS side. Without details it's hard to help.

 

[Jonathan Hughes]

It would be helpful to know the IP addresses involved and specific interface configurations, as per the "getting help" doc/forum rules.
Both on the hyper V side and the FreeNAS side. Without details it's hard to help.

I've added this information to the start of the thread but here as well:

Freenas Box:
Build FreeNAS-9.10.2-U1 (86c7ef5)
Platform Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
Motherboard ASRock Z97 EXTREME4 ATX LGA1150
Memory 32429MB
Single X540-T2 for 10gbe to Hyper-v
ix0: 100.100.100.2/24
ix1: 100.100.200.2/24

Hyper-V box:
HP DL-360 Gen 9
Windows Server 2012
Single X540-T2 for 10gbe to NAS
Port 1: 100.100.100.1/24
Port 2: 100.100.200.1/24

 

[c32767a]

So what does netstat -sI ix0 and netstat -sI ix1 produce? can you post it with code tags?
Also netstat -s.. Also in code tags.. it'll be a lot of output..

 

[Jonathan Hughes]

Output attached - thanks again for the help.

netstat -sI ix0

Code:

ip6 on ix0:
		0 total input datagrams
		0 datagrams with invalid header received
		0 datagrams exceeded MTU received
		0 datagrams with no route received
		0 datagrams with invalid dst received
		0 datagrams with unknown proto received
		0 truncated datagrams received
		0 input datagrams discarded
		0 datagrams delivered to an upper layer protocol
		0 datagrams forwarded to this interface
		0 datagrams sent from an upper layer protocol
		0 total discarded output datagrams
		0 output datagrams fragmented
		0 output datagrams failed on fragment
		0 output datagrams succeeded on fragment
		0 incoming datagrams fragmented
		0 datagrams reassembled
		0 datagrams failed on reassembly
		0 multicast datagrams received
		0 multicast datagrams sent
icmp6 on ix0:
		0 total input messages
		0 total input error messages
		0 input destination unreachable errors
		0 input administratively prohibited errors
		0 input time exceeded errors
		0 input parameter problem errors
		0 input packet too big errors
		0 input echo requests
		0 input echo replies
		0 input router solicitations
		0 input router advertisements
		0 input neighbor solicitations
		0 input neighbor advertisements
		0 input redirects
		0 input MLD queries
		0 input MLD reports
		0 input MLD dones
		0 total output messages
		0 total output error messages
		0 output destination unreachable errors
		0 output administratively prohibited errors
		0 output time exceeded errors
		0 output parameter problem errors
		0 output packet too big errors
		0 output echo requests
		0 output echo replies
		0 output router solicitations
		0 output router advertisements
		0 output neighbor solicitations
		0 output neighbor advertisements
		0 output redirects
		0 output MLD queries
		0 output MLD reports
		0 output MLD dones

netstat -sI ix1

Code:

ip6 on ix1:
		0 total input datagrams
		0 datagrams with invalid header received
		0 datagrams exceeded MTU received
		0 datagrams with no route received
		0 datagrams with invalid dst received
		0 datagrams with unknown proto received
		0 truncated datagrams received
		0 input datagrams discarded
		0 datagrams delivered to an upper layer protocol
		0 datagrams forwarded to this interface
		0 datagrams sent from an upper layer protocol
		0 total discarded output datagrams
		0 output datagrams fragmented
		0 output datagrams failed on fragment
		0 output datagrams succeeded on fragment
		0 incoming datagrams fragmented
		0 datagrams reassembled
		0 datagrams failed on reassembly
		0 multicast datagrams received
		0 multicast datagrams sent
icmp6 on ix1:
		0 total input messages
		0 total input error messages
		0 input destination unreachable errors
		0 input administratively prohibited errors
		0 input time exceeded errors
		0 input parameter problem errors
		0 input packet too big errors
		0 input echo requests
		0 input echo replies
		0 input router solicitations
		0 input router advertisements
		0 input neighbor solicitations
		0 input neighbor advertisements
		0 input redirects
		0 input MLD queries
		0 input MLD reports
		0 input MLD dones
		0 total output messages
		0 total output error messages
		0 output destination unreachable errors
		0 output administratively prohibited errors
		0 output time exceeded errors
		0 output parameter problem errors
		0 output packet too big errors
		0 output echo requests
		0 output echo replies
		0 output router solicitations
		0 output router advertisements
		0 output neighbor solicitations
		0 output neighbor advertisements
		0 output redirects
		0 output MLD queries
		0 output MLD reports
		0 output MLD dones

netstat -s

Code:

tcp:
		243066824 packets sent
				216904261 data packets (7523403190815 bytes)
				140 data packets (81076 bytes) retransmitted
				11 data packets unnecessarily retransmitted
				0 resends initiated by MTU discovery
				26143393 ack-only packets (3232 delayed)
				0 URG only packets
				0 window probe packets
				3875 window update packets
				15173 control packets
		75915793 packets received
				43443686 acks (for 7523403212938 bytes)
				53665 duplicate acks
				0 acks for unsent data
				38573861 packets (258238812697 bytes) received in-sequence
				46850 completely duplicate packets (22308468 bytes)
				0 old duplicate packets
				13702 packets with some dup. data (1739025 bytes duped)
				4 out-of-order packets (5840 bytes)
				0 packets (0 bytes) of data after window
				0 window probes
				6876 window update packets
				11 packets received after close
				0 discarded for bad checksums
				0 discarded for bad header offset fields
				0 discarded because packet too short
				0 discarded due to memory problems
		6976 connection requests
		8176 connection accepts
		0 bad connection attempts
		0 listen queue overflows
		1 ignored RSTs in the window
		15135 connections established (including accepts)
		15151 connections closed (including 38 drops)
				804 connections updated cached RTT on close
				814 connections updated cached RTT variance on close
				590 connections updated cached ssthresh on close
		17 embryonic connections dropped
		25100148 segments updated rtt (of 25093255 attempts)
		162 retransmit timeouts
				10 connections dropped by rexmit timeout
		0 persist timeouts
				0 connections dropped by persist timeout
		0 Connections (fin_wait_2) dropped because of timeout
		20 keepalive timeouts
				20 keepalive probes sent
				0 connections dropped by keepalive
		28387079 correct ACK header predictions
		32349908 correct data packet header predictions
		8176 syncache entries added
				1 retransmitted
				1 dupsyn
				0 dropped
				8176 completed
				0 bucket overflow
				0 cache overflow
				0 reset
				0 stale
				0 aborted
				0 badack
				0 unreach
				0 zone failures
		8176 cookies sent
		0 cookies received
		10 hostcache entries added
				0 bucket overflow
		0 SACK recovery episodes
		0 segment rexmits in SACK recovery episodes
		0 byte rexmits in SACK recovery episodes
		26 SACK options (SACK blocks) received
		3 SACK options (SACK blocks) sent
		0 SACK scoreboard overflow
		0 packets with ECN CE bit set
		253367302 packets with ECN ECT(0) bit set
		0 packets with ECN ECT(1) bit set
		51 successful ECN handshakes
		0 times ECN reduced the congestion window
		0 packets with valid tcp-md5 signature received
		0 packets with invalid tcp-md5 signature received
		0 packets with tcp-md5 signature mismatch
		0 packets with unexpected tcp-md5 signature received
		0 packets without expected tcp-md5 signature received
udp:
		1126773 datagrams received
		0 with incomplete header
		0 with bad data length field
		0 with bad checksum
		0 with no checksum
		131604 dropped due to no socket
		995053 broadcast/multicast datagrams undelivered
		0 dropped due to full socket buffers
		0 not for hashed pcb
		116 delivered
		32624 datagrams output
		0 times multicast source filter matched
ip:
		77047161 total packets received
		0 bad header checksums
		0 with size smaller than minimum
		0 with data size < data length
		0 with ip length > max ip packet size
		0 with header length < data size
		0 with data length < header length
		0 with bad options
		0 with incorrect version number
		0 fragments received
		0 fragments dropped (dup or out of space)
		0 fragments dropped after timeout
		0 packets reassembled ok
		77026314 packets for this host
		17628 packets for unknown/unsupported protocol
		0 packets forwarded (0 packets fast forwarded)
		3219 packets not forwardable
		0 packets received for unknown multicast group
		0 redirects sent
		243238340 packets sent from this host
		0 packets sent with fabricated ip header
		0 output packets dropped due to no bufs, etc.
		5995 output packets discarded due to no route
		0 output datagrams fragmented
		0 fragments created
		0 datagrams that can't be fragmented
		0 tunneling packets that can't find gif
		0 datagrams with bad address in header
icmp:
		115350 calls to icmp_error
		0 errors not generated in response to an icmp message
		Output histogram:
				destination unreachable: 115350
		0 messages with bad code fields
		0 messages less than the minimum length
		0 messages with bad checksum
		0 messages with bad length
		0 multicast echo requests ignored
		0 multicast timestamp requests ignored
		Input histogram:
				echo reply: 2
				destination unreachable: 16262
		0 message responses generated
		0 invalid return addresses
		0 no return routes
		ICMP address mask responses are disabled
igmp:
		1372 messages received
		0 messages received with too few bytes
		0 messages received with wrong TTL
		0 messages received with bad checksum
		0 V1/V2 membership queries received
		1372 V3 membership queries received
		0 membership queries received with invalid field(s)
		1372 general queries received
		0 group queries received
		0 group-source queries received
		0 group-source queries dropped
		0 membership reports received
		0 membership reports received with invalid field(s)
		0 membership reports received for groups to which we belong
		0 V3 reports received without Router Alert
		18 membership reports sent
carp:
		0 packets received (IPv4)
		0 packets received (IPv6)
				0 packets discarded for wrong TTL
				0 packets shorter than header
				0 discarded for bad checksums
				0 discarded packets with a bad version
				0 discarded because packet too short
				0 discarded for bad authentication
				0 discarded for bad vhid
				0 discarded because of a bad address list
		0 packets sent (IPv4)
		0 packets sent (IPv6)
				0 send failed due to mbuf memory error
arp:
		135 ARP requests sent
		36018 ARP replies sent
		364327 ARP requests received
		5708 ARP replies received
		370035 ARP packets received
		7 total packets dropped due to no ARP entry
		35 ARP entrys timed out
		0 Duplicate IPs seen
ip6:
		32514 total packets received
		0 with size smaller than minimum
		0 with data size < data length
		0 with bad options
		0 with incorrect version number
		0 fragments received
		0 fragments dropped (dup or out of space)
		0 fragments dropped after timeout
		0 fragments that exceeded limit
		0 packets reassembled ok
		16260 packets for this host
		0 packets forwarded
		0 packets not forwardable
		0 redirects sent
		32514 packets sent from this host
		0 packets sent with fabricated ip header
		0 output packets dropped due to no bufs, etc.
		0 output packets discarded due to no route
		0 output datagrams fragmented
		0 fragments created
		0 datagrams that can't be fragmented
		0 packets that violated scope rules
		0 multicast packets which we don't join
		Input histogram:
				TCP: 6
				UDP: 16254
				ICMP6: 16254
		Mbuf statistics:
				16260 one mbuf
				two or more mbuf:
						lo0= 16254
				296 one ext mbuf
				0 two or more ext mbuf
		0 packets whose headers are not contiguous
		0 tunneling packets that can't find gif
		0 packets discarded because of too many headers
		16264 failures of source address selection
		source addresses on a non-outgoing I/F
				16264 addresses scope=f
		Source addresses selection rule applied:
				16264 same address
icmp6:
		16254 calls to icmp6_error
		0 errors not generated in response to an icmp6 message
		0 errors not generated because of rate limitation
		Output histogram:
				unreach: 16254
		0 messages with bad code fields
		0 messages < minimum length
		0 bad checksums
		0 messages with bad length
		Input histogram:
				unreach: 16254
		Histogram of error messages to be generated:
				0 no route
				0 administratively prohibited
				0 beyond scope
				0 address unreachable
				16254 port unreachable
				0 packet too big
				0 time exceed transit
				0 time exceed reassembly
				0 erroneous header field
				0 unrecognized next header
				0 unrecognized option
				0 redirect
				0 unknown
		0 message responses generated
		0 messages with too many ND options
		0 messages with bad ND options
		0 bad neighbor solicitation messages
		0 bad neighbor advertisement messages
		0 bad router solicitation messages
		0 bad router advertisement messages
		0 bad redirect messages
		0 path MTU changes
rip6:
		0 messages received
		0 checksum calculations on inbound
		0 messages with bad checksum
		0 messages dropped due to no socket
		0 multicast messages dropped due to no socket
		0 messages dropped due to full socket buffers
		0 delivered
		0 datagrams output

 

[Jonathan Hughes]

Okay now here's a new log to put on the fire... The issue with dropped connectivity and massive system log outputs of event ids 7,9,20,27,153 only appear to be triggered by the presence of a Hyper-v Generation 2 VM. Generation 1 VM's are operating normally and triggering no issues / drops in connectivity.

 

[c32767a]

That netstat commnand is s and lowercase l, not -s and uppercase i. The fonts on this forum are hard to differentiate.. :)

Having said that, what you posted looks pretty clean. I don't think there's any thing obviously wrong at the network layer.

 

[Jonathan Hughes]

That netstat commnand is s and lowercase l, not -s and uppercase i. The fonts on this forum are hard to differentiate.. :)

Having said that, what you posted looks pretty clean. I don't think there's any thing obviously wrong at the network layer.

Thanks for getting back to me. At this point i'm starting to think this might just be a bug in this release as i have a couple older freenas boxes with a similar configuration that do not have this problem with hyper-v gen2 VM's

 

[bigphil]

please post the output of "dmesg | grep ix0"

This looks very similar to issues with the X710 cards. I want to see the driver and firmware versions.

 

[Jonathan Hughes]

Code:

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> mem 0xf0200000-0xf03fffff,0xf0404000-0xf0407fff irq 17 at device 0.0 on pci1
ix0: Using MSIX interrupts with 5 vectors
ix0: Ethernet address: a0:36:9f:3e:f7:1c
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: link state changed to UP
ix0: link state changed to DOWN
ix0: link state changed to UP
ix0: link state changed to DOWN
ix0: link state changed to UP
ix0: link state changed to DOWN
ix0: link state changed to UP
ix0: link state changed to DOWN
ix0: link state changed to UP

No problem - output attached.

 

[Jonathan Hughes]

Ok it definitely has nothing to do with generation 2 vs generation 1 vm's. I just had the issue reappear after i moved a particularly IO chatty SQL server (gen1) over to it. At this point i'm tempted to go back to 9.3 where i have VM's working nicely. The only other test i will perform first will be moving away from 2 nics (1 for each extent) to a single interface. Will report back here if that helps resolve anything.

 

[Jonathan Hughes]

Ok reporting back in.
Reverted back to 9.3 and everything is happy now functioning on both nic interfaces - no more windows errors and buttery smooth IO. Must be a bug with 9.10 is my best guess at this point.

 

[bigphil]

Check this bug report out. Same driver version that is in FreeNAS 9.10...3.1.13-k. I bet this will be fixed once the devs move FreeNAS 9.10 to FreeBSD 11.

 

[Jonathan Hughes]

Check this bug report out. Same driver version that is in FreeNAS 9.10...3.1.13-k. I bet this will be fixed once the devs move FreeNAS 9.10 to FreeBSD 11.

This definitely jives with the behavior i've seen. Thanks so much for sharing this!

 

[bigphil]

Here is the latest 3.1.14 driver if you want to try running 9.10.2-U1 again. Instructions for how to load the driver can be found here. For this driver, you'll need to change the variable to "if_ix_load" instead.

Update: Re-uploaded file

 

[Jonathan Hughes]

Here is the latest 3.1.14 driver I've compiled against FreeBSD 10.3-STABLE. So...if you want to trying running the 9.10.2-U1 again, you can try this driver out. Instructions for how to load the driver can be found here. For this driver, you'll need to change the variable to "if_ix_load" instead.

Thanks! I'm actually in a spot where i can test this out. Just decom'd an old freenas box to repurpose. Will give it a try on the latest stable and report back. It has the same NIC.