Jonathan Hughes
Dabbler
- Joined
- Jan 30, 2017
- Messages
- 12
I'm having an issue where i recently installed a x540-t2 into a new freenas box. I have both ports of that connected to a matching x540-t2 on the hyper-v server (no switch). I've assigned static IP's for each interface and there is connectivity between the two machines using ms iscsi. the problem arises every so often where the NIC's just stop transmitting and receiving any data. I notice this on the reporting page of the freenas box - the interfaces traffic levels just drop to 0.
The errors i get on the windows side are mostly these:
Event ID 153: The IO operation at logical block address 0x40918 for Disk 10 (PDO name: \Device\0000006a) was retried.
Event ID 51: An error was detected on device \Device\Harddisk12\DR12 during a paging operation.
Event ID 140: The system failed to flush data to the transaction log. Corruption may occur in VolumeId: M:, DeviceName: \Device\HarddiskVolume27.
(A device which does not exist was specified.)
Event ID 1: Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data.
Event ID 63: Can not Reset the Target or LUN. Will attempt session recovery.
There are more but i'm hoping this is enough to help someone help me get down the road of troubleshooting this. I'm kind of out of ideas of things to try at this point. Any additional questions or help would be much appreciated!
Freenas Box:
Build FreeNAS-9.10.2-U1 (86c7ef5)
Platform Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
Motherboard ASRock Z97 EXTREME4 ATX LGA1150
Memory 32429MB
Single X540-T2 for 10gbe to Hyper-v
ix0: 100.100.100.2/24
ix1: 100.100.200.2/24
Hyper-V box:
HP DL-360 Gen 9
Windows Server 2012
Single X540-T2 for 10gbe to NAS
Port 1: 100.100.100.1/24
Port 2: 100.100.200.1/24
Logs from messages (changed machine name to hyper-vServer.omitted.com):
Jan 30 08:05:16 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:05:46 hv4nas1 (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 34 73 00 00 02 28 00 00
Jan 30 08:05:46 hv4nas1 (0:3:0/0): Tag: 0x43e9c302, type 1
Jan 30 08:05:46 hv4nas1 (0:3:0/0): ctl_process_done: 420 seconds
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:29 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:25 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 73 tasks
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 164 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:44 hv4nas1 connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:07 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:35 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:57 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:03 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:31 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
The errors i get on the windows side are mostly these:
Event ID 153: The IO operation at logical block address 0x40918 for Disk 10 (PDO name: \Device\0000006a) was retried.
Event ID 51: An error was detected on device \Device\Harddisk12\DR12 during a paging operation.
Event ID 140: The system failed to flush data to the transaction log. Corruption may occur in VolumeId: M:, DeviceName: \Device\HarddiskVolume27.
(A device which does not exist was specified.)
Event ID 1: Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data.
Event ID 63: Can not Reset the Target or LUN. Will attempt session recovery.
There are more but i'm hoping this is enough to help someone help me get down the road of troubleshooting this. I'm kind of out of ideas of things to try at this point. Any additional questions or help would be much appreciated!
Freenas Box:
Build FreeNAS-9.10.2-U1 (86c7ef5)
Platform Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
Motherboard ASRock Z97 EXTREME4 ATX LGA1150
Memory 32429MB
Single X540-T2 for 10gbe to Hyper-v
ix0: 100.100.100.2/24
ix1: 100.100.200.2/24
Hyper-V box:
HP DL-360 Gen 9
Windows Server 2012
Single X540-T2 for 10gbe to NAS
Port 1: 100.100.100.1/24
Port 2: 100.100.200.1/24
Logs from messages (changed machine name to hyper-vServer.omitted.com):
Jan 30 08:05:16 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:05:46 hv4nas1 (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 34 73 00 00 02 28 00 00
Jan 30 08:05:46 hv4nas1 (0:3:0/0): Tag: 0x43e9c302, type 1
Jan 30 08:05:46 hv4nas1 (0:3:0/0): ctl_process_done: 420 seconds
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 08:05:46 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): tasks terminated
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:26 hv4nas1 kernel: ix0: link state changed to DOWN
Jan 30 08:59:29 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 08:59:31 hv4nas1 kernel: ix0: link state changed to UP
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:21 hv4nas1 kernel: ix1: link state changed to DOWN
Jan 30 09:00:25 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:00:27 hv4nas1 kernel: ix1: link state changed to UP
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 73 tasks
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:50:21 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 164 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:50:26 hv4nas1 connection
Jan 30 09:50:26 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:39 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; droppingWARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:51:44 hv4nas1 connection
Jan 30 09:51:44 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:07 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:12 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:35 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:52:40 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:52:57 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:02 hv4nas1 WARNING: 100.100.200.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:03 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:08 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Jan 30 09:53:31 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): connection error; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jan 30 09:53:36 hv4nas1 WARNING: 100.100.100.1 (iqn.1991-05.com.microsoft:hyper-vServer.omitted.com): waiting for CTL to terminate 1 tasks
Last edited: