TrueNAS migrated to new system, iSCSII stopped working

[Tancients]

I moved my TrueNAS from a Supermicro 846 to a Dell R742xd. Backed up the config, exported the pools, moved the drives, imported the config, imported the pools. So far so good. Network interfaces were different, so I moved the config to the NICs that were available. For the iSCSI, it's served up on a zvol across a directly attached 10Gb fiber connection to an ESXi server. This is the same setup as was on the Supermicro, and the ESXi server didn't have any changes.

The 10Gb shows that the link is up for both sides. If the ESXi is down or the cable is disconnected, they each show as link down.
They cannot ping each other, only themselves. MTU is set to 9000, TrueNAS is configured with 192.168.253.1/24 (same as before), ESXi is 192.168.253.101/24.
I've changed the cable just in case it was the cable, no ping.
I moved the connection to a 1Gb ethernet port for both sides, this comes up and ping and iSCSI and everything works.

There's no tcpdump traffic or anything I can see going across the wire, and I've removed as much as I can think of from the equation. Any suggestions on what I can/should check, especially on the TrueNAS side which could be preventing the two from talking?

 

[Tancients]

Okay, weird update.

I created a new vswitch, I added the 10Gb NIC to it, and now I can ping both ends. I tried to setup iSCSI on it from the FreeNAS (I haven't deleted the old one), but ESXi isn't discovering it or liking it when I try to key in the details manually. It IS talking however, so that's nice.

However if I move the NIC back to the old vswitch, then the talking stops working again...

 

[Tancients]

So I deleted both ends of the iSCSI config, recreated it on a new vswitch, had it working momentarily but then it dies off. During a reboot, tcpdump from the FreeNAS side shows the below:

Code:

4082], length 62636
03:35:45.371907 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 385745:391593, ack 4673, win 24576, options [nop,nop,TS val 866410096 ecr 4082], length 5848
03:35:45.371983 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 287317, win 524, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372048 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 367849, win 367, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372133 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 391593, win 524, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372164 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4673:4721, ack 391593, win 524, options [nop,nop,TS val 4082 ecr 866410096], length 48
03:35:45.372417 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [.], seq 391593:454229, ack 4721, win 24576, options [nop,nop,TS val 866410096 ecr4082], length 62636
03:35:45.372452 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [.], seq 454229:516865, ack 4721, win 24576, options [nop,nop,TS val 866410096 ecr4082], length 62636
03:35:45.372481 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 516865:522713, ack 4721, win 24576, options [nop,nop,TS val 866410096 ecr 4082], length 5848
03:35:45.372570 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 427385, win 506, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372623 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 507917, win 349, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372703 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 522713, win 524, options [nop,nop,TS val 4082 ecr 866410096], length 0
03:35:45.372739 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4721:4769, ack 522713, win 524, options [nop,nop,TS val 4082 ecr 866410096], length 48
03:35:45.372946 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [.], seq 522713:585349, ack 4769, win 24576, options [nop,nop,TS val 866410097 ecr4082], length 62636
03:35:45.372982 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 647985:653833, ack 4769, win 24576, options [nop,nop,TS val 866410097 ecr 4082], length 5848
03:35:45.373097 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 558505, win 506, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373139 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 639037, win 349, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373220 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 653833, win 524, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373251 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4769:4817, ack 653833, win 524, options [nop,nop,TS val 4082 ecr 866410097], length 48
03:35:45.373459 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 779105:784953, ack 4817, win 24576, options [nop,nop,TS val 866410097 ecr 4082], length 5848
03:35:45.373608 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 689625, win 506, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373659 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 770157, win 349, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373736 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 784953, win 524, options [nop,nop,TS val 4082 ecr 866410097], length 0
03:35:45.373769 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4817:4865, ack 784953, win 524, options [nop,nop,TS val 4082 ecr 866410097], length 48
03:35:45.373927 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 910225:916073, ack 4865, win 24576, options [nop,nop,TS val 866410098 ecr 4082], length 5848
03:35:45.374071 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 811797, win 524, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374116 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 892329, win 367, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374194 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 916073, win 524, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374224 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4865:4913, ack 916073, win 524, options [nop,nop,TS val 4082 ecr 866410098], length 48
03:35:45.374358 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1041345:1047193, ack 4913, win 24576, options [nop,nop,TS val 866410098 ecr 4082], length 5848
03:35:45.374498 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 951865, win 506, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374552 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1032397, win 349, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374628 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1047193, win 524, options [nop,nop,TS val 4082 ecr 866410098], length 0
03:35:45.374666 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4913:4961, ack 1047193, win 524, options [nop,nop,TS val 4082 ecr 866410098], length 48
03:35:45.374797 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1172465:1178313, ack 4961, win 24576, options [nop,nop,TS val 866410099 ecr 4082], length 5848
03:35:45.374939 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1082985, win 506, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.374990 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1163517, win 349, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375068 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1178313, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375089 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 4961:5009, ack 1178313, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 48
03:35:45.375205 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1303585:1309433, ack 5009, win 24576, options [nop,nop,TS val 866410099 ecr 4082], length 5848
03:35:45.375337 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1205157, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375398 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1285689, win 367, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375480 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1309433, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375505 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5009:5057, ack 1309433, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 48
03:35:45.375556 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1309433:1325865, ack 5057, win 24576, options [nop,nop,TS val 866410099 ecr 4082], length 16432
03:35:45.375684 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1325865, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 0
03:35:45.375715 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5057:5105, ack 1325865, win 524, options [nop,nop,TS val 4082 ecr 866410099], length 48
03:35:45.375745 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1325865:1325945, ack 5105, win 24576, options [nop,nop,TS val 866410100 ecr 4082], length 80
03:35:45.375856 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5105:5153, ack 1325945, win 524, options [nop,nop,TS val 4082 ecr 866410100], length 48
03:35:45.375889 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1325945:1326009, ack 5153, win 24576, options [nop,nop,TS val 866410100 ecr 4082], length 64
03:35:45.375995 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5153:5201, ack 1326009, win 524, options [nop,nop,TS val 4082 ecr 866410100], length 48
03:35:45.376039 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1326009:1326569, ack 5201, win 24576, options [nop,nop,TS val 866410100 ecr 4082], length 560
03:35:45.376187 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5201:5249, ack 1326569, win 524, options [nop,nop,TS val 4082 ecr 866410100], length 48
03:35:45.376230 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1326569:1327129, ack 5249, win 24576, options [nop,nop,TS val 866410100 ecr 4082], length 560
03:35:45.376370 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5249:5297, ack 1327129, win 524, options [nop,nop,TS val 4083 ecr 866410100], length 48
03:35:45.376560 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1343561, win 524, options [nop,nop,TS val 4083 ecr 866410100], length 0
03:35:45.376642 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5297:5345, ack 1343561, win 524, options [nop,nop,TS val 4083 ecr 866410100], length 48
03:35:45.376676 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [P.], seq 1343561:1343633, ack 5345, win 24576, options [nop,nop,TS val 866410101 ecr 4083], length 72
03:35:45.376818 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5345:5393, ack 1343633, win 524, options [nop,nop,TS val 4083 ecr 866410101], length 48
03:35:45.377011 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [.], ack 1360065, win 524, options [nop,nop,TS val 4083 ecr 866410101], length 0
03:35:45.377052 IP 192.168.253.101.20478 > 192.168.253.1.iscsi-target: Flags [P.], seq 5393:6465, ack 1360065, win 524, options [nop,nop,TS val 4083 ecr 866410101], length 1072
03:35:58.696815 IP 192.168.253.1.iscsi-target > 192.168.253.101.53013: Flags [FP.], seq 8043981:8044173, ack 20277, win 24576, options [nop,nop,TS val 4255738843 ecr 4490], length 192
03:35:59.836796 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [FP.], seq 11635849:11635993, ack 41409, win 24576, options [nop,nop,TS val 866424561 ecr 4487], length 144
03:36:06.576821 IP 192.168.253.1.iscsi-target > 192.168.253.101.53013: Flags [FP.], seq 8043981:8044173, ack 20277, win 24576, options [nop,nop,TS val 4255746723 ecr 4490], length 192
03:36:07.716822 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [FP.], seq 11635849:11635993, ack 41409, win 24576, options [nop,nop,TS val 866432441 ecr 4487], length 144
03:36:22.136818 IP 192.168.253.1.iscsi-target > 192.168.253.101.53013: Flags [FP.], seq 8043981:8044173, ack 20277, win 24576, options [nop,nop,TS val 4255762283 ecr 4490], length 192
03:36:23.276816 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [FP.], seq 11635849:11635993, ack 41409, win 24576, options [nop,nop,TS val 866448001 ecr 4487], length 144
03:36:37.696822 IP 192.168.253.1.iscsi-target > 192.168.253.101.53013: Flags [FP.], seq 8043981:8044173, ack 20277, win 24576, options [nop,nop,TS val 4255777843 ecr 4490], length 192
03:36:38.836817 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [FP.], seq 11635849:11635993, ack 41409, win 24576, options [nop,nop,TS val 866463561 ecr 4487], length 144
03:36:53.256818 IP 192.168.253.1.iscsi-target > 192.168.253.101.53013: Flags [FP.], seq 8043981:8044173, ack 20277, win 24576, options [nop,nop,TS val 4255793403 ecr 4490], length 192
03:36:54.396813 IP 192.168.253.1.iscsi-target > 192.168.253.101.20478: Flags [FP.], seq 11635849:11635993, ack 41409, win 24576, options [nop,nop,TS val 866479121 ecr 4487], length 144

Then it goes dead with length 0, and loses connection, reporting the iSCSI endpoints are timed out.

 

[Tancients]

So, further poking, it doesn't look like I can have an MTU of 9000 set anymore. Once I put the MTU on both sides back to 1500, the connection was made and it remained connected. I'm currently migrating items through two datastores, but would be interested if the low MTU is why I'm only getting 1Gbs throughput on the network or if I'm hitting the SSD limits (6 drive MX500 1TB in a raidz2 in the new zdev, some older SSDs x4 in a raidz1 on the old zdev).

Either way, at least it's up and running now, so I can finish zpool migration stuff and get some sleep...