Trying to use an iSCSI target as storage for ESXi 5.0

[freenascod]

Hello guys,

I've got a FreeNAS-8.2.0-BETA4-x64 (r11722) server set up in my environment. I have several iSCSI targets set up on this machine. These targets are accessibly from Windows boxes via the Microsoft iSCSI initiator and everything seems to be working well there.

What I'm trying to do now is use an iSCSI target as storage in ESXi so that I can store virtual machines on it. I've followed the guides and everything appears to be set up correctly, but when I scan to find/connect-to targets I am never able to connect. ESXi SEES the targets. But when I scan the freeNAS server for storage I see the following messages appear in the console:


Jul 2 09:36:32 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-f,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000002)
Jul 2 09:36:33 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:33 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-i,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000001)
Jul 2 09:36:34 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:34 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-i,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000002)
Jul 2 09:36:35 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:35 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-l,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000001)
Jul 2 09:36:36 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:36 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-l,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000002)
Jul 2 09:36:37 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:37 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-new-i,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000001)
Jul 2 09:36:38 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:38 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:cubesrv-new-i,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000002)
Jul 2 09:36:39 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:39 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:vmh01-appclient,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000001)
Jul 2 09:36:40 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:40 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:vmh01-appclient,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000002)
Jul 2 09:36:41 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c: 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (er
rno=32)
Jul 2 09:36:41 atlprdnsfreenas1 istgt[4243]: istgt_iscsi.c:4984:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.atlprdnsfreenas1
.ins.incomm.com:o-target,t,0x0001(iqn.1998-01.com.vmware:atlprdnsvmh01-22262620,i,0x00023d000001)

For every target I have in the system. Now only the bolded one is the target I want to connect to. I set the authentication for this to be the same as the authentication for the server itself, and since ESXi can log in and discover the targets on the server, I would assume everything would be correct there. It looks like I'm not alone in this problem either, as I've found this thread where people are having the same issue I am:

http://forums.freenas.org/showthread.php?1881-ESXi-5-0-use-FreeNAS-iSCSI-target-issue/page2

But I'm far past the version of FreeNAS they were using in that post. Does anyone have any insight into why this could be happening? Any help would be GREATLY appreciated. Thanks!

 

[jgreco]

Yeah, my guess is that the VMware host is dropping the connections due to poor responsiveness. Could you please engage tech support mode on the VMware host to look and tell us what it is logging as the reason for disconnect? The errno=32 basically means that FreeNAS is (probably) seeing a disconnection caused by the VMware host dropping the iSCSI session.

You may want to review FreeNAS bugs #1104 and #1531, among others, to gain some hints as to what to try/do/look at next. Basically I would guess that you need to do some tuning to make it happy. I have yet to find a case where setting vfs.zfs.txg.timeout="5" breaks things, it may lower overall performance somewhat but seems to help performance under load substantially. I do suggest you do that regardless of the remainder of this message.

There are some really pathological cases that can be built up in terms of ZFS workloads that can result in crazy bad cache flushing sessions where ZFS hangs userland I/O (and therefore the istgt process serving up iSCSI).

The real problem with this is that you can generate a heavy sequential workload with something like dd, and discover that you need to set vfs.zfs.txg.write_limit_override in some particular range (let's say 256MB), but then under a production workload that's much less sequential, you actually find out that you really need that to be more like 64MB. The flip side to that is that by setting such an override, you're then killing performance for any sequential workloads that you might have as part of your production traffic. This is very frustrating. I can't help the feeling that write_limit_override is the wrong knob to twiddle.

 

[freenascod]

Thanks for the insightful reply, jgreco. I did manage to get this solved by trying a different tact. I created an alias in FreeNAS for the iSCSI lagg. I then created a new iSCSI portal with the alias IP address. I then placed the one target that I wanted to use for ESX into this new portal and restarted the iSCSI service. Once I did that the LUN showed up immediately in ESXi. I'm good for now. Thanks!