istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed

[poldas]

Hi,

Software: FreeNAS-9.1.1-RELEASE-x64 (a752d35)

Hardware: DELL 2950, RAID Controler M1015 in JBOB mode, 16 GB RAM DDR2, 6 x SATA WD Red, Firmware on 16 GB USB stick.

Raid mode: RAID-Z2

We use FreeNAS like a iSCSI storage for 2 XenServers (10.0.0.2 and 10.0.0.3). Sometimes we notice freezing all VMs in the same time. Below are logs from the accident.

Code:

Nov 20 04:02:29 freenas istgt[2200]: istgt_lu_disk.c:5318:istgt_lu_disk_queue_start: ***WARNING*** transfer abort CmdSN=54051013
Nov 20 04:02:29 freenas istgt[2200]: istgt_iscsi.c:3924:istgt_iscsi_task_response: ***ERROR*** iscsi_write_pdu() failed
Nov 20 04:02:29 freenas istgt[2200]: istgt_lu.c:3255:luworker: ***WARNING*** LU2: lu_disk_queue_start() aborted
Nov 20 04:02:29 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:11e6b57a,time=0)
Nov 20 04:02:29 freenas istgt[2200]: istgt_iscsi.c:5392:sender: ***ERROR*** iscsi_task_response() CmdSN=187536258 failed on iqn.2011-03.org.freenas.istgt:target2,t,0x0001(iqn.2014-08.com.example:cfba1dc1,i,0x00023d020000)
Nov 20 04:02:29 freenas istgt[2200]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-03.org.freenas.istgt:target2,t,0x0001(iqn.2014-08.com.example:11e6b57a,i,0x00023d020000)
Nov 20 04:02:30 freenas istgt[2200]: Login from iqn.2014-08.com.example:cfba1dc1 (10.0.0.3) on iqn.2011-03.org.freenas.istgt:target2 LU2 (10.0.0.1:3260,1), ISID=23d020000, TSIH=6, CID=0, HeaderDigest=off, DataDigest=off
Nov 20 04:02:32 freenas istgt[2200]: Login from iqn.2014-08.com.example:11e6b57a (10.0.0.2) on iqn.2011-03.org.freenas.istgt:target2 LU2 (10.0.0.1:3260,1), ISID=23d020000, TSIH=7, CID=0, HeaderDigest=off, DataDigest=off
Nov 20 04:23:57 freenas ntpd[93184]: time reset +199.299585 s
Nov 20 08:06:06 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list  -v
Nov 20 08:06:07 freenas last message repeated 5 times
Nov 20 10:10:36 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list  -v
Nov 20 10:19:18 freenas last message repeated 5 times
Nov 21 06:04:46 freenas ntpd[93184]: time reset +3.687978 s
Nov 22 03:48:08 freenas ntpd[93184]: time reset +3.693005 s
Nov 25 02:06:52 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:cfba1dc1,time=0)
Nov 25 02:06:52 freenas istgt[2200]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-03.org.freenas.istgt:target1,t,0x0001(iqn.2014-08.com.example:cfba1dc1,i,0x00023d010000)
Nov 25 02:06:52 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:11e6b57a,time=0)
Nov 25 02:06:52 freenas istgt[2200]: istgt_iscsi.c:3924:istgt_iscsi_task_response: ***ERROR*** iscsi_write_pdu() failed
Nov 25 02:06:52 freenas istgt[2200]: istgt_iscsi.c:5392:sender: ***ERROR*** iscsi_task_response() CmdSN=154928186 failed on iqn.2011-03.org.freenas.istgt:target2,t,0x0001(iqn.2014-08.com.example:11e6b57a,i,0x00023d020000)
Nov 25 02:06:52 freenas istgt[2200]: Login from iqn.2014-08.com.example:11e6b57a (10.0.0.2) on iqn.2011-03.org.freenas.istgt:target2 LU2 (10.0.0.1:3260,1), ISID=23d020000, TSIH=8, CID=0, HeaderDigest=off, DataDigest=off
Nov 25 02:06:53 freenas istgt[2200]: Login from iqn.2014-08.com.example:cfba1dc1 (10.0.0.3) on iqn.2011-03.org.freenas.istgt:target1 LU1 (10.0.0.1:3260,1), ISID=23d010000, TSIH=5, CID=0, HeaderDigest=off, DataDigest=off
Nov 25 02:31:23 freenas ntpd[93184]: time reset +199.296725 s
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:cfba1dc1,time=0)
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:3924:istgt_iscsi_task_response: ***ERROR*** iscsi_write_pdu() failed
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:5392:sender: ***ERROR*** iscsi_task_response() CmdSN=1347438 failed on iqn.2011-03.org.freenas.istgt:target1,t,0x0001(iqn.2014-08.com.example:cfba1dc1,i,0x00023d010000)
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:11e6b57a,time=0)
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:3924:istgt_iscsi_task_response: ***ERROR*** iscsi_write_pdu() failed
Nov 26 12:30:53 freenas istgt[2200]: istgt_iscsi.c:5392:sender: ***ERROR*** iscsi_task_response() CmdSN=179010758 failed on iqn.2011-03.org.freenas.istgt:target2,t,0x0001(iqn.2014-08.com.example:11e6b57a,i,0x00023d020000)
Nov 26 12:30:55 freenas istgt[2200]: Login from iqn.2014-08.com.example:11e6b57a (10.0.0.2) on iqn.2011-03.org.freenas.istgt:target2 LU2 (10.0.0.1:3260,1), ISID=23d020000, TSIH=9, CID=0, HeaderDigest=off, DataDigest=off
Nov 26 12:30:56 freenas istgt[2200]: Login from iqn.2014-08.com.example:cfba1dc1 (10.0.0.3) on iqn.2011-03.org.freenas.istgt:target1 LU1 (10.0.0.1:3260,1), ISID=23d010000, TSIH=6, CID=0, HeaderDigest=off, DataDigest=off
Nov 26 12:53:22 freenas ntpd[93184]: time reset +199.297398 s
Nov 27 06:55:19 freenas ntpd[93184]: time reset +3.687517 s
Nov 27 16:53:04 freenas istgt[2200]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.2014-08.com.example:cfba1dc1,time=0)
Nov 27 16:53:04 freenas istgt[2200]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-03.org.freenas.istgt:target2,t,0x0001(iqn.2014-08.com.example:cfba1dc1,i,0x00023d020000)
Nov 27 16:53:05 freenas istgt[2200]: Login from iqn.2014-08.com.example:cfba1dc1 (10.0.0.3) on iqn.2011-03.org.freenas.istgt:target2 LU2 (10.0.0.1:3260,1), ISID=23d020000, TSIH=10, CID=0, HeaderDigest=off, DataDigest=off
Nov 27 17:29:29 freenas ntpd[93184]: time reset +228.825968 s
Nov 28 08:11:00 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list  -v
Nov 28 08:11:01 freenas last message repeated 2 times

 

[poldas]

Maybe I should use specific iscsi settings for a big LUN (7098GB) for example:

- Queue Depth (32)
- Logical Block Size (512)