My Freenas box had an "episode" last night where it logged a number of errors mostly the "force cleanup execute" error (actually a warning) and some write errors. I picked it up as my ESXi hosts reported lost connectivity to the storage. After a flurry of login and error messages it all went quiet again and everything looked to continue as normal, nothing unusual in the daily run email this morning. This is the first time my Freenas box has produced any errors, so I'm looking for any advice as to what the cause may be.
I've pasted my system summary and the error outputs below (sorry re the smileys, couldn't see how to turn them off)
Thanks in advance,
Dave
Hostname freenas.wpi-international.co.nz
Build FreeNAS-9.1.0-RELEASE-x64 (dff7d13)
Platform AMD Athlon(tm) II X2 260 Processor
Memory 7915MB
System Time Wed Mar 05 08:38:18 NZDT 2014
Uptime 8:38AM up 193 days, 20:46, 0 users
Load Average 0.69, 0.69, 0.63
Mar 4 11:11:27 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 4 11:11:27 freenas last message repeated 2 times
Mar 4 18:26:41 freenas istgt[2007]: istgt_lu_disk.c:5318:istgt_lu_disk_queue_start: ***WARNING*** transfer abort CmdSN=164105386
Mar 4 18:26:41 freenas istgt[2007]: istgt_lu.c:3255:luworker: ***WARNING*** LU5: lu_disk_queue_start() aborted
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=8, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=9, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=10, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=11, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=12, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 3 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=13, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=14, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=15, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=18, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=20, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=17, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=16, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=22, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=23, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=24, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=19, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=26, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=28, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=21, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=29, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz (10.0.0.169) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=400001370001, TSIH=25, CID=1, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=31, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:esx-utility-1c551b4c,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:esx-utility-1c551b4c,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=27, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:esx-inet-11e320b4,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:esx-inet-11e320b4,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz,i,0x400001370001)
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=30, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:wpi-esxi2-1d93781f,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:wpi-esxi2-1d93781f,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-utility-1c551b4c (10.0.0.101) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=32, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=33, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 19:06:05 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 4 19:06:06 freenas last message repeated 2 times
Mar 5 08:21:34 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 5 08:21:34 freenas last message repeated 2 times
I've pasted my system summary and the error outputs below (sorry re the smileys, couldn't see how to turn them off)
Thanks in advance,
Dave
Hostname freenas.wpi-international.co.nz
Build FreeNAS-9.1.0-RELEASE-x64 (dff7d13)
Platform AMD Athlon(tm) II X2 260 Processor
Memory 7915MB
System Time Wed Mar 05 08:38:18 NZDT 2014
Uptime 8:38AM up 193 days, 20:46, 0 users
Load Average 0.69, 0.69, 0.63
Mar 4 11:11:27 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 4 11:11:27 freenas last message repeated 2 times
Mar 4 18:26:41 freenas istgt[2007]: istgt_lu_disk.c:5318:istgt_lu_disk_queue_start: ***WARNING*** transfer abort CmdSN=164105386
Mar 4 18:26:41 freenas istgt[2007]: istgt_lu.c:3255:luworker: ***WARNING*** LU5: lu_disk_queue_start() aborted
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=8, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=9, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=10, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=11, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=12, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 3 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=13, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=14, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=15, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=18, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=20, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=17, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=16, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=22, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=23, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=24, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=19, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=26, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=28, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=21, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=29, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz (10.0.0.169) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=400001370001, TSIH=25, CID=1, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=31, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:esx-utility-1c551b4c,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:esx-utility-1c551b4c,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a (10.0.0.150) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=27, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:esx-inet-11e320b4,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:esx-inet-11e320b4,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:wpi-esxi1-0cf1cc6a,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1991-05.com.microsoft:wpi-vpulpman.wpi-international.co.nz,i,0x400001370001)
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:wpi-esxi2-1d93781f (10.0.0.152) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=30, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:1261:istgt_iscsi_write_pdu_internal: ***ERROR*** writev() failed (errno=32,iqn.1998-01.com.vmware:wpi-esxi2-1d93781f,time=0)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5414:sender: ***ERROR*** iscsi_write_pdu() failed on iqn.2011-08.wpinz.com.istgt:pulpman,t,0x0001(iqn.1998-01.com.vmware:wpi-esxi2-1d93781f,i,0x00023d000001)
Mar 4 18:26:48 freenas istgt[2007]: istgt_iscsi.c:5270:worker_cleanup: ***WARNING*** force cleanup execute
Mar 4 18:26:48 freenas last message repeated 2 times
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-utility-1c551b4c (10.0.0.101) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=32, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 18:26:48 freenas istgt[2007]: Login from iqn.1998-01.com.vmware:esx-inet-11e320b4 (10.0.0.105) on iqn.2011-08.wpinz.com.istgt:pulpman LU5 (10.0.0.104:3260,1), ISID=23d000001, TSIH=33, CID=0, HeaderDigest=off, DataDigest=off
Mar 4 19:06:05 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 4 19:06:06 freenas last message repeated 2 times
Mar 5 08:21:34 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list -v
Mar 5 08:21:34 freenas last message repeated 2 times