iSCSI lu_find_target() failed

[Michael Geoghegan]

We had a ZFS volume fail, and we were able to recreate it, however after we recreated, created the ZVOL, and reconfigured the iSCSI, I noticed we're getting a ton of errors in the console, pointing to the fact that the server is still looking for the old iSCSI target. (attached) I've checked the istgt.conf file, and the old target isn't referenced there, I'm running a GREP -lr for the name of the old target, but it hasn't come back yet. How do I clear this old target out so the system stops alerting?

Nov 8 15:58:45 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:45 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed
Nov 8 15:58:47 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:47 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed
Nov 8 15:58:49 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:49 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed
Nov 8 15:58:51 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:51 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed
Nov 8 15:58:53 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:53 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed
Nov 8 15:58:55 tier3 istgt[3024]: istgt_lu.c: 524:istgt_lu_find_target: ***WARNING*** can't find target iqn.2011-03.org.example.istgt:anchorcustomer
Nov 8 15:58:55 tier3 istgt[3024]: istgt_iscsi.c:2479:istgt_iscsi_op_login: ***ERROR*** lu_find_target() failed

 

[Michael Geoghegan]

Failed to mention, I initially suspected that this was an iSCSI initiator trying to connect to the target, but when I changed the IP of the FreeNAS box, the errors kept coming up. Here's my istgt.conf file:

[root@tier3] /usr/local/etc/istgt# cat istgt.conf
# Global section
[Global]
NodeBase "iqn.2011-03.org.example.istgt"
PidFile "/var/run/istgt.pid"
AuthFile "/usr/local/etc/istgt/auth.conf"
MediaDirectory /mnt
Timeout 300
NopInInterval 10
MaxR2T 32
DiscoveryAuthMethod None
MaxSessions 16
MaxConnections 8
FirstBurstLength 65536
MaxBurstLength 262144
MaxRecvDataSegmentLength 262144
MaxOutstandingR2T 16
DefaultTime2Wait 2
DefaultTime2Retain 60

[UnitControl]

# PortalGroup section
[PortalGroup1]
Portal DA1 172.26.99.200:3260

# InitiatorGroup section
[InitiatorGroup1]
InitiatorName "ALL"
Netmask ALL

# LogicalUnit section
[LogicalUnit1]
TargetName "anchorcloud"
TargetAlias "AnchorCloud"
Mapping PortalGroup1 InitiatorGroup1
AuthMethod Auto
UseDigest Auto
ReadOnly No
UnitType Disk
UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "0022191c053000"
UnitOnline yes
BlockLength 512
QueueDepth 64
LUN0 Storage /dev/zvol/AnchorTMI/TMI auto
LUN0 Option Serial 0022191c0530000
[LogicalUnit2]
TargetName "anchorcloud1"
TargetAlias "AnchorCloud1"
Mapping PortalGroup1 InitiatorGroup1
AuthMethod Auto
UseDigest Auto
ReadOnly No
UnitType Disk
UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "10000001"
UnitOnline yes
BlockLength 512
QueueDepth 32
LUN0 Storage /dev/zvol/Customers/AnchorTMI1 auto
LUN0 Option Serial 100000010
[root@tier3] /usr/local/etc/istgt#

 

[cyberjock]

I assume you were managing iSCSI through the WebGUI and deleted the old initiator settings and cycled the iSCSI service off and on?

The fact that you posted the istgt.conf file makes me think you did some modifications from the CLI before these problems. Am I correct? If so, what did you change?

 

[Michael Geoghegan]

1. Yes, managing and creating ISCSI through the WebGUI. I was able to delete the old target when the ZVOL failed. I've cycled the service and rebooted the server a handful of times. It's still looking for it.
2. I didn't make any modifications through the CLI, just trying to find why it's looking for the old target. Checked in the iSTGT.conf file to see if it was still there for some reason.

 

[cyberjock]

To be completely honest, those errors look like something is still trying to access the old iSCSI device. Going to go google the error and see if I can dig anything up for you though.

 

[Michael Geoghegan]

I dug around for a while, and I couldn't find anything, but the message is repeating a lot, I'm getting notifications on the CLI main screen that the message is repeating, over and over again, see attached image. I'm running the grep lr now trying to see what comes up. Thanks for the assist.

 

[cyberjock]

Ok, so please post your entire server hardware config and explain what the ISCSI device is for(if ESXi, how many VMs). Also mention any other problems you have(reliability or performance related).

 

[cyberjock]

I think your ls search is going to come back empty. I don't think its an issue on your FN server. I think another server that used to use the old iSCSI device is still trying to find it and that error is because it doesn't exist anymore.

 

[Michael Geoghegan]

Dell PE2950 with MD1000 attached through PERC6E card. ZFS is configured to use the MD1000 2TB disks for storage, and the 10k disks in the PE2950 as cache. Performance is spectacular, All firmware (PERC, BMC, DRAC, BIOS, etc) up to date. iSCSI target is connected to one VM as storage space for an application. Host box is Server 2008 R2 running Hyper-V, Virtual Machine is Server 2012 R2 Standard. iSCSI connection is done by the Virtual machine, host box is just handling the network passthrough. I've been getting errors in the application that it can't access the storage, however the developer thinks it's actually a memory dump that's causing the service to crash, thus causing the application to report errors accessing storage, I'm just trying to clean up errors and ensure that it's not a freenas problem. While the developer has logs indicating a memory dump issue, I have no concrete evidence that there is anything wrong with freenas, no errors in event log indicating iSCSI disconnects or disk problems.

 

[Michael Geoghegan]

I thought that as well, but I changed the ip address of the server and rebooted and tested. It was no longer answering on the old initiator ip address, however I was still getting blasted with errors. I'll test again tonight. Is there an iSCSI log file that I can look at, or is it all in /var/log/messages

 

[cyberjock]

I think all the logs are in /var/logs.

Secret trick.. if you make a dataset named syslog and reboot the logs will go to your dataset and be persistent through reboots.

 

[Michael Geoghegan]

I'll give it a try. I'm also fighting a few Unexpected sense errors that are like deciphering Sanskrit.

 

[cyberjock]

I'll give it a try. I'm also fighting a few Unexpected sense errors that are like deciphering Sanskrit.

Ah. I speak Bullsh*t, Sailor Colorful Vocabulary, and mumble-to-myself. But not Sanskrit. Sorry.

 

[Michael Geoghegan]

I'm familiar with that particular dialect (ET2(SW/AW) ) I changed the IP and restarted, waiting to see if the messages recur.

 

[cyberjock]

Former EM1/SS here.

 

[jpaetzel]

https://bugs.freenas.org/issues/5634

TL;DR This is an initiator that remembers a target you used to have still looking for it.