code = Unknown desc = unable to attach any iscsi devices

[sekroots]

Hello,

I have a Storage TrueNAS-12.0-U8 ( virtuallized Hyper-V) with a pool and iSCSI sharing. It was working fine until few hours ago:

Apr 7 08:28:07 truenas 1 2022-04-07T08:28:07.207272-07:00 truenas.mycompany.com.br ctld 1244 - - child process 18341 terminated with exit status 1
Apr 7 08:28:07 truenas 1 2022-04-07T08:28:07.209004-07:00 truenas.mycompany.com.br ctld 18342 - - 172.30.0.87 (iqn.1994-05.com.redhat:23c3b778d047): error returned from CTL iSCSI handoff request: cfiscsi_ioctl_handoff: target not found; dropping connection
Apr 7 08:28:07 truenas 1 2022-04-07T08:28:07.215481-07:00 truenas.mycompany.com.br ctld 1244 - - child process 18342 terminated with exit status 1

How can I Fix it? Thanks

Code:

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Mon Apr  4 03:45:05 2022
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 00:20:28 with 0 errors on Tue Mar  1 11:15:31 2022
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE       0     0     0
          gptid/4806796c-8f06-11eb-a787-00155d032ec0  ONLINE       0     0     0

errors: No known data errors
root@truenas[~]#

root@truenas[/var/log]# cat messages|grep pvc-10710a41-ed08-4844-b378-f99e671d251c
Apr 7 09:33:16 truenas 1 2022-04-07T09:33:16.028000-07:00 truenas.mycompany.com.br ctld 1244 - - no LUNs defined for target "iqn.2005-10.org.freenas.ctl:pvc-10710a41-ed08-4844-b378-f99e671d251c"
Apr 7 09:33:16 truenas 1 2022-04-07T09:33:16.080841-07:00 truenas.mycompany.com.br ctld 1244 - - failed to add port pg1-iqn.2005-10.org.freenas.ctl:pvc-10710a41-ed08-4844-b378-f99e671d251c
root@truenas[/var/log]#

 

[sekroots]

I guess this is weird.

 

[sekroots]

Code:

netstat- s

tcp:
    1536126234 packets sent
        1196146185 data packets (14425285524831 bytes)
        5098387 data packets (7326937446 bytes) retransmitted
        2259 data packets unnecessarily retransmitted
        1 resend initiated by MTU discovery
        331344239 ack-only packets (42432501 delayed)
        0 URG only packets
        0 window probe packets
        245099 window update packets
        3292401 control packets
    1856747468 packets received
        1455612760 acks (for 14425289806870 bytes)
        21467197 duplicate acks
        0 acks for unsent data
        1893610006 packets (1797039480165 bytes) received in-sequence
        806249 completely duplicate packets (341057237 bytes)
        2660 old duplicate packets
        3040 packets with some dup. data (2470086 bytes duped)
        8184338 out-of-order packets (34613705415 bytes)
        0 packets (0 bytes) of data after window
        0 window probes
        276533 window update packets
        71 packets received after close
        0 discarded for bad checksums
        0 discarded for bad header offset fields
        0 discarded because packet too short
        1697 discarded due to full reassembly queue
    428326 connection requests
    2483058 connection accepts
    0 bad connection attempts
    0 listen queue overflows
    65697 ignored RSTs in the windows
    2911382 connections established (including accepts)
        2909551 times used RTT from hostcache
        2909551 times used RTT variance from hostcache
        1440025 times used slow-start threshold from hostcache
    4291966 connections closed (including 247664 drops)
        1385641 connections updated cached RTT on close
        1385641 connections updated cached RTT variance on close
        921225 connections updated cached ssthresh on close
    0 embryonic connections dropped
    1129658990 segments updated rtt (of 859807382 attempts)
    385278 retransmit timeouts
        73 connections dropped by rexmit timeout
    0 persist timeouts
        0 connections dropped by persist timeout
    0 Connections (fin_wait_2) dropped because of timeout
    1862 keepalive timeouts
        1801 keepalive probes sent
        61 connections dropped by keepalive
    750812370 correct ACK header predictions
    658446334 correct data packet header predictions
    2483060 syncache entries added
        70 retransmitted
        85 dupsyn
        1 dropped
        2483058 completed
        0 bucket overflow
        0 cache overflow
        2 reset
        0 stale
        0 aborted
        0 badack
        0 unreach
        0 zone failures
    2483061 cookies sent
    0 cookies received
    1087 hostcache entries added
        0 bucket overflow
    4389029 SACK recovery episodes
    4701595 segment rexmits in SACK recovery episodes
    6759788717 byte rexmits in SACK recovery episodes
    28229133 SACK options (SACK blocks) received
    9908072 SACK options (SACK blocks) sent
    0 SACK scoreboard overflow
    0 packets with ECN CE bit set
    3999935 packets with ECN ECT(0) bit set
    0 packets with ECN ECT(1) bit set
    113 successful ECN handshakes
    0 times ECN reduced the congestion window
    0 packets with matching signature received
    0 packets with bad signature received
    0 times failed to make signature due to no SA
    0 times unexpected signature received
    0 times no signature provided by segment
    0 Path MTU discovery black hole detection activations
    0 Path MTU discovery black hole detection min MSS activations
    0 Path MTU discovery black hole detection failures
TCP connection count by state:
    0 connections in CLOSED state
    25 connections in LISTEN state
    0 connections in SYN_SENT state
    0 connections in SYN_RCVD state
    44 connections in ESTABLISHED state
    0 connections in CLOSE_WAIT state
    0 connections in FIN_WAIT_1 state
    0 connections in CLOSING state
    0 connections in LAST_ACK state
    0 connections in FIN_WAIT_2 state
    154 connections in TIME_WAIT state
udp:
    10182352 datagrams received
    0 with incomplete header
    0 with bad data length field
    4 with bad checksum
    236984 with no checksum
    1671211 dropped due to no socket
    6820047 broadcast/multicast datagrams undelivered
    0 dropped due to full socket buffers
    0 not for hashed pcb
    1691090 delivered
    456778 datagrams output
    0 times multicast source filter matched
ip:
    1865811657 total packets received
    0 bad header checksums
    0 with size smaller than minimum
    40 with data size < data length
    0 with ip length > max ip packet size
    0 with header length < data size
    0 with data length < header length
    0 with bad options
    0 with incorrect version number
    60622 fragments received
    18755 fragments dropped (dup or out of space)
    9807 fragments dropped after timeout
    8823 packets reassembled ok
    1865754982 packets for this host
    338 packets for unknown/unsupported protocol
    0 packets forwarded (0 packets fast forwarded)
    4498 packets not forwardable
    228298 packets received for unknown multicast group
    0 redirects sent
    1536587965 packets sent from this host
    604 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    0 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 tunneling packets that can't find gif
    0 datagrams with bad address in header
icmp:
    1071 calls to icmp_error
    0 errors not generated in response to an icmp message
    Output histogram:
        echo reply: 333
        destination unreachable: 733
    0 messages with bad code fields
    0 messages less than the minimum length
    0 messages with bad checksum
    0 messages with bad length
    0 multicast echo requests ignored
    0 multicast timestamp requests ignored
    Input histogram:
        echo reply: 2
        destination unreachable: 1191
        routing redirect: 32
        echo: 333
        time exceeded: 12
    333 message responses generated
    0 invalid return addresses
    0 no return routes
    ICMP address mask responses are disabled
igmp:
    494050 messages received
    0 messages received with too few bytes
    60 messages received with wrong TTL
    0 messages received with bad checksum
    0 V1/V2 membership queries received
    0 V3 membership queries received
    0 membership queries received with invalid field(s)
    0 general queries received
    0 group queries received
    0 group-source queries received
    0 group-source queries dropped
    29783 membership reports received
    0 membership reports received with invalid field(s)
    29784 membership reports received for groups to which we belong
    0 V3 reports received without Router Alert
    16 membership reports sent
carp:
    0 packets received (IPv4)
    0 packets received (IPv6)
        0 packets discarded for wrong TTL
        0 packets shorter than header
        0 discarded for bad checksums
        0 discarded packets with a bad version
        0 discarded because packet too short
        0 discarded for bad authentication
        0 discarded for bad vhid
        0 discarded because of a bad address list
    0 packets sent (IPv4)
    0 packets sent (IPv6)
        0 send failed due to mbuf memory error
arp:
    16231 ARP requests sent
    34672 ARP replies sent
    6331208 ARP requests received
    24374 ARP replies received
    6355582 ARP packets received
    930 total packets dropped due to no ARP entry
    4913 ARP entrys timed out
    0 Duplicate IPs seen
ip6:
    4606994 total packets received
    0 with size smaller than minimum
    0 with data size < data length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 fragments that exceeded limit
    0 atomic fragments
    0 packets reassembled ok
    4606978 packets for this host
    0 packets forwarded
    0 packets not forwardable
    0 redirects sent
    36 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    517 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 packets that violated scope rules
    2936482 multicast packets which we don't join
    Input histogram:
        hop by hop: 390106
        UDP: 1670496
        ICMP6: 2546392
    Mbuf statistics:
        4025816 one mbuf
        two or more mbuf:
            lo0= 18
        581160 one ext mbuf
        0 two or more ext mbuf
    0 packets whose headers are not contiguous
    0 tunneling packets that can't find gif
    0 packets discarded because of too many headers
    23 failures of source address selection
    source addresses on a non-outgoing I/F
        23 addresses scope=0xf
    Source addresses selection rule applied:
        23 same address
icmp6:
    16 calls to icmp6_error
    0 errors not generated in response to an icmp6 message
    0 errors not generated because of rate limitation
    Output histogram:
        unreach: 16
    0 messages with bad code fields
    0 messages < minimum length
    0 bad checksums
    0 messages with bad length
    Input histogram:
        unreach: 16
    Histogram of error messages to be generated:
        0 no route
        0 administratively prohibited
        0 beyond scope
        0 address unreachable
        16 port unreachable
        0 packet too big
        0 time exceed transit
        0 time exceed reassembly
        0 erroneous header field
        0 unrecognized next header
        0 unrecognized option
        0 redirect
        0 unknown
    0 message responses generated
    0 messages with too many ND options
    0 messages with bad ND options
    0 bad neighbor solicitation messages
    0 bad neighbor advertisement messages
    0 bad router solicitation messages
    0 bad router advertisement messages
    0 bad redirect messages
    0 path MTU changes
rip6:
    0 messages received
    0 checksum calculations on inbound
    0 messages with bad checksum
    0 messages dropped due to no socket
    0 multicast messages dropped due to no socket
    0 messages dropped due to full socket buffers
    0 delivered
    0 datagrams output

 

[sekroots]

failed to execute filesystem command: stat /dev/disk/by-path/ip-172.30.3.245:3260-iscsi-iqn.2005-10.org.freenas.ctl:pvc-4cb1a730-f778-49e4-9134-6f4e55f505fe-lun-0, response: {"code":1,"stdout":"","stderr":"stat: cannot stat '/dev/disk/by-path/ip-172.30.3.245:3260-iscsi-iqn.2005-10.org.freenas.ctl:pvc-4cb1a730-f778-49e4-9134-6f4e55f505fe-lun-0': No such file or directory\n","timeout":false}

[root@k8s by-path]# ls
acpi-VMBUS:00-scsi-0:0:0:0 acpi-VMBUS:00-scsi-0:0:0:0-part2
acpi-VMBUS:00-scsi-0:0:0:0-part1 acpi-VMBUS:00-scsi-0:0:0:0-part3
[root@k8s by-path]# pwd
/dev/disk/by-path
[root@k8s by-path]# stat
stat: missing operand
Try 'stat --help' for

 

[sekroots]

We solved it. It was a FreeBSD limitation to 256 LUNs.


References:

Other - ctld and ctladm

Hi list, I am currently trying to write a Perl module for automatic creation and removing on LUNs using ZVOL as backing store using a combination of ctladm and ctld. I have more or less solved all issues but I have a problem with three issues: LUNs created with ctladm do not survive a reboot...

forums.freebsd.org

ctl(4)

www.freebsd.org

⚙ D12836 ctl: Make max_luns and max_ports tunable variables instead of hardcoded defines.

reviews.freebsd.org