iSCSI with Proxmox VE - excessive connections from initiators?

[gargravarr]

Hi folks,

I'm using a single zvol via iSCSI as the VHD store for my 4-node USFF Proxmox cluster. Physically, each node has a 2.5Gb USB NIC connected to a dedicated 2.5Gb+10Gb unmanaged switch, then via a single 10Gb connection to the TrueNAS machine. All 4 nodes are hitting the same IP address on TrueNAS (192.168.4.1).

A while ago, I was having trouble setting up FreeIPA integration so I looked in /var/log/daemon.log, and was surprised to see excessive iSCSI connections - each node is initiating a connection every few seconds. This is still going today:

Apr 3 09:24:00 poseidon iscsi-scstd[4283]: Connect from 192.168.4.2:42296 to 192.168.4.1:3260
Apr 3 09:24:01 poseidon iscsi-scstd[4283]: Connect from 192.168.4.3:40324 to 192.168.4.1:3260
Apr 3 09:24:08 poseidon iscsi-scstd[4283]: Connect from 192.168.4.4:47322 to 192.168.4.1:3260
Apr 3 09:24:10 poseidon iscsi-scstd[4283]: Connect from 192.168.4.2:54034 to 192.168.4.1:3260
Apr 3 09:24:11 poseidon iscsi-scstd[4283]: Connect from 192.168.4.5:44648 to 192.168.4.1:3260
Apr 3 09:24:14 poseidon iscsi-scstd[4283]: Connect from 192.168.4.3:44626 to 192.168.4.1:3260
Apr 3 09:24:19 poseidon iscsi-scstd[4283]: Connect from 192.168.4.4:42414 to 192.168.4.1:3260
Apr 3 09:24:20 poseidon iscsi-scstd[4283]: Connect from 192.168.4.2:49942 to 192.168.4.1:3260
Apr 3 09:24:20 poseidon iscsi-scstd[4283]: Connect from 192.168.4.5:60002 to 192.168.4.1:3260
Apr 3 09:24:28 poseidon iscsi-scstd[4283]: Connect from 192.168.4.4:42954 to 192.168.4.1:3260
Apr 3 09:24:28 poseidon iscsi-scstd[4283]: Connect from 192.168.4.3:40454 to 192.168.4.1:3260
Apr 3 09:24:31 poseidon iscsi-scstd[4283]: Connect from 192.168.4.5:35296 to 192.168.4.1:3260
Apr 3 09:24:31 poseidon iscsi-scstd[4283]: Connect from 192.168.4.2:39792 to 192.168.4.1:3260
Apr 3 09:24:38 poseidon iscsi-scstd[4283]: Connect from 192.168.4.4:44964 to 192.168.4.1:3260
Apr 3 09:24:40 poseidon iscsi-scstd[4283]: Connect from 192.168.4.2:50074 to 192.168.4.1:3260
Apr 3 09:24:41 poseidon iscsi-scstd[4283]: Connect from 192.168.4.5:50144 to 192.168.4.1:3260
Apr 3 09:24:44 poseidon iscsi-scstd[4283]: Connect from 192.168.4.3:42202 to 192.168.4.1:3260
Apr 3 09:24:49 poseidon iscsi-scstd[4283]: Connect from 192.168.4.4:43406 to 192.168.4.1:3260

There's nothing in the logs for iscsid on the hypervisors to indicate abnormal behaviour.

I thought iSCSI connections were supposed to be persistent, and I'm not finding much online to say whether or not this is correct behaviour. Seems like a performance limitation if it keeps closing and re-opening the iSCSI connection that all my VMs/CTs are sharing. At the very least, it's writing an awful lot of logs to the system SSD every day. Anyone have any insight?

Proxmox VE, also being Debian, is using the Open-iSCSI initiator, v2.1.8.

My TrueNAS Scale machine is running 23.10.2. It has an Intel Celeron N5100 processor with 16GB of DDR4 (12GB maximum for ZFS ARC) and a dual-port Intel X520 10Gb NIC.

Thanks!