[SOLVED]: Strange LUN and iSCSI Errors - Not Managed by the Block Backend?

[EagleTG]

Hi all,

I noticed some odd corruption on a VM running on a Proxmox host connected via 'ZFS Over iSCSI' to my FreeNAS instance.

I found entries in /var/log/messages from early this morning:

Code:

Oct 29 12:23:44 freenas ctld[1397]: failed to reload configuration
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 17 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/base-102-disk-0", CTL lun 17
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 13 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/base-104-disk-0", CTL lun 13
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 7 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-100-disk-0", CTL lun 7
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 10 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-103-disk-0", CTL lun 10
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 18 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-103-disk-1", CTL lun 18
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 19 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-105-disk-0", CTL lun 19
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 26 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-106-disk-0", CTL lun 26
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 22 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-107-disk-0", CTL lun 22
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 23 is not managed by the block backend
Oct 29 12:23:45 freenas ctld[1397]: failed to modify lun "freenasssd01/vm-108-disk-0", CTL lun 23
Oct 29 12:23:45 freenas ctld[1397]: LUN modification error: LUN 25 is not managed by the block backend

I've searched high and low on the internet and can't seem to find anyone else reporting this type of error. I did find some references in the FreeBSD forums, but they don't seem to apply to what I'm seeing. Interestingly, the errors started "small", only relating to 3 of the LUNs and have expanded to a lot more of them. (Not all of the LUNs though).

Additionally, the errors have since stopped in the logs. The log snippet above is the last entry reported (so the errors occurred for around 20 minutes). Everything looks OK in the FreeNAS UI, but I still seem to be having unusual issues connecting to the LUNs via iSCSI. Specifically, I'm seeing 'iscsiadm: No session found' from the Proxmox hosts for new/not-currently-established connections. Additionally, I ran into the "corruption" of one of the VMs mentioned above, and can't seem to get that one reconnected properly in order to see if it was just a protocol issue of some sort.

Even more strange, most of the active VMs appear to still be working.

On the Proxmox hosts, 'iscsiadm -m session' returns 'iscsiadm: No active sessions' which I also find very strange.

I am assuming that attempting to reload ctld will cause a disruption in connectivity, so I have not tried that (as I do not want to shut down VMs unless absolutely necessary).

There are no firewalls between the FreeNAS machine and my Proxmox hosts. Network connectivity has been confirmed.

Thanks for any help or guidance.

 

[EagleTG]

Turns out that an entire environment reboot has "resolved" the issues.

I am still digging for a root cause. I had a number of weird issues bulk-cloning some of the Proxmox VMs, and I feel like the issues I saw here stemmed from that cloning session.