Upgraded from 9.3 to 9.10. VMWare datastores using FreeNAS became inactive/inaccessible. Reboot didn't help. Checking logs on the esx servers showed it was seeing the LUNs; however, it thought they were snapshots -- not the original LUN.
May 29 12:20:11 esx2 vmkernel: 0:00:28:47.218 cpu5:4110)LVM: 7465: Device naa.6589cfc000000f62e26a0a78138cf908:1 detected to be a snapshot:
May 29 12:20:11 esx2 vmkernel: 0:00:28:47.218 cpu5:4110)ScsiDevice: 2201: Successfully registered device "naa.6589cfc000000f62e26a0a78138cf908" from plugin "NMP" of type 0
May 29 12:20:28 esx2 vmkernel: 0:00:29:04.715 cpu2:4112)LVM: 7465: Device naa.6589cfc000000f62e26a0a78138cf908:1 detected to be a snapshot:
Found a tech note on how to force mount and/or re-label the datastores. Running the commands got:
[root@esx2 ~]# esxcfg-volume -l
VMFS3 UUID/label: 56dcabe3-590e15c1-704d-6805ca1b4fe6/riva0.1
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000f62e26a0a78138cf908:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56db464e-d006ecef-6820-6805ca1e0b2a/riva0.0
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000ff5cd958d3585892b62:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56df354c-f8fd1b2d-709e-6805ca1e0b2a/riva_ng0.1
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000b0398b9d1d045959bca:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56db4745-41382dd9-191d-6805ca1e0b2a/riva_ng0.0
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc00000024e9ca4f200ece8f00c:1 range: 0 - 1992191 (MB)
I was able to manually mount with the -m/-M options to esxcfg-volume; however, it wasn't persistent across reboots. The only thing that did work for me was to use the -r option to resignature the stores. After being resignatured, they shows up as:
snap-16058425-riva0.1
snap-23b3aaf8-riva_ng0.0
snap-68fae7c6-riva_ng0.1
(forgot to write down the name for riva0.0 before renaming it)
I then had to remove all the vms on these datastores from the inventory so it would "delete" (forgot) about the original datastore names, and thus allow me to rename the "snap" datastores back to their original names. Finally after renaming, I browsed each of the datastores and added all the vms back into the inventory.
So while I "fixed" the problem, I still don't know what caused it. Does anyone know what changed in the iSCSI configuration or implementation between 9.3 and 9.10 that could explain this? From what I gather this type of behavior usually happens when VMWare thinks the LUN has been copied or the iSCSI implementation starts presenting the LUNs using different names or in a different way. The example in the VMWare tech note was a SAN upgrade.
https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=1011387
May 29 12:20:11 esx2 vmkernel: 0:00:28:47.218 cpu5:4110)LVM: 7465: Device naa.6589cfc000000f62e26a0a78138cf908:1 detected to be a snapshot:
May 29 12:20:11 esx2 vmkernel: 0:00:28:47.218 cpu5:4110)ScsiDevice: 2201: Successfully registered device "naa.6589cfc000000f62e26a0a78138cf908" from plugin "NMP" of type 0
May 29 12:20:28 esx2 vmkernel: 0:00:29:04.715 cpu2:4112)LVM: 7465: Device naa.6589cfc000000f62e26a0a78138cf908:1 detected to be a snapshot:
Found a tech note on how to force mount and/or re-label the datastores. Running the commands got:
[root@esx2 ~]# esxcfg-volume -l
VMFS3 UUID/label: 56dcabe3-590e15c1-704d-6805ca1b4fe6/riva0.1
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000f62e26a0a78138cf908:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56db464e-d006ecef-6820-6805ca1e0b2a/riva0.0
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000ff5cd958d3585892b62:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56df354c-f8fd1b2d-709e-6805ca1e0b2a/riva_ng0.1
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc000000b0398b9d1d045959bca:1 range: 0 - 1992191 (MB)
VMFS3 UUID/label: 56db4745-41382dd9-191d-6805ca1e0b2a/riva_ng0.0
Can mount: Yes
Can resignature: Yes
Extent name: naa.6589cfc00000024e9ca4f200ece8f00c:1 range: 0 - 1992191 (MB)
I was able to manually mount with the -m/-M options to esxcfg-volume; however, it wasn't persistent across reboots. The only thing that did work for me was to use the -r option to resignature the stores. After being resignatured, they shows up as:
snap-16058425-riva0.1
snap-23b3aaf8-riva_ng0.0
snap-68fae7c6-riva_ng0.1
(forgot to write down the name for riva0.0 before renaming it)
I then had to remove all the vms on these datastores from the inventory so it would "delete" (forgot) about the original datastore names, and thus allow me to rename the "snap" datastores back to their original names. Finally after renaming, I browsed each of the datastores and added all the vms back into the inventory.
So while I "fixed" the problem, I still don't know what caused it. Does anyone know what changed in the iSCSI configuration or implementation between 9.3 and 9.10 that could explain this? From what I gather this type of behavior usually happens when VMWare thinks the LUN has been copied or the iSCSI implementation starts presenting the LUNs using different names or in a different way. The example in the VMWare tech note was a SAN upgrade.
https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=1011387
