iSCSI datastore not showing up on ESXI

[Ctiger256]

I have an iSCSI export on FreeNAS 11.2-beta3 consisting of a 5TB zvol exported to multiple ESXI hosts who all share a single datastore on it to store/migrate their vms. Everything worked fine.

I recently changed the underlying hardware of the machine while keeping the same disk controller and disks that host the zpool on which the zvol is (i.e., new mobo, nics, boot drive--same storage array). I reinstalled FreeNAS, upload my config file, and made the necessary tweaks to support the new hardware (changing network interfaces etc.). All works well including the other shares off my zpool (smb, nfs). I re-exported the zvol over iSCSI and here is what happens--

The iSCSi target shows up as a device in "Storage" on ESXi, but no datastores show up or mount.
Looking at vmkernel.log showed me an error "the Physical block size “16384” reported by the device naa.XXXXX is not supported. The only supported physical blocksizes are 512 and 4096"

So, I turned on the flag on the iSCSI extent on FreeNAS that causes it to suppress reporting the blocksize.

That cleared that error, but then I started getting an error message in vmkernel.log that said the datastore was being recognized only as a snapshot--

"LVM: 11136: Device naa.XXXXX:1 detected to be a snapshot:"

So, i ran esxcli storage vmfs snapshot resignature -l "[DATASTORE NAME"] to resignature it (i've also tried permanently mounting it, with same results)

That cleared that error, and so far, this is application of some well documented, if a bit esoteric, steps. (e.g., https://forums.freenas.org/index.php?threads/vmfs-partitions-gone.62031/)

But now I get the following error which I can't find any info on:

"WARNING: Vol3: 3102: [DATASTORE]/[UUID]: Invalid physDiskBlockSize 16384"
"FSS: 6092: No FS driver claimed device '[DATASTORE]/[UUID]': No filesystem on the device"

And this is where I'm stuck. I've tried this from multiple ESXI hosts with the same result. (the size set in the iSCSI extent is 512--which is what it was on the old machine where it worked) Does this mean the vmfs or underlying zvol is actually corrupted? That seems weird because everything else on the zpool imported fine and works fine.

Things are backed up so this isn't mission critical, but I'm wondering if anyone has any ideas or things I could check. I'd just as soon not have to restore all my vms. (I appreciate this is arguably more a VMware than a FreeNAS issue, but it seemed like this forum was better at dealing with these kinds of questions than elsewhere.)

I guess an alternative--short of starting over with a new zvol--is to put everything back on the old hardware, but that's a pain.

Any ideas?

Much appreciated.

 

[kdragon75]

Can you post the contents of /etc/ctl.conf from your NAS? cat /etc/ctl.conf

 

[Ctiger256]

Code:

portal-group default {

}


portal-group pg1 {

tag 0x0001

discovery-filter portal-name

discovery-auth-group no-authentication

listen 192.168.241.4:3260

listen 192.168.242.4:3260

listen 192.168.240.4:3260

option ha_shared on

}


lun "vmstorage-extent" {

ctl-lun 0

path "/dev/zvol/magstorage/iSCSI/vmstorage"

blocksize 512

option pblocksize 0

serial "000c29e84c7500"

device-id "iSCSI Disk	  000c29e84c7500				 "

option vendor "FreeNAS"

option product "iSCSI Disk"

option revision "0123"

option naa 0x6589cfc000000e2f28f3aa7d1aeeb9c9

option insecure_tpc on

option rpm 1

}


target iqn.2005-10.org.freenas.ctl:vmstorage {

alias "vmstorage"

portal-group pg1 no-authentication


lun 1 "vmstorage-extent"

}

 

[Ctiger256]

And here is a little more from the vmkernel.log

Code:

2018-10-15T14:21:58.183Z cpu1:2098855 opID=4e04fdd5)LVM: 11279: Device naa.6589cfc000000e2f28f3aa7d1aeeb9c9:1 detected to be a snapshot:

2018-10-15T14:21:58.183Z cpu1:2098855 opID=4e04fdd5)LVM: 11286:   queried disk ID: <type 1, len 22, lun 1, devType 0, scsi 0, h(id) 16332449340354091777>

2018-10-15T14:21:58.183Z cpu1:2098855 opID=4e04fdd5)LVM: 11293:   on-disk disk ID: <type 1, len 22, lun 1, devType 0, scsi 0, h(id) 2100371657885435021>

2018-10-15T14:21:58.185Z cpu1:2098855 opID=4e04fdd5)LVM: 4351: Device naa.6589cfc000000e2f28f3aa7d1aeeb9c9:1 is detected as being in volume 5bb6ce6b-9e87f7d6-9742-0c4de9a4b046 (0x451a1e51ae38)

2018-10-15T14:21:58.191Z cpu0:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.194Z cpu0:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.197Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.200Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.200Z cpu6:2104857)WARNING: NFS: 1227: Invalid volume UUID 5bb6ce6b-9e87f7d6-9742-0c4de9a4b046

2018-10-15T14:21:58.203Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.205Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.208Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.211Z cpu6:2104857)WARNING: Vol3: 3102: freeNASvmstore/5bb6ce6b-ce08b89e-d757-0c4de9a4b046: Invalid physDiskBlockSize 16384

2018-10-15T14:21:58.213Z cpu6:2104857)FSS: 6092: No FS driver claimed device '5bb6ce6b-9e87f7d6-9742-0c4de9a4b046': No filesystem on the device

 

[kdragon75]

have you rebooted the ESXi host since altering the physical reporting option? I have found that a rescan of the HBA is not enough in all cases. Im sure there is a live way of forcing a rescan and re-enumerating the properties but a reboot always works. ;)
If you have, Ill have to think on it.

 

[Ctiger256]

Yes. No dice. I've also tried from one of the ESXi hosts to mount it as a snapshot rather than resignature, but it still ends up throwing the same Invalid physDiskBlockSize error.

 

[Ctiger256]

I figured this out--and it was basically an error on my part but maybe it will help someone in the future.

Basically the problem was fixed by turning on block size reporting on the extent. The reason I had turned that off in the first place was because I had gotten the message: ""the Physical block size “16384” reported by the device naa.XXXXX is not supported. The only supported physical blocksizes are 512 and 4096" in vmkernel.log, and I assumed that was what was blocking the mount. It turns out that wasn't blocking the mount--it was the snapshot issue that was blocking it, which I fixed later (but missed the first time)--but by that time I had already turned off block size reporting which led to the "invalid physdiskblocksize error".

Once I turned blocksize reporting back on (unchecking the flag in the UI) I went back to getting the "not supported" blocksize error messages in vmkernel.log, but everything mounts fine. I just have my log spammed with those errors.

I guess I'm curious if there is a "better" way to do this that allows the volume to mount without the "not supported blocksize" errors, but I am back in business without a restore.

To the extent there's a takeaway for anyone else, it's that, at least in some situations, turning off blocksize reporting on an iSCSI extent can actually cause ESXi to not be able to mount a datastore on that extent.