The real issue is that you had a knowledge gap you didn't know about. Nothing new on this forum, a lot of people don't know a lot of stuff they didn't realize they didn't know. That's the whole freakin' problem. People don't know what they don't know and don't want to do the required homework to figure out what they don't know.
I'm the first to acknowledge that there are a lot of things I don't know - this may be one of them.
That's kind of why I went looking for help/advice...
Never seen that error before.. except for people that use VMs. Still sure its a FreeNAS problem?
That sure does sound like a ESXi problem...
Pretty confident.
OpenSolaris/Nexenta has no problems neither has any Linux distribution I tried.
I only experience problems with FreeNAS/FreeBSD.
Of course, that doesn't mean that the others aren't working around a bug in ESXi, but it seems less likely that they do.
The only time I've seen geometry errors(aside from the standard USB geometry error on FreeNAS bootup) is bad hard drives.
That is not the case here. Disk works fine outside out the server, in the server and for other VM's (using the same device mapping). It's only an issue with ESXi+Physical RDM and FreeNAS.
So big picture, the issue is with ESXi and how it virtualizes stuff and not FreeNAS/FreeBSD. And like many people, they'll assume its a FreeNAS issue. Then they'll go to something else(Windows, Linux, whatever) and the problem will magically go away further proving their (incorrect) assumption that FreeNAS is the problem. But correlation is not causation. And if you dig deep enough, you'll probably find something is not quite right with ESXi.
I came here after confirming as much as I could that this wasn't a problem with other OS's - in the same config. Rather it seems pretty specific to FreeBSD in this particular scenario - physical RDM's.
This may be a ESXi bug/feature, but given that several other OS's work in the exact same config and are able to detect the geometry correctly it may just be that I am correct and there's something unique about the way FreeBSD/NAS read the geometry.
There isn't anything blowing my hair back with ESXi and RDM. The issues are very complex, not solved by the average user, and for most people not obvious until they find they've screwed up and lost their zpool. I did play this ESXi game in Dec 2012. I spent 3 weeks experimenting iwth it trying to figure out if RDM and PCI Passthrough can work on ESXi. I gave up after 3 weeks because it became very obvious that RDM just won't work and my hardware didn't want to behave with PCI Passthrough. Not surprising considering my hardware wasn't purchased for ESXi and PCI passthrough in particular.
Sorry you spent so long and where not able to get it to work.
My setup worked very well with minimum effort setting up.
That is until last week when I lost a drive and decided that I wanted to expand capacity rather than do a direct replacement.
I am confident that I could have replaced my drive with a 2TB drive without any issues as I could have stayed with the old RDM setup.
As it is now I will just drop this whole thing.
It's not worth it for me to spend time on trying to fixing something that isn't considered broken by others.
I will retire the FreeNAS setup as soon as I'm done migrating the data to a different VM running Nexenta on the new drives.
- - - Updated - - -
cyberjock, thank you for taking the time to reply!