VMware 6.5 - NFS 4.1 Mounting Single Share across multiple hosts

[tbrim]

I spent the greater part of 4 hours on the phone with VMware support just trying to mount and NFS sharepoint across 2 hosts in my cluster. The problem that I am seeing is that no matter how I try to mount the datastore, it always ends up being 2 separate mounts (one for each host). Therefore, HA migrations are not possible because it appears to be 2 different storage devices...

Has anyone here seen any such behavior before? Any ideas what I should try and look at?

 

[tbrim]

I spent the greater part of 4 hours on the phone with VMware support just trying to mount and NFS sharepoint across 2 hosts in my cluster. The problem that I am seeing is that no matter how I try to mount the datastore, it always ends up being 2 separate mounts (one for each host). Therefore, HA migrations are not possible because it appears to be 2 different storage devices...

Has anyone here seen any such behavior before? Any ideas what I should try and look at?

Oh my, I am so sorry I did not include in the body of my message, the details of the type of connection that I am trying to make... Here it goes...

I am trying to connect via NFS 4.1 as show in the pic below:

Basically I am trying to convert over to NFS 4.1 from NFS 3. The main reason that I want to convert to NFS 4.1 is because of it's ability to better handle snapshots.

 

[HoneyBadger]

Basically I am trying to convert over to NFS 4.1 from NFS 3.

Is that folder also being exported as NFSv3 right now? VMware doesn't prevent double-mounts in that case, so if it's mounted as a v3 somewhere else in the cluster a v4 mount might show up as Datastore (2)

https://docs.vmware.com/en/VMware-v...UID-E5A8D15D-B883-41AF-B9CE-0166C1F84FC1.html

 

[tbrim]

Is that folder also being exported as NFSv3 right now?

No, I have severed all connections to the datastore and mounted the datastore as a NFS 4.1 datastore.

On a side note, someone has pointed out to me that FreeNAS does not support NFS 4.1. but they do support NFS4 (I guess 4.0?). Anyways, whatever the case, it is also interesting that VMware does not support "NFS 4.0" but they do support "NFS 4.1" (WHAT???).

iXsystems is telling me to use NFS3. By the time all this is over I might end up going back to iSCSI. But I still am a little more hopeful than the average user. That is why I wanted to see what the masses had to say about the issue.

 

[Poached_Eggs]

I spent the greater part of 4 hours on the phone with VMware support just trying to mount and NFS sharepoint across 2 hosts in my cluster. The problem that I am seeing is that no matter how I try to mount the datastore, it always ends up being 2 separate mounts (one for each host). Therefore, HA migrations are not possible because it appears to be 2 different storage devices...

Has anyone here seen any such behavior before? Any ideas what I should try and look at?

YES,YES,YES - I have been hitting my head against the wall for almost a week with the same issue!! I've got 3node cluster.. it's like a game of chance if I will get all nodes to mount to 'same' datastore!! Nfs3 works with no issue...

I was just spending my evening looking into going to iscsi myself!

I swear last month this worked without issue ... But not now.. even with going to old versions of esxi and vsphere...

 

[HoneyBadger]

What's the FreeNAS and ESXi versions in question here? I want to try reproducing this.

 

[Poached_Eggs]

What's the FreeNAS and ESXi versions in question here? I want to try reproducing this.

Here is where I am - Sometimes I will get (1), sometimes (2), (3) - Total chance - Not always the same Server groupings. Followed suggestions that it was mounting with IP vs FQDN..

192.168.2.250
[root@ESXI1:~] esxcfg-nas -l
NFS is /NFS from 192.168.2.30 mounted available
SSDArray is /mnt/SSDPool/NFS from 192.168.2.33 mounted available
HPRaid is /NFS from 192.168.2.34 mounted available
DatastoreSS is /mnt/SSDMirror/NFS from 192.168.2.118 mounted available
[root@ESXI1:~]

lrwxr-xr-x 1 root root 35 Jun 1 14:02 DatastoreSS -> fc367918-78daee7f-0000-000000000000
lrwxr-xr-x 1 root root 35 Jun 1 14:02 HPRaid -> 6b74e70f-0265771f-0000-000000000000
lrwxr-xr-x 1 root root 17 Jun 1 14:02 NFS -> 01a59542-83ac6997
lrwxr-xr-x 1 root root 35 Jun 1 14:02 SSDArray -> 5eafd18e-9e60111a-0000-000000000000

192.168.2.251
[root@ESXI2:~] esxcfg-nas -l
NFS is /NFS from 192.168.2.30 mounted available
HPRaid is /NFS from 192.168.2.34 mounted available
SSDArray is /mnt/SSDPool/NFS from 192.168.2.33 mounted available
DatastoreSS (1) is /mnt/SSDMirror/NFS from 192.168.2.118 mounted available

lrwxr-xr-x 1 root root 35 Jun 1 13:45 DatastoreSS (1) -> 9cf5136d-a907d5b1-0000-000000000000
lrwxr-xr-x 1 root root 35 Jun 1 13:45 HPRaid -> 6b74e70f-0265771f-0000-000000000000
lrwxr-xr-x 1 root root 17 Jun 1 13:45 NFS -> 01a59542-83ac6997
lrwxr-xr-x 1 root root 35 Jun 1 13:45 SSDArray -> 5eafd18e-9e60111a-0000-000000000000


ESXI/FREENAS was giving it a different UUID for the symlink - but if I went to add node server into the other datastore (1) it would say it already exists - even though they are different UUID's.

---------------------------------------------------
VERSIONS
---------------------------------------------------
Have been able to consistently duplicate it is any various miss match of versions below.

FreeNAS-11.2-U4.1
FreeNAS-11.2-U3
FreeNAS-11.2-U2

VMware-ESXi-6.7.0-9484548-HPE-Gen9plus-670.10.3.5.6-Sep2018
VMware-ESXi-6.7.0-Update2-13006603-HPE-Gen9plus-670.U2.10.4.1.8-Apr2019

VMware-VCSA-all-6.7.0-13643870
VMware-VCSA-all-6.7.0-10244745


3x Cluster Node:
Manufacturer :HP
Model: ProLiant DL20 Gen9
CPU: 4 CPUs x Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz
Memory: 63.91 GB


My Current 2x FreeNAS servers
SuperMicro
OS Version:
FreeNAS-11.2-U4.1
(Build Date: May 10, 2019 21:33)
Processor:
2x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (32 cores)
Memory:
256 GiB

HP DL380e G8
OS Version:
FreeNAS-11.2-U2
(Build Date: Feb 14, 2019 20:8)
Processor:
2x Intel(R) Xeon(R) CPU E5-2407 v2 @ 2.40GHz (8 cores)
Memory:
160 GiB



-------------------------------------------------------------------
FWIW.
1- Centos7 with NFS4.1 server works.
2- FreeNAS NFS3 works

I was going on the notion that perhaps creating the mount with NFS3, deleting then recreating as NFS4.1 may have been a work around - but I haven't been able to reproduce this solution.

 

[HoneyBadger]

1- Centos7 with NFS4.1 server works.

Hmm. This does seem to indicate that the issue is on the FreeNAS side somehow.

My ESXi is still in 6.5 land, I'll see if I can spin up an 11.2 FreeNAS and repro it myself. Feels like bug report time though.

 

[Ender117]

No, I have severed all connections to the datastore and mounted the datastore as a NFS 4.1 datastore.

On a side note, someone has pointed out to me that FreeNAS does not support NFS 4.1. but they do support NFS4 (I guess 4.0?). Anyways, whatever the case, it is also interesting that VMware does not support "NFS 4.0" but they do support "NFS 4.1" (WHAT???).

iXsystems is telling me to use NFS3. By the time all this is over I might end up going back to iSCSI. But I still am a little more hopeful than the average user. That is why I wanted to see what the masses had to say about the issue.

yup, I ran into that as well when Iwas trying to have my ESXi host mount FN NFS share with kerberos. It just doesn't work, had to use NFS3. It's pretty laughable as NFS 4.1 was out like 10 years ago but FN still don't have it.

 

[tbrim]

So I had a great conversation with a support person from VMware concerning NFS3 vs NFS4 vs NFS4.1 vs iSCSI. I will attempt to explain as briefly as I can:

1. NFS3 (tried and true)
2. NFS4.0 never had a life worth mentioning and was very quicly replaced by NFS4.1 due to some major issues early on.
3. NFS4.1 Very solid with pay-for-products but not necessarily with FreeNAS. You know the drill concerning open source software - it's community supported which is why we speak.
4. iSCSI (tried and true) Multipathing and logging...

I never really thought about it much, but it is true that iSCSI protocol gives VMware important logging information and other features because of VMware's VAAI by default. Although you can still receive this via NFS (or other networking protocols) it requires the configuration of a plugin.

Furthermore talking with VMware, and having had some experience with iSCSI myself (and a great one at that), I, like many other people in the world was very quick to jump on the networking protocol bandwagon thinking that it was supposed to be better. Then the other day I went into my FreeNAS box and decided to have another look at iSCSI configuration. I setup the target, portal, initiator, authorized access (CHAP), device extent and then associated the target with the extent. It wasn't long afterward that I felt like I was home again after a long rocky journey into NFS land...

Here is when it clicked for me... It happened when I went into the ESXi, defined the new CHAP password and the clicked the discover button in VMware's GUI... Aahh! it just works!!!

You see friends, when you properly configure iSCSI, it is a loyal relationship that you establish between 2 pieces of hardware. It's not about understanding what one device needs to say to the other in order to be heard. It has more to do with the design of the devices. That's why there are so many pieces to the puzzle of iSCSI. You (the creator) are able to establish the design of both the host as well as the storage device so that they naturally communicate (according to your design). ESXi desires a connection and as soon as the iSCSI device is online, it just happens. The only way to break iSCSI is to mis-configure or alter the rules in such a way that the host and the SAN disagree on what is right and what is wrong. Selah... This is getting deep now... The only other way that it doesn't communicate is when the physical media is no longer present or has been corrupted, but it's all because of the failure of both sides to see truth. And what is the truth? Well, the truth is what the designer intended... If the designer intended communication he would have made a way for it.

Stop... Dave....... Dave... Stop....... Stop........ Dave............

Ok, I'll stop now...

Parallels are so fun...

 

[RegularJoe]

I had the same issue yesterday and something I changed made it work, I will try to figure it out:

FreeNAS : 11.2-U8
VMware : ESXi 6.5 update 3 Build 15177306
vCenter: 6.5.0.32000 Build 15259038

NFS Share:
x All Dirs
Maproot User : root
Maproot Group : wheel
security : sys

NFS Service:
x Allow non-root mount
x enable NFSv4

restart the NFS service

I did notice in the service console the error "nfsd: can't register svc name" when saving the NFS service settings or restarting it.

I did make another nfs v3 dataset/share and had that setup first with VMware mounting it. Not sure why doing a NFS v3 folder makes it happy but so far that is the last man standing. :-(

 

[RegularJoe]

I think I have it figured out now... if you apply Authorized networks it breaks something. If I have nothing and only bind NFS to my two 10 gig NICs and leave the 1gb nic for management it works fine. when I try to say the share has two authorized networks "192.168.201.0/24 192.168.202.0/24"
I now get exactly what you describe with NFS v4.1 from vCenter or from a single host and add the mount to other hosts via vCenter.