NFS and iSCSI share not recognized by ESXi 6.0 U2 - solved

[maul0r]

Good evening,

Situation: I had a running old freenas box w/o IPMI capability. I now migrated freenas to my Gen8 Microserver. I have successfully imported my volume and and created another striped volume from two SSDs for ISCSI. I purposely didn't backup my configuration because I wanted a fresh start without any garbage configuration.
I have set up a cisco switch with several VLANs.
Freenas is in vlan 30 and reachable
ESXi Hosts are trunked. ESXi host can ping freenas.
The wizard is useless if the folder that is to be shared exists already (at least it seems that way).

Problem 1: NFS
Manually set up NFS Share.
Service:
# of Servers 4,
Bind IP to first interface (there are two available on the same subnet)
Enable NFSv4 (ESXi 6 uses NFSv4 I believe)
Share:
Path: /mnt/vol1/Files (This folder has been imported from the old NAS and contains lots of ISOs.)
All Directories: yes
Mapall User: nobody
Mapall Group: nobody

vmkernel.log on ESXI says "Unable to connect to NFS Server"
My MAC connects (slowly) to the same path.

I am at a loss.

showmount -e on my mac says: RPC failed:: RPC: Timed out
from finder, "Connect to Server" and then using the complete path (nfs://X.30.0.3/mnt/vol1/Files) it works.
from finder, "Connect to Server" to and then using only the server, doesn't work. It tells me that I don't have permissions.

I'll post the ISCSI problem once I can get NFS running.

Could somebody please assist me in troubleshooting this? I clearly don't have enough understanding of this to do this by myself. PS: /mnt/vol1/Files has permissions set to nobody recursive.

Any help is greatly appreciated. Let me know if you need more output.

 

[Mirfster]

Bind IP to first interface (there are two available on the same subnet)

I would suggest isolating NFS traffic on a separate NIC and Subnet.

Path: /mnt/vol1/Files (This folder has been imported from the old NAS and contains lots of ISOs.)

Is that leading to a viable DataSet and are the permissions set correctly?

I'll post the ISCSI problem once I can get NFS running.

So you are wanting to run both NFS and ISCI? Can I ask why?

Can you post more info about your System Configuration and Setup?

 

[maul0r]

I would suggest isolating NFS traffic on a separate NIC and Subnet.

The Nas is on vlan 30 - I didn't find any way to tell esxi which vmkernel port to use for Nfs traffic. Any tipps ?
Edit: I reset the entire network configuration and only connected a nic directly connected to vlan 30. FreeNas can ping the esxi host - still no joy. I'd like to rule out a network misconfiguration.

Is that leading to a viable DataSet and are the permissions set correctly?

Considering the fact that my MAC does connect to the NFS share I would say yes.
mnt/vol1/Files exist, owner user & group are both nobody and rwx is set for owner and group, rx is set for other.

So you are wanting to run both NFS and ISCI? Can I ask why?

ISCSI, I am told is more performant than NFS is for VMs. Furthermore, I only use NFS because I want to use the ISOs that I have collected on my every day NAS and I need those to be in a datastore on esxi in order to be able to mount them as CD for VMs.

Can you post more info about your System Configuration and Setup?

Sure, which particular part of it?
I am using a CISCO 3750-E 24 port switch that connects freenas and two esxi boxes together.
freenas is connected with two ports both vlan 30. the two esxi boxes are connected with two nics which are trunked.
The esxi boxes are using virtual switches. They have a the following vmkernel ports:
VLAN 20 - Mgmt Traffic
VLAN 30 - ISCSI
VLAN 50 - vMotion
VLAN 60 - Fault Tolerance

The MAC that connected to the nfs share is on vlan 10. Full routing between all vlans. the esxi host can ping freenas but i suspect it does so from vlan 20.

 

[Rapha]

Hi there,

having a similar setup here and using NFS3 and iSCSI. Everything works fine.

ESXi 6.0 does not support NFS4, only NFS 3 and 4.1 is supported. FreeNAS can only serve NFS3 and 4 not 4.1 in current 9.10-Release. I'm not sure about FreeNAS 10.
Try connecting ESXi using NFS3-setting in FreeNAS.

 

[jgreco]

The Nas is on vlan 30 - I didn't find any way to tell esxi which vmkernel port to use for Nfs traffic. Any tipps ?

Put a vmkernel port on the NFS server network and that's it.

Edit: I reset the entire network configuration and only connected a nic directly connected to vlan 30. FreeNas can ping the esxi host - still no joy. I'd like to rule out a network misconfiguration.

And vmkping can ping the FreeNAS host?

What shows up in /var/log/vmkernel.log? You can do "tail -f /var/log/vmkernel.log &" while also doing "esxcli storage nfs add --host 10.2.3.4 --share=/mnt/pool/stuff --volume-name=datastorename" and see what's what.

ISCSI, I am told is more performant than NFS is for VMs. Furthermore, I only use NFS because I want to use the ISOs that I have collected on my every day NAS and I need those to be in a datastore on esxi in order to be able to mount them as CD for VMs.

Yes, yes, that's fine, everybody does that I think. Otherwise maintaining an ISO library is crazy inconvenient.

Sure, which particular part of it?
I am using a CISCO 3750-E 24 port switch that connects freenas and two esxi boxes together.
freenas is connected with two ports both vlan 30. the two esxi boxes are connected with two nics which are trunked.
The esxi boxes are using virtual switches. They have a the following vmkernel ports:
VLAN 20 - Mgmt Traffic
VLAN 30 - ISCSI
VLAN 50 - vMotion
VLAN 60 - Fault Tolerance

The MAC that connected to the nfs share is on vlan 10. Full routing between all vlans. the esxi host can ping freenas but i suspect it does so from vlan 20.

Then put a vmkernel interface on 30.

If you're having trouble grasping your vSphere IP networks, you can intentionally break inter-vlan routing by getting rid of the default route on ESXi and then sitting there with vmkping to make sure that the things you think should be pingable are in fact pingable.

 

[maul0r]

I kept getting that ESXi was unable to find the NFS server, which made me dig into firewall and nfs client services on the esxi host. Out of a hunch I checked my DNS settings. I wasn't aware that NFS is so DNS reliant, but since I had changed my ESXi DNS Server to the prospective DC it didn't get a DNS answer. Once I switched it back to my router, NFS Datastores mounted just fine. --> Problem solved

However, I am still unable to use the "all directories" feature. I expected that if I share the "/mnt/vol1/Files" Dataset and share it with "all directories" set, that I should be able to mount "/mnt/vol1/Files/Iso/OS" and other folders in the sub structure. That didn't work though. Any tipps on that? My workaround is sharing out each individual subfolder and getting really creative in describing the allowed networks (0.0.0.0, 10.0.0.0, 10.30.0.0 etc..)

Put a vmkernel port on the NFS server network and that's it.

Does ESXi always use the 'closest' NIC? For VM Storage Traffic you can set a checkbox which I believe is iSCSI only, but for NFS?

If you're having trouble grasping your vSphere IP networks, you can intentionally break inter-vlan routing by getting rid of the default route on ESXi and then sitting there with vmkping to make sure that the things you think should be pingable are in fact pingable.

That would be a way to ensure storage traffic stays on the correct vlan, you are right.

ESXi 6.0 does not support NFS4, only NFS 3 and 4.1 is supported. FreeNAS can only serve NFS3 and 4 not 4.1 in current 9.10-Release. I'm not sure about FreeNAS 10.
Try connecting ESXi using NFS3-setting in FreeNAS.

I was trying it with NFS 3 - but thanks for that! Aren't 4 and 4.1 interoperable?


My iSCSI issues I had, may have been because of instability or the same DNS issue. I reinstalled esxi without the custom ixgbe vib, mounted the iSCSI Datastores and then installed the ixgbe drivers. No issues so far.

All in all, thanks for your support!

 

[Rapha]

I was trying it with NFS 3 - but thanks for that! Aren't 4 and 4.1 interoperable?

NFS4.1 should be compatible to 4 but VMwares implementation relies on NFS 4.1 and will not connect to NFS4 shares.

 

[jgreco]

I kept getting that ESXi was unable to find the NFS server, which made me dig into firewall and nfs client services on the esxi host. Out of a hunch I checked my DNS settings. I wasn't aware that NFS is so DNS reliant, but since I had changed my ESXi DNS Server to the prospective DC it didn't get a DNS answer. Once I switched it back to my router, NFS Datastores mounted just fine. --> Problem solved

However, I am still unable to use the "all directories" feature. I expected that if I share the "/mnt/vol1/Files" Dataset and share it with "all directories" set, that I should be able to mount "/mnt/vol1/Files/Iso/OS" and other folders in the sub structure. That didn't work though. Any tipps on that? My workaround is sharing out each individual subfolder and getting really creative in describing the allowed networks (0.0.0.0, 10.0.0.0, 10.30.0.0 etc..)

Unclear why that wouldn't work. Works fine here, just tried. NFS has been able to do that like forever.

The exception to that...

Sharing a dataset with other datasets mounted on top of it does not result in the client knowing about the other datasets. You would need to mount the sub-datasets on the client.

Also: ESXi will happily let you specify an IP address instead of a domain name when performing a mount.

Does ESXi always use the 'closest' NIC?

Yes, I can't even begin to imagine how else it could work. You're either on the relevant network or you're not. If not, then the interface hosting the default route would need to be used.

For VM Storage Traffic you can set a checkbox which I believe is iSCSI only, but for NFS?

No, there is no checkbox for VM storage traffic. There is for management and vmotion.

iSCSI can be bound to a vmk (or multiple vmk's) but that's a function of the iSCSI subsystem, not the networking.