9.2.1.6 NFS Daemon / ESXi / DNS Required?

[hidden72]

I upgraded to 9.2.1.6-Release this weekend, and while I was at it, did some additional ad-hoc maintenance in the lab. I noticed one oddity that I haven't experienced before.

Essentially, the ESXi hosts weren't able to mount the NFS share on the newly-upgraded FreeNAS box. The ESXi hosts could ping each other and the FreeNAS server.. and vice-versa, but it wouldn't mount. VMware showed the NFS share as invalid/inaccessible/unmounted (or something like that). I have a handful of VMs stored on this NFS share and they were all temporarily orphaned/inaccessible.

I ended up tracking the problem down to a non-working DNS server. (Actually, the DNS Server's datastore was hosted on the FreeNAS box, so I had a bit of a chicken/egg problem). I was able to get a DNS server up and running - and a quick stop/start of the NFS service in FreeNAS and everything came back to life.

I'm pretty sure I've done similar maintenance in the past and don't remember DNS being required for an NFS connection to be successful. This is pretty easy to test. Change the DNS server in FreeNAS to something non-existent and then reboot. When it comes back up, ESXi won't be able to mount an NFS datastore.

When I get some time, I'll try and give it a go with an older FreeNAS version. Just wanted to see if this is expected behavior.

 

[cyberjock]

Well, generally speaking, virtualizing your networking services is just a bad idea. You end up with a chicken/egg problem just like you saw. So I'd say this was probably expected for conditions.

 

[c32767a]

I'm pretty sure I've done similar maintenance in the past and don't remember DNS being required for an NFS connection to be successful. This is pretty easy to test. Change the DNS server in FreeNAS to something non-existent and then reboot. When it comes back up, ESXi won't be able to mount an NFS datastore.

If you specify the mount points in the NFS datastore with hostnames, then DNS will need to be functional for ESX to find and complete the mount.

I can't say why it's worked in the past. Possibly your ESX boxes had access to the server name in a DNS cache or another server somewhere.

At $dayjob we run lots of infrastructure services in ESX. The key is to ensure that you can bootstrap your VMWare infrastructure when fundamental services are not available. We use DNS hostnames for NFS services and other ESX configuration, but we host DNS on a physical box outside the ESX environment.. Another alternative is to use all IP addresses, which isn't as flexible or maintainable, but doesn't require as much physical hardware.

 

[hidden72]

If you specify the mount points in the NFS datastore with hostnames, then DNS will need to be functional for ESX to find and complete the mount.

I can't say why it's worked in the past. Possibly your ESX boxes had access to the server name in a DNS cache or another server somewhere.

At $dayjob we run lots of infrastructure services in ESX. The key is to ensure that you can bootstrap your VMWare infrastructure when fundamental services are not available. We use DNS hostnames for NFS services and other ESX configuration, but we host DNS on a physical box outside the ESX environment.. Another alternative is to use all IP addresses, which isn't as flexible or maintainable, but doesn't require as much physical hardware.

The ESXi hosts mount the FreeNAS NFS share using IP addresses... no hostnames/DNS involved there. Additionally, it's the ESXi hosts that initiate the connection to the FreeNAS share... which is the puzzling part about it. Why would FreeNAS need DNS enabled in order to receive an NFS connection?

You're absolutely right regarding the ability to bootstrap the VMware infrastructure. $stuff broke until I could get the NFS mount up and running - was just surprised to find out that DNS was the cause.

 

[c32767a]

The ESXi hosts mount the FreeNAS NFS share using IP addresses... no hostnames/DNS involved there. Additionally, it's the ESXi hosts that initiate the connection to the FreeNAS share... which is the puzzling part about it. Why would FreeNAS need DNS enabled in order to receive an NFS connection?

You're absolutely right regarding the ability to bootstrap the VMware infrastructure. $stuff broke until I could get the NFS mount up and running - was just surprised to find out that DNS was the cause.

Did you specify your ESX hosts in your exports file on the FreeNAS as hostnames or IPs? The NFS daemon will try and reverse an IP into a DNS name to compare against the exports file when the mount request comes in. In theory, it should try to match IPs first before going to DNS, but perhaps it's doing a DNS lookup in all cases.. If that behavior of the NFS server changed between versions, that might explain what you're seeing.

 

[hidden72]

So I did a little more digging and have a better understanding of the behavior. This condition only occurs when FreeNAS has a configured DNS server address that is currently unreachable. It does not happen when there aren't _any_ DNS servers configured, nor does it happen when FreeNAS is configured with a reachable DNS server. I tested 4 versions and they all have consistent behavior (9.1.1, 9.2.1.2/9.2.1.4/9.2.1.6).

The net-net here: if you add a DNS server to FreeNAS, it must be reachable when FreeNAS boots, otherwise NFS won't work until you either do one of the following:
a.) delete the DNS server & restart NFS, or
b.) add a rechable DNS server & restart NFS, or
c.) go to Network / Host Name Database and manually add host entries for the ESXi servers (what I ended up doing).