NFS sharing is not visible because a DNS issue on FreeNAS

[Ophion]

I have a Proxmox 5.1-35 (172.16.10.101) and FreeNAS 11.0-U4 (172.16.10.10) server, everything was fine for about 3 weeks until I started getting this message on Proxmox "NFS Shares: storage is not online (500)" on WebGUI but using command line I was able to mount the NFS on FreeNAS. So I posted the issue on Proxmox's forum: https://forum.proxmox.com/threads/nfs-shares-storage-is-not-online-500-why.38778/ and turns out this happened to another user https://forum.proxmox.com/threads/nfs-shares-storage-is-not-online-500.11121/. After a long time of posting and reading, I finally realized the problem was, in my case, FreeNAS does DNS lookup (using defined DNS server on WebGUI) to somehow allow NFS mounts. As previous links suggest, a good idea is to add an entry in the field "Host name data base" as follows:

https://imgur.com/CmqvhXi

However, when I run

Code:

nslookup pve01.eleka.c

it returns nothing, the query just time out. Why isn't this file able to solve an IPv4 address defined on /etc/hosts? This is critical for starting all VMs and Containers on Proxmox. Have been stuck here for weeks. Thanks in advance.

 

[Nick2253]

After a long time of posting and reading, I finally realized the problem was, in my case, FreeNAS does DNS lookup (using defined DNS server on WebGUI) to somehow allow NFS mounts.

If you use a DNS name to allow your NFS clients, then FreeNAS has to do a DNS lookup to determine the IP address. What happens when you allow the Proxmox server via IP?

If the problem is as you suggest it is, the solution is to fix your DNS server. What are you using for you DNS server? Also, the nslookup command queries a DNS server. Defining a hostname in /etc/hosts on your local FreeNAS has nothing to do with your DNS sever.

 

[Ophion]

If you use a DNS name to allow your NFS clients, then FreeNAS has to do a DNS lookup to determine the IP address. What happens when you allow the Proxmox server via IP?

If the problem is as you suggest it is, the solution is to fix your DNS server. What are you using for you DNS server? Also, the nslookup command queries a DNS server. Defining a hostname in /etc/hosts on your local FreeNAS has nothing to do with your DNS sever.

I'm not pointing a name for FreeNAS for Proxmox, e.g. freenas.eleka.c. On Proxmox WebGUI you just need to click on Datacenter->Storage->Add->NFS and on field Server I put 172.16.10.10. For instance, I have a NFS share named Backups which contains all critical data such as virtual disks, VMs and Containers backups. But VMs and CTs won't autostart. However if I run the mount command using command line it will let me to run "qm start 100". Now, If I put the primary DNS server (172.16.10.1) on FreeNAS WebGUI I won't be able to run the command "mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups", it will time out. I'm very tired about this issue.

 

[Nick2253]

I'm going to suggest we take a step back and reset ourselves here. Let's start troubleshooting back from the beginning.

Step one, we need to get everything in a sane configuration. FreeNAS is not a DNS server, so you should point FreeNAS to your primary DNS server.

Step two, we need to get DNS entries on your primary DNS server for your servers: Proxmox and FreeNAS.

Step three, we need to understand what you've set up for the NFS shares. Can you provide a screenshot of the NFS settings, including the advanced settings?

 

[Ophion]

 

[Ophion]

I'm having a lot of troubles to post embedded images. In order to save your time, could you please tell me a proper way to post embedded pictures? I already taken some screenshots to the current configs.

 

[Ophion]

Step one, we need to get everything in a sane configuration. FreeNAS is not a DNS server, so you should point FreeNAS to your primary DNS server.

The thing is my DNS server is also my PDC which is virtualized in Proxmox as a VM, but as any VM or CT do autostart, when FreeNAS trying to reach 172.16.10.1 with a DNS query it doesn't find it because the PDC is down. Anyway, I have already added 172.16.10.1 (DNS and PDC) as primary DNS server on FreeNAS as follows:

Step two, we need to get DNS entries on your primary DNS server for your servers: Proxmox and FreeNAS.

Done on my PDC which runs under Windows 2012 Server:

Step three, we need to understand what you've set up for the NFS shares. Can you provide a screenshot of the NFS settings, including the advanced settings?

The NFS Storage on Proxmox:

The output for

Code:

mount | grep nfs

and finally, the way I find the VMs and CTs at startup (ALL DOWN), because no one does autostart since the NFS shows "NFS Shares: storage is not online (500)", even with a correct output for mount | grep nfs which indicates there is a NFS mount. This is so confusing for me.

 

[Ophion]

Wasn't able to attach this one to the prior post. When I turn off the physical server and start it again, Proxmox is not able to go to the NFS share and use the virtual disks to start VMs and CTs. However mount | grep nfs shows the opposite.

 

[Nick2253]

Based on what you've provided to me so far, I'm leaning towards this being a Proxmox problem. The fact that you are able to mount the share manually seems to indicate that FreeNAS has no problem letting Proxmox access the share. So, the fact that the automount is not working seems to be on the Proxmox side.

Furthermore, in the FreeNAS thread you quoted as an explanation for your problem, the user was unable to mount the share in any capacity. The fact that you can manually mount your share points to your situation being very different from that case. That makes me wonder if your problem might have something to do with any differences between the manual command you are running and the automatic command. How are you manually mounting the share?

The other problem you have is an architecture problem. From your Proxmox screenshots, it doesn't look like you have a second domain controller set up. I would recommend configuring either a stand-alone domain controller (Microsoft best practices), or at least a second domain controller VM to both prevent network downtime when one VM is offline, and to provide redundancy if either domain controller is corrupted. Lastly, you might want to look into running a read-only DNS server to provide DNS when your hypervisor is offline. Personally, I use BIND on my pfSense firewall to provide the capability (it syncs from my PDC), and if I have any problems with my hypervisor, and my domain conrollers are offline, it still allows me to access other network resources and the internet without having to manually tweak anything.

 

[Ophion]

They say (Proxmox forum) the problem is on FreeNAS side XD. Anyway, a common point here is to use pfSense with bind9 as a read-only DNS server, but the thing is I don't have a pfSense router, I'm just running pfSense as a virtual machine. So, if my hypervisor is down, then pfSense will be too. I live in Cuba, so buying a pfSense router is not even an option because the embargo. Anyway, thanks for your help. I'm considering doing a fresh install of Proxmox to see what's going on. That won't be a problem since Backups are stored in FreeNAS, just will have to create the VMs and CTs manually. It is healthy to mark this thread as solved?

 

[Nick2253]

I wouldn't give up just yet! :D

Your architecture problem is not insurmountable, it's just a bit more difficult to overcome than the "typical" solutions. What are you using for your router? You could also configure BIND on the Proxmox host, or you could put BIND in a FreeNAS jail.

Let's take a look at a few more things:

1. What are the NFS setting on FreeNAS? Post a screenshot, and make sure to include all the advanced settings.
   
2. What command are you using to mount the NFS share on Proxmox? Can you post the output from /etc/fstab? (make sure to sanitize any passwords).

My prediction is that one of two things (or possibly both) are happening: (1) there's a difference between the manual mount command and the automatic mount command that's leading to problems, and (2) you are authorizing hosts in the NFS share by DNS name.

 

[Ophion]

Hi @Nick2253 I was out of work yesterday. Before posting here I was thinking to set up a DNS server on Proxmox server but has my doubts. I'm using just a single switch to communicate Proxmox and FreeNAS, but to communicate LAN, SERVERS and WAN I'm using pfSense running as a Virtual Machine on Proxmox. Every time I power on the physical Proxmox server I have to sit on the servers room and mount the NFS manually because pfSense won't autostart and I won't have any access from LAN without pfSense running. Anyway, following your instructions, I took several pictures about my settings:

1. What are the NFS setting on FreeNAS? Post a screenshot, and make sure to include all the advanced settings.

Here are some pictures of FreeNAS side

- Storage tab

- Sharing tab -> NFS shares

- A view of Backups NFS share in "Edit" mode (Advanced). I have to say there is only a defined Authorized networks but there is no data for Authorized IP addresses or hosts, maybe this could be the issue

2. What command are you using to mount the NFS share on Proxmox? Can you post the output from /etc/fstab? (make sure to sanitize any passwords).

Now the pictures for Proxmox side (if you need more pictures from FreeNAS just let me know and I would be pleased to post more)

- Adding a new NFS storage on Proxmox. When I put the Server param (IP address), automatically the Export param lists all the NFS shares on FreeNAS, which means Proxmox can "see" FreeNAS and also its NFS shares without any problem. Why doesn't do the same at startup?

- When the NFS ISOS is added, Proxmox even shows the content and general data for this NFS share at FreeNAS

- The last thing and a curious picture is the content of /etc/fstab file. There is no entry for any NFS share such as Backups and ISOS. At the very beginning when I started to config Proxmox and FreeNAS (everything running without any issue) this file was also the same way. Maybe Proxmox keeps a different version or other data about the NFS in other location. Maybe, by not having any entry about NFS in this file is the reason Proxmox won't mount the NFS shares at startup and of course the VMs and CTs won't autostart

Again thanks Nick! I've come this far because of your support.

 

[Nick2253]

I was hoping Proxmox used fstab for these mounts, because it makes troubleshooting much easier.

At this point, I'm somewhat confident that the problem is with the automount command that Proxmox is using, whether there's a timing issue that's breaking the mount, or an unsupported option, or something weird. However, I'm out of my depth with Proxmox, and I'd suggest creating a post on the Proxmox forum with a title like: "Automount not working, but manual mount from shell works fine".

 

[Ophion]

Hahahahah bouncing around between Proxmox forum and FreeNAS forum but hey I have to say in both I had a lot of help. I guess my quest still continues and next to do is to create that thread on Proxmox. Thanks again @Nick2253 !

 

[Ophion]

If you want to follow the topic: https://forum.proxmox.com/threads/a...ual-mount-from-shell-works-fine-sequel.39168/