NFS Share Timing Out

[BrentFreeNAS]

Hello,

I'm intermittently unable to mount a FreeNAS NFS share from ubuntu server 16.04.5LTS. It seems to happen once I restart FreeNAS. My solution in the past has been to retry continuously for a few days, to a week and eventually it would mount after several restarts. Once I'm able to mount once, I can remount with no issues from several machines, usually until I restart the NAS. In each case, I'm able to ping the NAS from the machine I'm trying to mount from, and I've also set permissions to 777 in the past to try and fix the problem, with no luck. I recently learned of the -v option when using mount and have this output:

Code:

mount.nfs: timeout set for Fri Sep 28 11:41:00 2018
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.50.200,clientaddr=192.168.80.201'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.50.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot UDP port 699
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.50.200 prog 100005 vers 3 prot TCP port 699
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: Connection timed out

The FreeNAS machine (192.680.50.200) has 192.168.80.1/24 as an authorized network in the nfs sharing section.
Does anyone have any insight as to what I'm doing wrong?

Thank you,
Brent

 

[Elliot Dierksen]

The FreeNAS machine (192.680.50.200) has 192.168.80.1/24 as an authorized network in the nfs sharing section.

That is the wrong format. It should be 192.168.80.0/24. Also, the FreeNAS IP you list (192.680.50.200) isn't valid. Are they supposed to be on the same IP network? It is hard to say without more info, but it sounds like a network configuration issues.

 

[BrentFreeNAS]

That is the wrong format. It should be 192.168.80.0/24. Also, the FreeNAS IP you list (192.680.50.200) isn't valid. Are they supposed to be on the same IP network? It is hard to say without more info, but it sounds like a network configuration issues.

The FreeNAS IP was a typo on my part. The IP is 192.168.50.200.
As for 192.168.80.1/24 being incorrect, I wasn't aware of that. It has worked in the past. I just went ahead and changed it to what you provided, and had no success. I also removed it completely from the authorized network, and put the specific IP address in the correct box, also with no success.

This configuration has worked in the past, and no changes were made to the network or configuration (until I changed the /24 address to what you provided), so I do not believe this is a network configuration issue.

If there is some sort of additional information I could provide to help narrow down the problem, please let me know. I appreciate your reply.

-Brent

 

[Elliot Dierksen]

What version of FreeNAS are you running? What is doing the routing between the IP networks you are listing (192.168.50.0/24 and 192.168.80.0/24)? I would be interested in seeing the output of netstat -in and netstat -rn from both boxes. Ideally that output would be inside a code block.

 

[BrentFreeNAS]

What version of FreeNAS are you running? What is doing the routing between the IP networks you are listing (192.168.50.0/24 and 192.168.80.0/24)? I would be interested in seeing the output of netstat -in and netstat -rn from both boxes. Ideally that output would be inside a code block.

Thanks for the reply.

I am running FreeNAS-11.1-U6. A PfSense box is handling the routing between vlans. There are firewall rules in place allowing all traffic from the client to the server, and vice versa.

output from the client:

Code:

$ netstat -in
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR	TX-OK TX-ERR TX-DRP TX-OVR Flg
ens18	  1500 0	 19455	  0	  5 0		 18723	  0	  0	  0 BMRU
lo		65536 0	 12142	  0	  0 0		 12142	  0	  0	  0 LRU

Code:

$ netstat -rn
Kernel IP routing table
Destination	 Gateway		 Genmask		 Flags   MSS Window  irtt Iface
0.0.0.0		 192.168.80.1	0.0.0.0		 UG		0 0		  0 ens18
192.168.80.0	0.0.0.0		 255.255.255.0   U		 0 0		  0 ens18

output from the server:

Code:

# netstat -in																									   
Name	Mtu Network	   Address			  Ipkts Ierrs Idrop	Opkts Oerrs  Coll											   
igb0   1500 <Link#1>	  0c:c4:7a:cf:28:fa	 1341	 0	 0	 1351	 0	 0											   
igb0	  - 192.168.50.0/ 192.168.50.200		1488	 -	 -	 1236	 -	 -											   
igb1*  1500 <Link#2>	  0c:c4:7a:cf:28:fb		0	 0	 0		0	 0	 0											   
igb2*  1500 <Link#3>	  0c:c4:7a:cf:28:fc		0	 0	 0		0	 0	 0											   
igb3*  1500 <Link#4>	  0c:c4:7a:cf:28:fd		0	 0	 0		0	 0	 0											   
lo0   16384 <Link#5>	  lo0				   5382	 0	 0	 5382	 0	 0											   
lo0	   - ::1/128	   ::1					 36	 -	 -	   36	 -	 -											   
lo0	   - fe80::%lo0/64 fe80::1%lo0			  0	 -	 -		0	 -	 -											   
lo0	   - 127.0.0.0/8   127.0.0.1			 5098	 -	 -	 5107	 -	 -		   

Code:

# netstat -rn																									   
Routing tables																													 
																																   
Internet:																														   
Destination		Gateway			Flags	 Netif Expire																		
default			192.168.50.1	   UGS		igb0																			   
127.0.0.1		  link#5			 UH		  lo0																			   
192.168.50.0/24	link#1			 U		  igb0																			   
192.168.50.200	 link#1			 UHS		 lo0																			   
																																   
Internet6:																														 
Destination					   Gateway					   Flags	 Netif Expire											 
::/96							 ::1						   UGRS		lo0													 
::1							   link#5						UH		  lo0													 
::ffff:0.0.0.0/96				 ::1						   UGRS		lo0													 
fe80::/10						 ::1						   UGRS		lo0													 
fe80::%lo0/64					 link#5						U		   lo0													 
fe80::1%lo0					   link#5						UHS		 lo0													 
ff02::/16						 ::1						   UGRS		lo0	 

-Brent

 

[Elliot Dierksen]

In my experience, firewalls are not the best choice for permissive inter-vlan routing. I would almost bet you some money that the issue is in PfSense. It might be an ACL, or perhaps a NAT (address translation) thing. Have you looked at the PfSense logs? I suspect there will be some kind of bark in there.

 

[BrentFreeNAS]

In my experience, firewalls are not the best choice for permissive inter-vlan routing. I would almost bet you some money that the issue is in PfSense. It might be an ACL, or perhaps a NAT (address translation) thing. Have you looked at the PfSense logs? I suspect there will be some kind of bark in there.

What would you recommend for rules regarding inter-vlan routing?

I've looked at the logs as best I know how. I see no record of traffic being blocked that is going from the client to the server. Pings get through with no issues. With this being an intermittent issue, do you have any guess as to what would be the problem? As far as I can tell, NAT is working fine.

 

[Elliot Dierksen]

Pings are connection-less, so they always pass more easily than TCP or UDP traffic. Generally speaking, you only need a permit from lower security to higher security. The firewall will build a connection object so the return traffic can get back. Disclaimer that I have no experience on PfSense, although I do have a lot of experience with Cisco ASA's. Is there a zone or security level associated with each interface? If so, which one is higher/more secure? I would think since this is an inside connection, you wouldn't want the IP addresses on either side to be translated. Is there some security reason you have these VLAN's separated?

 

[BrentFreeNAS]

Pings are connection-less, so they always pass more easily than TCP or UDP traffic. Generally speaking, you only need a permit from lower security to higher security. The firewall will build a connection object so the return traffic can get back. Disclaimer that I have no experience on PfSense, although I do have a lot of experience with Cisco ASA's. Is there a zone or security level associated with each interface? If so, which one is higher/more secure? I would think since this is an inside connection, you wouldn't want the IP addresses on either side to be translated. Is there some security reason you have these VLAN's separated?

I'm not sure what you mean by security level. I have most things on my network separated into vlans. I keep my NAS on it's own to limit access to it from within my network (for example, I don't want wifi guests having access). The client machine sits in my homelab, which is also isolated into it's own vlan.

 

[Elliot Dierksen]

Usually there is some kind of indication of how much a particular network segment/VLAN is trusted. Internet facing would be the lowest level/least trusted. If you are willing to put an authorized host list that allows the entire home lab network to mount NFS shares, why not use one of the un-allocated interfaces of the FreeNAS box on the homelab network? Having the NFS traffic pass through a firewall will definitely have a negative impact on throughput anyway. I am assuming you don't have a layer 3 switch or router to test removing PfSense from the equation.

 

[BrentFreeNAS]

If you are willing to put an authorized host list that allows the entire home lab network to mount NFS shares, why not use one of the un-allocated interfaces of the FreeNAS box on the homelab network? Having the NFS traffic pass through a firewall will definitely have a negative impact on throughput anyway. I am assuming you don't have a layer 3 switch or router to test removing PfSense from the equation.

Hmm, that's an interesting idea. I hadn't considered doing that. I'll have to mess around with that and see what I can do. You're correct, I do not have a L3 switch to test out removing pfsense. I still don't understand why this would cause the issue though, when it works some of the time? Once I'm able to mount, I transfer data daily for weeks and months at a time without issue.

 

[Elliot Dierksen]

The part that is easily explained is that once the connection is built, the firewall allows bi-directional traffic as long as the connection stays active. Something about how the connection is terminated when FreeNAS is rebooted is annoying the firewall, and it is not letting the traffic pass. My guess is that eventually the connection times out in PfSense, and then PfSense allows the new connection to be built. Here is an interesting idea. After rebooting FreeNAS, reboot PfSense. Then see if the NFS mount is allowed. If it does, that confirms that something about the connection is being retained by PfSense, but it isn't allowing the re-establishment of the connection. I still wouldn't want to run an NFS connection through a firewall unless I had no other choice, or some security constraint that required it.

 

[BrentFreeNAS]

The part that is easily explained is that once the connection is built, the firewall allows bi-directional traffic as long as the connection stays active. Something about how the connection is terminated when FreeNAS is rebooted is annoying the firewall, and it is not letting the traffic pass. My guess is that eventually the connection times out in PfSense, and then PfSense allows the new connection to be built. Here is an interesting idea. After rebooting FreeNAS, reboot PfSense. Then see if the NFS mount is allowed. If it does, that confirms that something about the connection is being retained by PfSense, but it isn't allowing the re-establishment of the connection. I still wouldn't want to run an NFS connection through a firewall unless I had no other choice, or some security constraint that required it.

Thanks for the explanation!

I tried rebooting in the order you described and am still unable to mount. I will give your idea about putting one of the free interfaces on the FreeNAS box on the same vlan as the homelab and see if that fixes the issue.

 

[KrisBee]

Isn't NFSv3 supposed to be firewall unfriendly, as @Elliot Dierksen 's comment implies? Mind you, changing to NFSv4 is not a guarantee of fixing time out issues as this post shows:
https://forums.freenas.org/index.ph...s-shares-timeout-on-ubuntu.50139/#post-346939