SOLVED NFS Lock Manager missing

[Oko]

I am running two FreeNAS 9.2.1.5 file servers. I have asked already many questions about this machine so just look back posts if you need exact specifications. I tried to mount /home directory from my bigger server (which was behaving well in the past) on my new OpenBSD shell gateway via NFS. After getting the message

Code:

NFS Portmap: RPC: Program not registered

I start poking around little bit and discovered something interesting.

From my OpenBSD gateway I get

Code:

# rpcinfo -p gaia.int.autonlab.org
  program vers proto  port
    100000    4  tcp    111  portmapper
    100000    3  tcp    111  portmapper
    100000    2  tcp    111  portmapper
    100000    4  udp    111  portmapper
    100000    3  udp    111  portmapper
    100000    2  udp    111  portmapper
    100000    4    7    111  portmapper
    100000    3    7    111  portmapper
    100000    2    7    111  portmapper
    100005    1  udp  4002  mountd
    100005    3  udp  4002  mountd
    100005    1  tcp  4002  mountd
    100005    3  tcp  4002  mountd
    100003    2  tcp  2049  nfs
    100003    3  tcp  2049  nfs
    100024    1  udp  4000  status
    100024    1  tcp  4000  status

and the issue is definitelly not the firewall because I get the same on the server

Code:

[root@gaia ~]# rpcinfo -p localhost                                         
  program vers proto  port  service                                       
    100000    4  tcp    111  rpcbind                                       
    100000    3  tcp    111  rpcbind                                       
    100000    2  tcp    111  rpcbind                                       
    100000    4  udp    111  rpcbind                                       
    100000    3  udp    111  rpcbind                                       
    100000    2  udp    111  rpcbind                                       
    100000    4 local    111  rpcbind                                       
    100000    3 local    111  rpcbind                                       
    100000    2 local    111  rpcbind                                       
    100005    1  udp  4002  mountd                                         
    100005    3  udp  4002  mountd                                         
    100005    1  tcp  4002  mountd                                         
    100005    3  tcp  4002  mountd                                         
    100003    2  tcp  2049  nfs                                           
    100003    3  tcp  2049  nfs                                           
    100024    1  udp  4000  status                                         
    100024    1  tcp  4000  status 

Now on the smaller file server which had to have its firewall disabled to work around one of the bugs I get what I would expect

Code:

[root@neill-zfs ~]# rpcinfo -p localhost                                     
  program vers proto  port  service                                       
    100000    4  tcp    111  rpcbind                                       
    100000    3  tcp    111  rpcbind                                       
    100000    2  tcp    111  rpcbind                                       
    100000    4  udp    111  rpcbind                                       
    100000    3  udp    111  rpcbind                                       
    100000    2  udp    111  rpcbind                                       
    100000    4 local    111  rpcbind                                       
    100000    3 local    111  rpcbind                                       
    100000    2 local    111  rpcbind                                       
    100005    1  udp  4002  mountd                                         
    100005    3  udp  4002  mountd                                         
    100005    1  tcp  4002  mountd                                         
    100005    3  tcp  4002  mountd                                         
    100003    2  tcp  2049  nfs                                           
    100003    3  tcp  2049  nfs                                           
    100024    1  udp  4000  status                                         
    100024    1  tcp  4000  status                                         
    100021    0  udp  4001  nlockmgr                                       
    100021    0  tcp  4001  nlockmgr                                       
    100021    1  udp  4001  nlockmgr                                       
    100021    1  tcp  4001  nlockmgr                                       
    100021    3  udp  4001  nlockmgr                                       
    100021    3  tcp  4001  nlockmgr                                       
    100021    4  udp  4001  nlockmgr                                       
    100021    4  tcp  4001  nlockmgr  

What the hack am I doing wrong here? I tried to have configurations as close as possible. The only difference one bigger file server got to 9.2.1.5 via upgrade from 9.2.1 while the smaller one was upgraded from the fresh installation 9.2.1.4.

 

[solarisguy]

Try just restarting NFS service. It appears that there was a bug https://bugs.freenas.org/issues/4290 and may be somehow it is still present in the upgraded system.

 

[dansband]

Did you find a solution? I have the same problem with OpenBSD and it didn't work out by restarting.

 

[Ericloewe]

Did you find a solution? I have the same problem with OpenBSD and it didn't work out by restarting.

OpenBSD is a completely different project. FreeBSD != OpenBSD

 

[Oko]

OpenBSD is a completely different project. FreeBSD != OpenBSD

His NFS client machine runs OpenBSD Don't assume that people who hang here are idiots.

 

[Oko]

Did you find a solution? I have the same problem with OpenBSD and it didn't work out by restarting.

Nope it still doesn't work with OpenBSD. Now in the mean time lock manager has magically reappeared but I still get that ugly

Code:

NFS Portmap: RPC: Program not registered

error only on OpenBSD NFS client. Since I had only one OpenBSD NFS client which was to be used as shell gateway I ended up creating tiny home directories locally so that people can save their dotfiles and log further but if it was more than one machine it would be a big deal. Last week I set up one NFS server on Red Hat and that one worked flawlessly including port restricting to work with the firewall. My feeling is that NSF needs to get little love on FreeNAS, enabling NFSv4 would be nice too. I also have few ideas about mass monitoring of FreeNAS system and backups. FreeNAS feels little like a HomeNAS rather than an enterprise solution to me (after 5-6 months of use). I have 2 FreeNAS servers but I am little scared to deploy more so I went back to hardware and software RAID 6 on Linux. I am also experimenting with Hammer on DragonFlyBSD but the project is tiny and they are missing lots of "enterprise" infastructure.

 

[Ericloewe]

His NFS client machine runs OpenBSD Don't assume that people who hang here are idiots.

Diagonal reading rearing its ugly head... My apologies for jumping the gun.

 

[solarisguy]

Oko, could you please on both FreeNAS servers execute: egrep 'nfs|rpc' /var/tmp/rc.conf.freenas

And: ps awx | egrep 'nfs|rpc'

 

[Oko]

Oko, could you please on both FreeNAS servers execute: egrep 'nfs|rpc' /var/tmp/rc.conf.freenas

And: ps awx | egrep 'nfs|rpc'

Code:

[root@gaia] ~# ps awx | egrep 'nfs|rpc'
2670 ??  Is        0:08.00 /usr/sbin/rpcbind
2688 ??  Is        0:00.05 nfsd: master (nfsd)
2689 ??  I        62:03.35 nfsd: server (nfsd)
2700 ??  Is        0:01.60 /usr/sbin/rpc.statd -p 4000
2711 ??  Ss        1:46.78 /usr/sbin/rpc.lockd -p 4001
33458  0  S+        0:00.00 egrep nfs|rpc

Code:

[root@neill-zfs] ~# ps awx | egrep 'nfs|rpc'
2970 ??  Ss      0:00.48 /usr/sbin/rpcbind
2988 ??  Is      0:00.05 nfsd: master (nfsd)
2989 ??  I        8:26.93 nfsd: server (nfsd)
3000 ??  Ss      0:00.33 /usr/sbin/rpc.statd -p 4000
3011 ??  Ss      1:39.61 /usr/sbin/rpc.lockd -p 4001
26549  0  S+      0:00.00 egrep nfs|rpc

neill-zfs is the one where I had to switch off firewall both on the server and the clients to work even with Linux clients.

On both servers

Code:

mountd(8) bind port: 4002

I really hate keeping anything but 111, 2049, 4000, 4001, and 4002 open.

 

[dansband]

I checked in Serve UDP NFS clients and now it works! Don't know what this means so maybe it's more of a workaround.

 

[solarisguy]

In my opinion, your investigation has left only two possibilities:

1. There is some other process on port 4001 on gaia.
2. The firewall on gaia blocks port 4001.

1. If you shutdown NFS on gaia, did you check whether the port 4001 was taken by anything (netstat -na|grep 4001) ?

2. If you shutdown the firewall, does nlockmgr automagically appear on port 4001 ?

 

[Oko]

In my opinion, your investigation has left only two possibilities:

1. There is some other process on port 4001 on gaia.
2. The firewall on gaia blocks port 4001.

1. If you shutdown NFS on gaia, did you check whether the port 4001 was taken by anything (netstat -na|grep 4001) ?

2. If you shutdown the firewall, does nlockmgr automagically appear on port 4001 ?

firewall is shut on neill-zfs. I could try switching ports on neill-zfs to something else. Gaia works like a charm behind the firewall with Linux machines but had trouble with OpenBSD NFS client. Not any more after I checked UDP NFS client in the options as suggested.

 

[Oko]

I checked in Serve UDP NFS clients and now it works! Don't know what this means so maybe it's more of a workaround.

I can confirm this! I can mount both my FreeNAS file servers on NFS client running on OpenBSD :) Thank you so much!!!

 

[mopcrust]

This was helpful for me as well. I was wondering (as I am using an OpenBSD client as well) what kind of copy speeds are you getting to and from the NAS? Also what options do you have in your fstab for any optimizations? I am getting extremely slow speeds and strange disk activity.

 

[Oko]

I really could care less about the speed as I just need to mount /home directories via NFS to my OpenBSD ssh gateway so that people can log into the lab. Once they are logged they have to pick one of computing nodes anyway. Nodes run RedHat and NFS mounts can be done with far more options. From /etc/fstab I have one file server using default OpenBSD udp NFS client settings and one forcing OpenBSD to use tcp protocol to mount NFS which is default FreeNAS NFS server behavior.

Code:

gaia:/mnt/zfsauton/home /zfsauton/home  nfs  nosuid,nodev,noexec,rw,soft,intr,noatime 0 0
neill-zfs:/mnt/zfsneill/home /zfsneill/home  nfs  nosuid,nodev,noexec,rw,tcp,soft,intr,noatime 0 0

 

[mopcrust]

Thanks for getting back. I tried the above and am still getting very slow write speeds to the NFS mount. I guess I won't be able to squeeze very much more performance out of this and will likely just move back to FreeBSD which makes obtains very good read/write speeds.

 

[solarisguy]

@mopcrust, we know nothing about your setup (server hardware, network, clients, topology etc.). Your problem can be as trivial as having/using Realtek network cards...

 

[mopcrust]

This thread I think can be closed. The issue basically boiled down to a few bottlenecks in OpenBSD that made it difficult to reach optimal speed. I've sense moved the clients to FreeBSD and the issue in question is gone. Perhaps later on I can revisit OpenBSD but for now the better performance over NFS is via FreeBSD.

 

[Oko]

This thread I think can be closed. The issue basically boiled down to a few bottlenecks in OpenBSD that made it difficult to reach optimal speed. I've sense moved the clients to FreeBSD and the issue in question is gone. Perhaps later on I can revisit OpenBSD but for now the better performance over NFS is via FreeBSD.

This is not very helpful post. A few more details about bottlenecks would be surely appreciated by the community.

 

[mopcrust]

You are right Oko. I guess I was ready to move on after having spend nearly a week on trying to get this to work in a satisfactory manner. The current set up is a Thinkpad X200 that acts as a client connected to a FreeNAS Mini server running FreeNAS 9.2.1.7. The ThinkPad is Intel through and through and is connected to the Mini via Intel gigabit ethernet.

In terms of settings
The following were the options that I used on the OpenBSD client with "Serve UDP NFS Clients" enabled on the FreeNAS box.

server:/mnt/files /mnt nfs rw,soft,intr,tcp 0 0

I was able to get acceptable read speeds across the network from the server, approximately 25-30 megabytes per second. However, writing to the drive is another story. A 600 megabyte copied file starts transferring at ~15 megabytes per second and decreasing after 20 seconds or so, to ~200-300 kilobytes per second. When trying to cancel the transfer manually, the system hangs for a solid minute or so before actually cancelling the transfer.

I read a bit further and decided to try the following:

server:/mnt/files /mnt nfs rw,nodev,nosuid,soft,intr 0 0

Which actually made the situation worse. Slow file transfers again and when I tried to cancel the transfer the drive to which the file was being copied began producing a very loud audible buzzing sound (not normal at all). In order to get it to stop, I had to perform a cold shutdown.

This was under OpenBSD 5.5 (patched stable) and I read the OpenBSD man pages first concerning mount and client NFS as well as other resources scattered on the Internet and posted a similar question on several other forums. Its not a FreeNAS issue as currently I am using FreeBSD as a client on the same Thinkpad and get ~40/50 megabytes read/write speeds with the following settings:

rw,tcp,intr,noatime,nfsv3,wsize=262144,rsize=262144 0 0

which is quite acceptable for my specific use case.

OpenBSD was the first choice as reliable suspend/resume was an important consideration, but being able to connect to the file server and interact with material was a far more important priority for now. I hope this helps someone and I will continue to revisit OpenBSD to see if the issue has been resolved in a future iteration. If it is my own error in the matter, I will discover sooner or later and will gladly modify what I've written here.