NFS related login issue

[Oko]

I am running two FreeNAS 9.2.1.5 amd64 file servers capacity 21TB and 8TB. Both servers are industrial grade and have sufficient RAM and CPU power. I am having a very strange issue on the smaller server.
Namely smaller server hosts home directories for 9 of my users which are exported vis NFS on 15 RedHat computing nodes. Both computing nodes and FreeNAS file server use LDAP for directory services. LDAP is a stock OpenBSD server. The larger 21 TB server is set identically and hosts home directories for about 30 people. It is exported on the same computing nodes using the same LDAP setup.
Home directories on the 21TB server reside on a data set size 3TB. The server works like a champ. However users whose directories sits on the smaller file server started complaining about login times on computing nodes as soon as I mounted their home directories.

I nailed login problem to NSF mount and the size of data set. When I don't mount home folders login times are normal as soon as I mount home folders the login time sucks. The speed of the login increases as I decrease the size of data set which hosts home folders. However there is no practical user for home folders which sits on the top of 50GB data set and I see this problem with data sets as small as 1TB (for all home folders).

I am using lz4 compression. De-duplication is disabled. I put a hard bound 2TB on the size of data set holding home directories and children.


Has anybody seen something crazy like the above? Thank you for your help.

P.S. I have also seen problem with some subdirectories not being visible to users even though I use export all NFS option and everything is on the same data set. Have you guys seen that?

 

[cyberjock]

Can you post your hardware for the smaller system? You say sufficient RAM and CPU power but there's more to the hardware than just having enough to get things done.

Is this performance problem existent even with only 1 user's data and only that user is using the server?

How much data are we talking inside the home directories?

Also have you verified you don't have a LAN problem and have a bad cable that is forcing you to 100Mb(or even 10Mb)?

 

[Oko]

Thanks for showing interest cyberjock!

The smaller server is OEM Supermicro with 1xXeon E5-2620v2 (6 cores), 32 GB=4x8GB DDR-1600 ECC Registered 2R , 4x1 Gigabit Intel LSI 9207-8i 6Gb/s SAS HBA 8port and 6x2TB HDD industrial grade configured as RAIDZ2

I have this problem without any active users. I have yet to move the data to the file server so those home directories are empty. The lab is wired for 1Gb traffic but some machines like bigger file server have 10Gb network cards. Now checking LAN cable on the smaller file server is good hunch as I had before cables accidentally loosen up. Now what worries me is that the problem disappears when I create and export data set of size let say 50Gb.

 

[cyberjock]

Have you tweaked any ZFS properties, zfs tunables, nfs settings? Your hardware should be cruising through this without a problem. So my hunch is that something is very unique to your situation... bad hardware, bad setting, etc.

 

[Oko]

No I have not tweaked any ZFS tunables and nfs settings. Not yet at least when it comes to NFS. Hardware is new :) That is way I am so bewildered by the whole situation.

 

[cyberjock]

Yeah, I have no recommendations. I would say that if you want to do some consultation I could probably narrow it down if I start looking at everything about your box. But in terms of the cause for your problems I don't see any smoking guns. There's so many ways I could reproduce your problem I don't even know where to begin to point you.

 

[Oko]

I think I am hitting some nasty NFS bug https://bugs.freenas.org/issues/4290 which was suppose to be fixed. I configured NFS to use ports 111, 2049, 4000, 4001, 4002 just like on the bigger file server however this is what I find in the log.

Code:

May 16 22:16:12 neill-zfs mountd[51253]: mount request succeeded from 192.168.6.
61 for /mnt/zfsneill/home
May 16 22:17:07 neill-zfs kernel: NLM: failed to contact remote rpcbind, stat =
5, addr = 192.168.6.61, port = 28416
May 16 22:17:32 neill-zfs kernel: NLM: failed to contact remote rpcbind, stat =
5, addr = 192.168.6.61, port = 28416
May 16 22:19:43 neill-zfs last message repeated 3 times
May 16 22:27:57 neill-zfs last message repeated 9 times

How does that look for /syslog/log/messages
Disabling PF on FreeNAS box and IPTables on Red Hat clients seems to fix the problem. I am just too tired to think about it right now. Luckily whole this is happening behind OpenBSD firewall so I am temporary OK with this "solution".