Optimizing NFS Number of servers

[Will Dormann]

Hi folks,

I've done an experiment where I had 100 VMs all concurrently using an NFS share exported by FreeNAS. It resulted in timeouts galore. I then realized that there's a "Number of servers" configurable setting in the NFS share options. This defaults to 4. The FreeNAS documentation indicates that this setting should not exceed the number of CPU cores on a system.

I'm having a hard time finding out what an optimal value for this setting is, though. A random Cisco guide says: "If you plan to have more than four people connecting to shares on FreeNAS using this protocol, you can increase the number of servers. However, it's recommended that you only have up to six users/servers at one time." This doesn't make too much sense to me, and the use of "people" as the metric makes me trust the guidance even less.

Another guide says: "A rule of thumb is to have at least 4 servers for each client plus a few extras. You set this value as a runtime flag to nfsd (you use the -n flag), use this formula:
(Number-of-clients + 1) * 4.". Which would give me 404 NFS servers in my case. And the FreeNAS web gui won't allow a setting of larger than 256.

One more guide indicates that on Linux the default is 8, but you could try something like 20 if you have a lot of clients. It also says that you can confirm the number of threads you currently have by using the ps command and counting the nfsd instances. In my testing, this doesn't seem to apply to FreeNAS/FreeBSD, though. Regardless of the number of servers setting, ps shows something like:

# ps aux | grep nfs
root 57807 0.0 0.0 14064 1788 ?? Is 10:10AM 0:00.00 nfsuserd: master (nfsuserd)
root 57808 0.0 0.0 14064 1792 ?? S 10:10AM 0:00.00 nfsuserd: slave (nfsuserd)
root 57809 0.0 0.0 14064 1792 ?? S 10:10AM 0:00.00 nfsuserd: slave (nfsuserd)
root 57810 0.0 0.0 14064 1792 ?? S 10:10AM 0:00.00 nfsuserd: slave (nfsuserd)
root 57811 0.0 0.0 14064 1792 ?? S 10:10AM 0:00.00 nfsuserd: slave (nfsuserd)
root 57880 0.0 0.0 108268 31244 ?? Is 10:10AM 0:00.10 nfsd: master (nfsd)
root 57881 0.0 0.0 9916 5616 ?? I 10:10AM 0:00.10 nfsd: server (nfsd)
root 16470 0.0 0.0 16284 1884 12 S+ 11:29AM 0:00.00 grep nfs

At first, I thought that perhaps the FreeNAS web gui wasn't actually setting the value properly. However, running sysctl vfs.nfsd.threads reflects the value specified in the "Number of servers" setting in the web gui. fs.nfsd.minthreads and vfs.nfsd.maxthreads are also set to the same value.

The questions I have at this point are:

1. What is the optimal value for "Number of servers"?
2. If it's something like == # of CPU cores, then perhaps could FreeNAS be smarter about its defaults?
3. What is the harm in having a too-high value for "Number of servers"? The FreeNAS documentation states to not exceed the number of CPU cores, however it's not clear what the consequences are. Lower performance? Wasted RAM? Something more catastrophic?
   
4. Is having vfs.nfsd.minthreads == vfs.nfsd.maxthreads == vfs.nfsd.threads perhaps non-optimal in some way?

For the benefit of others, in the process of typing this message, I realized that ps auxH will show threads as if they were individual processes, and this shows count of nfsd: server (nfsd), which reflects the "Number of servers" setting in the FreeNAS web gui.

 

[cyberjock]

1. The optimal value is totally dependent on your workloads. For some I've seen 3 being optimal, for other larger systems I've seen 256.
2. The number of CPUs is a guide, but is not a solid "do not exceed this value".
3. If your number is too high or too low, you just won't get the kind of performance you'll get with the optimal value set.
4. Yes, you don't want to force a specific number of threads. You want nfsd to figure out what it thinks is best, within reason.

 

[Will Dormann]

4. Yes, you don't want to force a specific number of threads. You want nfsd to figure out what it thinks is best, within reason.

Thanks for the info! My concern with item 4. is that if you set the number of NFS servers (threads) via the FreeNAS GUI, you get a fixed setting of threads, min, and max. (all three are the same single number that you provide).

For optimal NFS tuning (letting NFSD figure out what it needs), you need to set these values manually somewhere? And if so, what's the recommended mechanism for setting the values? I'm sure sysctl in a terminal works, however I'd like to see a supported technique, in that it survives reboots, OS upgrades, etc.

 

[toadman]

If you can use sysctl can't you just set a tunable? i.e. System > Tunables > Add Tunable?

 

[Will Dormann]

Ok, I'll give that a shot. Thanks.

 

[cyberjock]

Thanks for the info! My concern with item 4. is that if you set the number of NFS servers (threads) via the FreeNAS GUI, you get a fixed setting of threads, min, and max. (all three are the same single number that you provide).

Umm.. let me ask someone more knowledgeable than I and see if this has changed or what since last time I talked about it.

 

[cyberjock]

Alrighty, got an answer from a developer.

Normally, if you were setting up nfsd on FreeBSD, you'd specify the minimum and maximum number of servers (threads). nfsd does an 'okay' job of trying to determine the proper number to maximize performance. It's not perfect though and there's situations where you should force it. Especially if workload is fluctuating frequently. This dynamic nature is supposed to minimize having too many threads specified, which can hurt performance. Likewise if you don't have enough threads, that can hurt performance too.

FreeNAS, by is design, simply sets a static number (which disables the dynamic feature that nfsd has). The basic reasoning from the devs is that unlike a FreeBSD install, FreeNAS is designed to be a NAS and for the most part nothing else. So you aren't in a situation where you are competing with lots of other applications like X Windows to compete for limited resources. So there's little to gain from the dynamic nature of nfsd and lots to gain from simply forcing a given number of threads.

Hope this explains things. ;)

 

[Will Dormann]

Makes sense. Thanks!