The SO_SNDBUF and SO_RCVBUF settings are overkill. You need to tweak these values (ex. from /etc/sysctl.conf):
Code:
net.inet.tcp.recvspace=262144
net.inet.tcp.sendspace=131072
net.inet.udp.recvspace=131072
Some other settings that would make sense with your massively multicore setup would be:
Code:
# Not needed for 9.x+ with the eventtimer framework.
kern.hz=100
Some other values I've tweaked:
Code:
# For my RAID controller.
hw.mfi.max_cmds
kern.ipc.nmbclusters
# Not really necessary in home scenarios, but I did it for my own testing.
kern.maxfilesperproc
vm.kmem_size
vm.kmem_size_max
vfs.zfs.arc_max
Another couple tidbits to consider...
There are other knobs that can be tweaked in the base system to improve AIO performance if you have the CPU/cores to back it up. Here are the sysctls:
Code:
$ sysctl vfs.aio
vfs.aio.max_buf_aio: 16
vfs.aio.max_aio_queue_per_proc: 256
vfs.aio.max_aio_per_proc: 32
vfs.aio.unloadable: 0
vfs.aio.aiod_lifetime: 3000
vfs.aio.aiod_timeout: 1000
vfs.aio.num_buf_aio: 0
vfs.aio.num_queue_count: 0
vfs.aio.max_aio_queue: 1024
vfs.aio.target_aio_procs: 4
vfs.aio.num_aio_procs: 4
vfs.aio.max_aio_procs: 32
In particular:
- vfs.aio.num_buf_aio and vfs.aio.num_queue_count are the numbers you should watch so you can tweak vfs.aio.max_buf_aio and vfs.aio.max_aio_queue .
- You should look at vmstat -z and watch the stats in the following rows:
Code:
AIO: 208, 0, 1, 107, 5, 0, 0
AIOP: 32, 0, 4, 905, 131, 0, 0
AIOCB: 480, 0, 0, 584, 7626491, 0, 0
AIOL: 128, 0, 0, 0, 0, 0, 0
AIOLIO: 272, 0, 0, 0, 0, 0, 0
If particular, if USED is non-zero and FREE is either zero or approaching zero and things have peaked and/or are slowing down, you should try increasing the maximum sysctls (
NOTE: some of the reported values are redundant with the sysctls that I showed above).
Warning: tweaking some values can dramatically boost/reduce performance in a variety of ways. For example, increasing the jumbo9k frames ipc mbuf cluster count torqued another component in the system and made things slower. So empirical testing of all components -- if possible -- is a must, and just because a certain set of values works for someone on the list with a certain set of hardware.
