ZFS memory tuning

[uutzinger]

While the FAQ lists options on memory tuning for ZFS and there is also a tuning guide for FreeBSD (http://wiki.freebsd.org/ZFSTuningGuide), those guides are primarily geared to deal with low memory scenarios.

I recently added 6 more drives to my storage unit and I started getting kernel panics. I have quite a bit of memory (16 G byte) but its less than the recommended 1Gbyte per Tbyte of storage.

I used /usr/local/www/freenasUI/tools/arcstat.pl to view the ZFS memory settings and I found that the settings exceed the memory I have on my server. The vm.kmem_size was set to approximately the amount of total memory I have and vm.kmem_size_max was set to 300Gbytes (!). The vfs.zfs.arc_max was 7% lower than the vm.kmem_size. It appears to me that those values are too high.

Reading through RAM requirements one can assume that FreeNAS requires about 2 G bytes to run plus additional RAM for more involved services.
It would be useful if there would be a guidance article to set appropriate /boot/loader.config variables for ZFS. For that purpose I like to make a suggestion below and hope to get feedback for alternative settings.

# ZFS tuning
####################################################
vm.kmem_size_max="aaaM" # where aaa = total system RAM - 4 G byte
vm.kmem_size="aaaM"
vfs.zfs.arc_max="bbbM" # where bbb = 3/4 -5/6 of vm.kmem_size_max
vfs.zfs.vdev.cache.size="cccM" # where ccc = 5..10 Mbyte

An alternative is to set aaa = 1/2 of total system RAM and vm.kmem_size to 1/4 of total system RAM. However I am not sure about arc_max under these conditions.

Urs

 

[uutzinger]

It looks like this problem is going to be resolved in FreeNAS 8.2 because of Garrett Cooper's autotune.py script https://support.freenas.org/browser/freenas/trunk/src/autotune/files/autotune.py

where for a 64 bit FreeNAS

vm.kmem_size_max = max(Hardware Mem - User Mem (e.g. 2.25G), 128M+1024M)
vm.kmem_size=0.8*vm.kmem_size_max
vfs.zfs.arc_max=max(Hardware Mem - (User Mem + 768M), 1024M)

Hardware Mem is your physical memory
User Mem is 2.25G plus what your other services might need.

 

[uutzinger]

I am still getting panic: kmem alloc too small.
There might be an error in autotune.py and vfs.zfs.arc_max needs to be smaller than vm.kmem_size.
I have 16Gbyte RAM
vm.kmem_size="9830M"
vm.kmem_size_max="12299M"
vfs.zfs.arc_max="11500M"
vfs.zfs.vdev.cache.size="10M"
and it does not work.

 

[peterh]

are you running 32-bits by any chance ?

 

[uutzinger]

No its 64 bit. Its not possible to address more than 4Gbyte with 32bit OS and the OS reports 16Gbyte.

I noticed that when setting the values above in loader.conf, vfs.zfs.arc_min is becoming bigger than vfs.zfs.arc_max which is probably not a good thing. I use /usr/local/www/freenas../tools/arc_summary.py to observe the values. This might be because I have 18 drives.

So I used the autotune.py which is part of FreeNAS 8.2 and ran through its calculations. I modified vfs.zfs.arc_max calculation to make sure its smaller than vm.kmem.size and I made vfs.zfs.arc_min 4Gbyte. I hope that will work.
Otherwise I will need to get more RAM.