Noobie kernel panics and tuning

[404wanderer]

Hello all,
noobie here, so please be gentle.

I am running a 32bit freenas system (constrained by motherboard) running ZFS. I have 2GB of RAM.
The freenas box keeps crashing because of running out of kernel memory.

My current tuning is -
vfs.zfs.arc_max 100M
vm.kmem_size_max 1.6G
vm.kmem_size 1.4G

My questions are -

Are these reasonable values?
The ZFS tuning guide mentions that the kernel needs rebuilding to enable kernel sizes greater than 512M. Do I need to do this or is the freenas kernel already build with this option? If I do need to, can somebody point me how I do it?

Thanks in advance

 

[cyberjock]

Please read the manual. http://www.freenas.org/images/resou...s8.3_guide.html#__RefHeading__7608_1957652121

You are told you need 6GB of RAM to use ZFS in the manual. Not sure how you think you are going to get by with 1/3 of the recommended.

This has been discussed over and over.

Thread closed.

 

[James]

Thread re-opened as only SPAM threads should be closed.

 

[404wanderer]

Thanks. I'm currently waiting on RAM to upgrade the syetem to the maximum it can take, bearing in mind this is a 32bit system. I have therefore disabled prefetching as advised. I couldn't find a reference to 6GB being the minimum system requirements.

The part of the manual I was looking at was -

1.4.2 RAM

"The best way to get the most out of your FreeNAS® system
is to install as much RAM as possible. If your RAM is
limited, consider using UFS until you can afford better
hardware. ZFS typically requires a minimum of 8 GB of RAM in
order to provide good performance. The more RAM, the better
the performance, and the FreeNAS® Forums provide anecdotal
evidence from users on how much performance is gained by
adding more RAM. For systems with large disk capacity
(greater than 6 TB), a general rule of thumb is 1 GB of RAM
for every 1TB of storage. This post describes how RAM is
used by ZFS.

NOTE: by default, ZFS disables pre-fetching (caching) for
systems containing less than 4 GB of usable RAM. Not using
pre-fetching can really slow down performance. 4 GB of
usable RAM is not the same thing as 4 GB of installed RAM as
the operating system resides in RAM. This means that the
practical pre-fetching threshold is 6 GB, or 8 GB of
installed RAM. You can still use ZFS with less RAM, but
performance will be effected.

If you plan to use ZFS deduplication, a general rule of
thumb is 5 GB RAM per TB of storage to be deduplicated. "


The manual references the ZFS tuning guide, which says -
"i386

Typically you need to increase vm.kmem_size_max and
vm.kmem_size (with vm.kmem_size_max >= vm.kmem_size) to
not get kernel panics (kmem too small). The value depends
upon the workload. If you need to extend them beyond 512M,
you need to recompile your kernel with increased KVA_PAGES
option, e.g. add the following line to your kernel
configuration file to increase available space for
vm.kmem_size beyond 1 GB:

options KVA_PAGES=512
"


So, can anybody confirm what KVA_PAGES option freenas is compiled with, or how I find out?

 

[praecorloth]

Excellent, I'm glad this was reopened. I don't know if this has any more information than what you already know, however,

http://forums.freenas.org/showthrea...rough-FreeNAS-on-a-32bit-proc-with-low-memory

There's the link anyways. Regarding KVA_PAGES, I believe FreeNAS is compiled with a lower KVA_PAGES set. If I understand correctly, according to the last paragraph you posted, if KVA_PAGES=512 was set in the kernel during compile time, I would have been able to use vm.kmem_size(_max) to something larger than 512MB. However, when I was playing with this on a 32bit system, any entry larger than 512MB resulted in a system that wouldn't boot.

I am curious to know more as you find out more.

 

[paleoN]

Regarding KVA_PAGES, I believe FreeNAS is compiled with a lower KVA_PAGES set

[r13282] FREENAS.i386, as the OP couldn't be bothered to mention what version he's using.

 

[404wanderer]

[r13282] FREENAS.i386, as the OP couldn't be bothered to mention what version he's using.

It wasn't an intentional oversight.

Build FreeNAS-8.3.0-RELEASE-p1-x86 (r12825)
Platform Intel(R) Pentium(R) 4 CPU 3.00GHz
Memory 2030MB

 

[404wanderer]

Just looking here and found this mentioned -
http://forums.freenas.org/faq.php?faq=general_freenas

I have a 32-bit processor with more than 2GB of RAM, how do I recompile the kernel so I can adjust settings in loader.conf?

BEFORE 8.0.3, the highest you could set your vm.kmem_size in /boot/loader.conf was ~768M. This has been changed in 8.0.3 and the instructions below are only needed for versions before 8.0.3 (KVA_PAGES=512 is the default now)

How To Compile The Kernel
Cross-Compiling
Using-ZFS-on-FreeNAS-with-a-32bit-proc-and-low-memory.
Spontaneous-reboots

After downloading the sources explained in the link above, you'll need to edit the file FreeNAS.i386 and add the line options KVA_PAGES=512. You'll also need to change a couple of settings in your /boot/loader.conf, after you confirm that your new kernel boots. Those two variable in loader.conf are called vm.kmem_size & vm.kmem_size_max.

To begin, try setting vm.kmem_size_max equal to about 1/2 of your RAM, and vm.kmem_size about 1/2 of the max setting. So if you have 4G of RAM, vm.kmem_size_max=2G, and vm.kmem_size=1024M for example. These settings are only an example of where to begin.

So, I believe that means that the KVA pages setting means I can use more than 1GB memory.

I'm still finding it strange that the system is reporting 1.2GB memory free but still crashes with an out of kernel memory issue. I need to monitor kernel memory usage more closely, but with 1.4GB of 2GB allocated to the kernel, showing 1.2GB of total memory free, this sort of indicates that it isn't using the kernel memory that has been allocated to it.

Can anybody explain this?

 

[404wanderer]

Ok, system just crashed again with kmem_map too small.

I was monitoring the kernel memory at the time -

Code:

Sat Feb 16 15:11:41 CST 2013
TEXT=14145304, 13.49 MB
DATA=125017088, 119.226 MB
TOTAL=139162392, 132.716 MB

Sat Feb 16 15:11:46 CST 2013
TEXT=14145304, 13.49 MB
DATA=127869952, 121.946 MB
TOTAL=142015256, 135.436 MB

Sat Feb 16 15:11:51 CST 2013
TEXT=14145304, 13.49 MB
DATA=127925248, 121.999 MB
TOTAL=142070552, 135.489 MB

Sat Feb 16 15:11:56 CST 2013
TEXT=14145304, 13.49 MB
DATA=127841280, 121.919 MB
TOTAL=141986584, 135.409 MB

Sat Feb 16 15:12:01 CST 2013
TEXT=14145304, 13.49 MB
DATA=127945728, 122.019 MB
TOTAL=142091032, 135.509 MB

Sat Feb 16 15:12:06 CST 2013
TEXT=14145304, 13.49 MB
DATA=130203648, 124.172 MB
TOTAL=144348952, 137.662 MB

Given my tunables say -
vm.kmem_size_max 1.6G
vm.kmem_size 1.4G

Why is it crashing?