kmem_map too small

[UndyingShadow]

My FreeNAS server is getting kernal panics every time I do large file transfers. It stays up for about an hour, and then crashes with an error in the console. It's reproducible every time.

Error message:
Panic: kmem_malloc(32768): kmem_map too small: 1607741440 total allocated

I burned FreeNAS-8.2.0-BETA2-x64.iso to a disc and then used another PC to install to a 8gb flash drive. AFAIK, this is the correct 64-bit version.

My hardware is as follows:
HP ProLiant N40L Microserver
2X Kingston 4GB 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10600)
5X SAMSUNG EcoGreen F4 HD204UI 2TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive
1X WD 2TB Green Drive
Intel EXPI9301CTBLK Network Adapter


I read the FAQ, and the manual, and it seems to indicate I shouldn't need to turn vm.kmem_size since I'm running the x64 version. Is this incorrect?

 

[louisk]

Does this happen with 8.0.4?

 

[ilium007]

Yep - sure does. I posted my issues here in the forums:

http://forums.freenas.org/showthread.php?6541-kernel-panic-error&p=25660&viewfull=1#post25660

 

[louisk]

Hmm. Does FreeNAS see all the memory? FreeBSD 8.x and newer is supposed to do a reasonable job auto-tuning ZFS.

Have you tried swapping RAM sticks to see if you have some faulty memory?

 

[voodoo]

I have the same issue. When I import many files from my 2T disks to my FreeNAS server, I'm getting kernel panics. It's reproducible every time.

version: FreeNAS-8.2.0-BETA2-x64
4gb flash drive for freenas.

Error message:
Panic: kmem_malloc(32768): kmem_map too small: 1607741440 total allocated

My hardware:
16G (4X4GB) DDR3 1333
8X 3TB 3.5" Internal Hard Drive
amd cpu + mainboard.

 

[darkseed86]

I am also having the same issue. It seems to only be happening to me when the system is under load (i.e. streaming media). I have left it for a day, not doing anything at all, and it stayed up fine, then when I streamed a video from the system to my Desktop, the error occurred and I had to manually turn it off, there seems to be no other way of rebooting the system when this error occurs, and it is totally unresponsive to any sort of input at that stage.

My hardware:

Dual Core Intel 2.8Ghz CPU
3 x 3TB Seagate Barracuda Drives
5 x 2TB WD Green WDEARS Drives
4GB DDR3 1333 RAM

EDIT: I've seen the question about Freenas seeing all of the RAM, and mine does:

Build FreeNAS-8.2.0-RELEASE-p1-x86 (r11950)
Platform Pentium(R) Dual-Core CPU E5500 @ 2.80GHz
Memory 3536MB
System Time Sat Aug 11 09:52:06 BST 2012
Uptime 9:52AM up 11 mins, 0 users
Load Average 0.00, 0.09, 0.11

 

[voodoo]

I'm #5. After I deleted all the items in 'autotune', this issue has gone.

 

[darkseed86]

Thanks for replying, but can you tell me exactly how you did that?

 

[darkseed86]

Don't worry about this now, it has been resolved. Just in case anyone else has the problem, all I did was enable autotune (which is disabled by default) in System>Settings>Advanced and tick autotune. I've been running now for a few hours without the problem (and transferring data for the entire time, still without any incident). This seems to have resolved the issue.

Thanks for the help and pointing me in the right direction.

 

[voodoo]

Hi Dardseed, glad you made it. I'm using v8 beta version still. In my case, the autotune is enable by default, so I disable it, everything became fine. Thank you for updating it, now we know autotune trick. ;)

 

[jaguiar]

Thanks!!

Don't worry about this now, it has been resolved. Just in case anyone else has the problem, all I did was enable autotune (which is disabled by default) in System>Settings>Advanced and tick autotune. I've been running now for a few hours without the problem (and transferring data for the entire time, still without any incident). This seems to have resolved the issue.

Thanks for the help and pointing me in the right direction.

Appreciate the tip...looking forward to fixing this annoying problem.

 

[Dan Walton]

I also have the same isusse with FreeNAS-8.2.0-RELEASE-p1-x64 (r11950), HP ProLiant N40L Microserver, Large transfers bomb out with 'kmem_map too small' message on the command line. Gonna try this now will post if it also works for me. I just upgraded from 8.0.2 Beta and from UFS to ZFS raid5z so thought it was that! fingers crossed!

 

[Dan Walton]

No still get 'kmem_map too small' even with 'Enable autotune' turned either on/off?? any suggestions? Got about an hour and forty five minutes into the copy this time anyone got any Ideas

 

[William Grzybowski]

How much RAM? If you probably running too short on RAM.

 

[Dan Walton]

The machine has got 8Gb of Ram, 8Tb of Disk space, Raidz5, ZFS.

UPDATE, I noticed a few hours ago that even after turning off the 'Enable autotune' that in tunables It still had the following settings:
vfs.zfs.arc_max 1024M,
vm.kmem_size 1536M,
vm.kmem_size_max 6022951040
I manually deleted them and left 'Enable autotune' off then rebooted, It seems ok, transferred about 750Gb without it crashing. i'll post in a few days if it works out

 

[jaguiar]

Tunables?

The machine has got 8Gb of Ram, 8Tb of Disk space, Raidz5, ZFS.

UPDATE, I noticed a few hours ago that even after turning off the 'Enable autotune' that in tunables It still had the following settings:
vfs.zfs.arc_max 1024M,
vm.kmem_size 1536M,
vm.kmem_size_max 6022951040
I manually deleted them and left 'Enable autotune' off then rebooted, It seems ok, transferred about 750Gb without it crashing. i'll post in a few days if it works out

Same problem here - going to try disable AutoTune again (for some reason, turned itself back on) but do not know how to get to "tunables" to delete as you explained. Would love to get this problem resolved (as I'm sure all of us would).
Thanks!!

 

[bazzw6]

i had loads of probs shuffling lage files around i the nas kept crashing i had to manually turn off ( no reboot possible) with vm.kmem_size
then i got probs with kmem_malloc

i enabled auto-tune and it worked well then more crashes i randomly got 'ada0 not found' 'rescan started' and had to manually turn off upon reboot all was fine
i used a new usb drive same happend again after 7-8 days
i took my mobo out today and 2 of the capacitors are rounded ou top so i removed them an replaced them with 2 from an old m/b ...... hey presto no vm.kmem or malloc and
looks like the usb's were intermittent so the dongle was'nt being picked up (hence the ada0 not found ect) all the fault of 2 capacitors good job i had all my stuff backed up :)
i've just transfered 5.6gb onto a drive in 12 minutes.. so all is looking good.
now its taking 6.82gb 19mins 5.4m/bs a second not bad for and old pc
specs
gigabite k8n-f9
athon xp 64 3.2
4 gb ram
2X750 gb
2X320 gb
_

i am a complete pleb noob with this coding stuff but correct me if i'm wrong

you need to mount the iso you intend to use on your system (power iso or similar)
browse to boot/defaults/loader config open this with note pad and change the values of the afore mentioned vw.kmem
save changes....burn disc...load into cd drive of freenas unit, reboot from cd, choose upgrade,....reboot job done ????

as i said correct me if i'm wrong cos i would like to learn this stuff

 

[Dan Walton]

Hi Jaguiar, To manually delete the tunables, in the GUI you go to System/Tunables (on the left hand side). If there are any your'll see them listed.

 

[Dan Walton]

Update to my problem! Success!!
"Large transfers bomb out with 'kmem_map too small' message on the command line." I tried turning on and off the 'Enable autotune' option in Settings/Advanced/ with no luck then I saw that under tunables menu (in the GUI you go to System/Tunables (on the left hand side)) some setting for:
vfs.zfs.arc_max 1024M,
vm.kmem_size 1536M,
vm.kmem_size_max 6022951040

I deleted then manually and so far I have transferred several terabytes of data over the last two days with no problems.

My machine previously had no problems with 8.2.0 Beta1, but when upgrading to FreeNAS-8.2.0-RELEASE-p1-x64 (r11950) this is when the problems started! I upgraded rather than fresh instail, so maybe there lies the problem?? in any case it seems to all be about the tunables!

good luck fixing your machines

 

[jaguiar]

Hi Jaguiar, To manually delete the tunables, in the GUI you go to System/Tunables (on the left hand side). If there are any your'll see them listed.

DOH! I can't believe I didn't see that...I was clicking on System at the top, not on the side. Oh well, learned something new today! Thanks for the reply...had the items you listed and have since deleted them from there. We'll see in the morning. Appreciate it!

UPDATE: Unfortunately, the problem came back after the second day. Very frustrating. Any other suggestions would be greatly appreciated!!