FreeNas 8.3-p4 crashing every friday night

[mnt_schred]

I have a freenas machine which crashes every friday. Either de webGui is unreachable or this panic is shown:



I don't know where to start looking. The syslog is empty:

Code:

# tail -n 1000 -f /var/log/messages
Sep  3 04:00:00 fs1 newsyslog[16471]: logfile turned over due to size>100K
Sep  3 12:05:54 fs1 su: thies to root on /dev/pts/0

 

[William Grzybowski]

How much RAM? x86 or x64

 

[mnt_schred]

4 GB Ram
x64

 

[mnt_schred]

I've OCR'd the error on the screen:

Code:

swp_pager_meta_build() at swp_pager_meta_build+0x1b1
swap_pager_copy() at swap_pager_copy+0x160
vm_coject_collapse() at vm_coject_collapse+0x436
vm_object_deallocate() at vm_object_deallocate+0x6ea
vm_map_entry_deallocate() at vm_map_entry_deallocate+0x4c
... 
amd64_syscall() at amd64_syscall+0x1f4
Xfast_syscall() at Xfast_syscall+0xfc
--- syscall (59, FreeBSD ELF64, execve), rip = 0x800b0398c, rsp = 0x7fffffffe568, rbp = 0x7fffffffe670 ---
panic: sleeping thread
cpuid = 1
Code:

maybe the amount of RAM is insufficiënt?

	

		
			
		

		
	



	

		
			
		

		
	

 

[jgreco]

That seems like an entirely reasonable guess, if it's freaking out trying to do a syscall and it's losing it in swap_pager stuff.

 

[mnt_schred]

Maybe it's related to the RSYNC backup jobs running to the host. At 0:00 the memory utilisation jumps to 3 GB:

and won't return to the old utilisation. Is there a known memory leak in freenas due to rsync?

 

[jgreco]

That's the answer right there. No, it's probably not a memory leak, but ZFS is very memory-piggy, as frequently described in these forums. This is partly due to default choices made by the FreeNAS developers, and you may very well be able to tune your system so as to not use as much memory for ZFS.

What's happening is that you suddenly have a bunch of demands for memory due to activity - a lot of what ZFS does will show up under "Wired", and look at how that explodes upwards. That means that if there's something in userland that also wants memory, the system may begin swapping - which it appeared to be doing, given the error you provided earlier.

There are some tuning guides for smaller memory systems floating around. Suggest you try some of them, my guess is you can make it work.

 

[mnt_schred]

but I've also read that it's good practice to have 1 GB ram/TB storage in Freenas.. http://forums.freenas.org/showthread.php?7459-ZFS-in-a-low-memory-setup
maybe I won't get away with tuning? What would you guys reccomend? spent de € 123,- at memory? Or try to tune the system?

 

[jgreco]

FreeNAS is largely an administrative UI and featureset built on top of FreeBSD. Part of the goal is to build an appliance-style system that doesn't need to be endlessly twiddled, and there's a definite benefit to specifying a hardware baseline. However, fundamentally, FreeNAS can likely be made to work on anything that FreeBSD could also work on, and people have gotten ZFS working on systems with much less than 8GB of RAM using FreeBSD.

You should be able to get away with tuning; the question is mostly how to identify when your tuning has been successful in resolving crashes. This is problematic: is it "fixed" if you try something and it doesn't crash next Friday? Maybe it gets lucky and only close to crashing. That's the frustrating bit: you don't know if you've really solved the problem until you have several dozen Fridays under your belt. However - you can read multiple success stories on these forums of people tuning for 4GB or less, especially as some of the platforms that are popular for FreeNAS only accept up to 4GB (Intel Atom, etc). Pay attention to those lessons and you're likely to have success.

On the other hand, RAM is cheap, and if that's cheaper than your time and frustration, RAM appears as though it would solve your problem. Shooting for 8GB is a great idea. The 1GB/TB rule is going to be more meaningful if you have a really busy system, and you can probably safely downplay that a bit, especially if you don't mind the performance not being peak. Presumably an rsync backup system doesn't need to be super-snappy, it just needs not to crash.

 

[mnt_schred]

Okay, thanks! I made my decision. I'll purchase 1 4GB dimm, effectively adding 2 GB. That's relatively inexpensive and less costly then a few fridays..