FreeNas 8.3-p4 crashing every friday night

Status
Not open for further replies.

mnt_schred

Dabbler
Joined
Aug 6, 2012
Messages
29
I have a freenas machine which crashes every friday. Either de webGui is unreachable or this panic is shown:

Photo 31-08-12 12 42 21.jpg

I don't know where to start looking. The syslog is empty:

Code:
# tail -n 1000 -f /var/log/messages
Sep  3 04:00:00 fs1 newsyslog[16471]: logfile turned over due to size>100K
Sep  3 12:05:54 fs1 su: thies to root on /dev/pts/0
 

mnt_schred

Dabbler
Joined
Aug 6, 2012
Messages
29
I've OCR'd the error on the screen:
Code:
swp_pager_meta_build() at swp_pager_meta_build+0x1b1
swap_pager_copy() at swap_pager_copy+0x160
vm_coject_collapse() at vm_coject_collapse+0x436
vm_object_deallocate() at vm_object_deallocate+0x6ea
vm_map_entry_deallocate() at vm_map_entry_deallocate+0x4c
... 
amd64_syscall() at amd64_syscall+0x1f4
Xfast_syscall() at Xfast_syscall+0xfc
--- syscall (59, FreeBSD ELF64, execve), rip = 0x800b0398c, rsp = 0x7fffffffe568, rbp = 0x7fffffffe670 ---
panic: sleeping thread
cpuid = 1
Code:

maybe the amount of RAM is insufficiënt?

	

		
			
		

		
	

screenshot.28.jpg

	

		
			
		

		
	
 

Attachments

  • screenshot.26.jpg
    screenshot.26.jpg
    28.1 KB · Views: 257

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
That seems like an entirely reasonable guess, if it's freaking out trying to do a syscall and it's losing it in swap_pager stuff.
 

mnt_schred

Dabbler
Joined
Aug 6, 2012
Messages
29
Maybe it's related to the RSYNC backup jobs running to the host. At 0:00 the memory utilisation jumps to 3 GB:
screenshot.29.jpg
and won't return to the old utilisation. Is there a known memory leak in freenas due to rsync?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
That's the answer right there. No, it's probably not a memory leak, but ZFS is very memory-piggy, as frequently described in these forums. This is partly due to default choices made by the FreeNAS developers, and you may very well be able to tune your system so as to not use as much memory for ZFS.

What's happening is that you suddenly have a bunch of demands for memory due to activity - a lot of what ZFS does will show up under "Wired", and look at how that explodes upwards. That means that if there's something in userland that also wants memory, the system may begin swapping - which it appeared to be doing, given the error you provided earlier.

There are some tuning guides for smaller memory systems floating around. Suggest you try some of them, my guess is you can make it work.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
FreeNAS is largely an administrative UI and featureset built on top of FreeBSD. Part of the goal is to build an appliance-style system that doesn't need to be endlessly twiddled, and there's a definite benefit to specifying a hardware baseline. However, fundamentally, FreeNAS can likely be made to work on anything that FreeBSD could also work on, and people have gotten ZFS working on systems with much less than 8GB of RAM using FreeBSD.

You should be able to get away with tuning; the question is mostly how to identify when your tuning has been successful in resolving crashes. This is problematic: is it "fixed" if you try something and it doesn't crash next Friday? Maybe it gets lucky and only close to crashing. That's the frustrating bit: you don't know if you've really solved the problem until you have several dozen Fridays under your belt. However - you can read multiple success stories on these forums of people tuning for 4GB or less, especially as some of the platforms that are popular for FreeNAS only accept up to 4GB (Intel Atom, etc). Pay attention to those lessons and you're likely to have success.

On the other hand, RAM is cheap, and if that's cheaper than your time and frustration, RAM appears as though it would solve your problem. Shooting for 8GB is a great idea. The 1GB/TB rule is going to be more meaningful if you have a really busy system, and you can probably safely downplay that a bit, especially if you don't mind the performance not being peak. Presumably an rsync backup system doesn't need to be super-snappy, it just needs not to crash.
 

mnt_schred

Dabbler
Joined
Aug 6, 2012
Messages
29
Okay, thanks! I made my decision. I'll purchase 1 4GB dimm, effectively adding 2 GB. That's relatively inexpensive and less costly then a few fridays..
 
Status
Not open for further replies.
Top