System freezes when writing data

[jsmith8954]

I don't know which category to post this, cause i'm unsure of the exact problem.

Built my freeNAS box today. Got Freenas 8.0 installed (i366) and configured. I set up my volume (ZFS - raidz1). I set up my share.

I successfully mounted my share on my Windows 2008 server box and started transferring my data. I was transferring big chunks, (400+ gb's and 200+ gbs) and on 4 separate times I loose connection to freeNAS and I can't ping it or pull it up over the web GUI either. It would start the transfer and then after a few minutes drop. Of 2 transfers i got 14 of 400+gb's and 2 of 200+ gb's.

When not transferring data, and just letting the box run, it'll stay up no problem, only seems to be when transferring files.

Also when transferring files it seemed like the speed was low, like 13-20mb/s. both machines have 1gb NICs with a gig router.

Hardware of FreeNAS box:
P4 3.4Ghz
4Gb RAM DDR2 533Mhz
4 x 1TB WD Blue drives

When the box freezes, i have to go back there and power cycle it to get it to come back up.
Any clues?

thanks!

 

[matthewowen01]

were you copying lots of small files or lots of large files. samba Hates small files (speed wise that is)

first up, run a memcheck on it. probably not it but it's a good place to start.

next, when you do a copy, log in via the webGui and click on the 'Display System Processes' button down at the bottom of the tree. let us know which processes are taking the most cpu time.

copying data is pretty cpu intensive with zfs, how well is your system cooled? strap a very large fan onto the side just to test. it will be noisy but it's not a fix, it's a test. this may be more stress than your system has ever experience and maybe overheating.


Let us know what you find.

 

[jsmith8954]

Hey, thanks for the reply,

were you copying lots of small files or lots of large files. samba Hates small files (speed wise that is)

The first chunk was video files. Each video is broken down into smaller sub-files. I checked some of the sizes out. They range from kb's up to 500mb. The second chunk of data was normal documents/pictures/music etc.

first up, run a memcheck on it. probably not it but it's a good place to start.

What's the best way to run a memcheck on freenas? I'm ssh'd into at the moment.

next, when you do a copy, log in via the webGui and click on the 'Display System Processes' button down at the bottom of the tree. let us know which processes are taking the most cpu time.

There is only 1 running process, the samba process (smbd). I also took a screen shot here as well so that you could see it during a file transfer. (www.aces.edu/~jps0003/freenasprocesstable.jpg)
you can note that transfer speed. That is because i'm testing out a different client which is a laptop on a wireless connection. I'm also testing 1 video folder instead of all of them together. It took about an hour to transfer that 4.35gb folder, but again it is wireless and a test, and it did work and not crash the freenas box.

copying data is pretty cpu intensive with zfs, how well is your system cooled? strap a very large fan onto the side just to test. it will be noisy but it's not a fix, it's a test. this may be more stress than your system has ever experience and maybe overheating.

The system is cooled fairly well. Its a new case and the drives are spaced out for air flow with 2 120mm fans blowing directly over them. There is also 140mm exhaust fan well pulling air out the top of the case. Is there a way to check the temperature?
I can try the big fan trick later on either tonight or in the morning.

Thanks again

 

[jsmith8954]

so i just did another test from the Windows server 2008. I transferred one video folder similar to the above mentioned folder. This folder was 3.19gbs. I transferred it straight to freenas share and it took about 30 seconds at a speed of 45-50mb/s. http://www.aces.edu/~jps0003/test2.jpg

Is this a samba issue? does it not like large chunks of data? i don't normally transfer chunks at 400+ gb.

 

[esamett]

I am having problems writing to zfs. The devs think there may be a problem with z2 volumes

 

[matthewowen01]

Hey, thanks for the reply,



The first chunk was video files. Each video is broken down into smaller sub-files. I checked some of the sizes out. They range from kb's up to 500mb. The second chunk of data was normal documents/pictures/music etc.



What's the best way to run a memcheck on freenas? I'm ssh'd into at the moment.



There is only 1 running process, the samba process (smbd). I also took a screen shot here as well so that you could see it during a file transfer. (www.aces.edu/~jps0003/freenasprocesstable.jpg)
you can note that transfer speed. That is because i'm testing out a different client which is a laptop on a wireless connection. I'm also testing 1 video folder instead of all of them together. It took about an hour to transfer that 4.35gb folder, but again it is wireless and a test, and it did work and not crash the freenas box.



The system is cooled fairly well. Its a new case and the drives are spaced out for air flow with 2 120mm fans blowing directly over them. There is also 140mm exhaust fan well pulling air out the top of the case. Is there a way to check the temperature?
I can try the big fan trick later on either tonight or in the morning.

Thanks again

you can download a cd that will check your memory when you boot from it. http://www.memtest86.com/ let it run for at least one full cycle.

zip together a large collections of files so you have a file at least 10GB. test with that, otherwise it will be impossible to diagnose anything. also, don't use wireless. as for checking the temp, i generally use my IR thermometer, baring that, i use my finger. however it is a bit difficult to send your finger results...

Once you can get back to reliably crashing, test to see if heat is the trouble.

 

[survive]

Hi guys,

Try this for checking your cpu temperature:

sysctl -a | grep temper

What happens when you "dd" out a file?

change to your zfs volume and run something like this:

dd if=/dev/zero of=testfile bs=8192k count=1000 <---make sure you know what this does\you change to your zfs volume

if it completes try upping count=1000 to 2000. This will write out an 8 gig (or 16 gig) file to your volume. This will help isolate the fault to the storage or the network.

-Will

 

[louis-m]

i have this with an intel d510mo mobo. disabled hyperthreading etc but nothing would work. read around and i took 1x stick of ram out so i went from 4gb (2x 2gb) down to a 1x 2gb and freenas is rock solid. the memory isn't faulty and is perfectly matched. the error i was getting is panic: vm_page_insert cpuid=3

 

[jsmith8954]

just download that memtest86, going to try and check that out. I'm also in the process of zipping up a folder of 70ish gb's to try and copy.

I ran that command to try and find the temp, didn't display anything. I left off the grep and tried manually searching through the list for a temperature, but no luck

I ran the "dd" command worked with both 1000 and 2000 count

%dd if=/dev/zero of=testfile bs=8192k count=1000
1000+0 records in
1000+0 records out
8388608000 bytes transferred in 116.468273 secs (72024834 bytes/sec)
%dd if=/dev/zero of=testfile bs=8192k count=2000
2000+0 records in
2000+0 records out
16777216000 bytes transferred in 239.485560 secs (70055230 bytes/sec)


After the zipped folder test i'll try the memtest86, see what it reports and also try troubleshooting with the ram

thanks for the replies everyone!

 

[scorpius]

Sounds the same here

I am getting the same issues here, runs solid for days, but then when I try to run several transfers (even over wireless which is slow) it locks my freenas system. I found if I only do one transfer at a time, it seems to stay steady, but kinda defeats the purpose if only one person can access at a time. I think I read on another forum (didn't have time to read it all though) that it may have to do with some limitation where it doesn't utilize the ram properly and crashes after utilizing 500mb worth, even though I have 4gb installed. Any thoughts?

When it crashes and I hook a monitor to the system the error is as follows:

panic: kmem_malloc(###): hmem_map too small: 331603968 total allocated

the ### part is a little different. The error comes up 3 times, the first time ### is a different number than the other two. But I hope this can help someone to form a solution for us.

 

[William Grzybowski]

This is a problem running the 386 install...
ZFS needs a lot of kernel memory, by default it is limited to 512mb for x86

 

[jsmith8954]

That's is an interesting thought William, I will reformat it tonight with a 64 bit version.

Should I get the one labeled "-amd64"?

Also, has any tried the 8.1 beta? Thought i might try that since it came after I dounloaded the image.

Thoughts?

 

[William Grzybowski]

Yeah, try amd64

I guess you mean 8.0.1-BETA? If you are not planning put it into a production without testing try http://nightly.freenas.org/FreeNAS-8.0.1-BETA3/AMD64/

 

[Buhmi]

Having the same issue. Using Freenas 8.0.1 Beta2 AMD64 on an Asrock A330 with 1.6Ghz, 4GB RAM and 4 WD20EARS configured as RAIDZ. Getting rates around 30-40mb/sec, which is fine for me (if anyone has a suggestion how to speed this up I would be happy to hear it anyway ;) ). But everytime I start copying a lot of stuff on the Server it freezes. No output on the screen, nothing. Just freeze :( Any ideas? CPU-Temps are normal, around 40-50°. I will try with only one RAM module and tell the result here...

 

[ProtoSD]

Buhmi,

Have you tried changing the default settings in /boot/loader.conf ?

You might want to do some reading on tuning ZFS here http://wiki.freebsd.org/ZFSTuningGuide

But pay attention to vm.kmem_size_max and vm.kmem_size just as a starting point, there are lots of guides and info if you search. Since you're using the AMD64 build, you won't need to recompile the kernel.

UPDATE: Just found this in another thread here and thought it would be helpful.
http://forums.freenas.org/showthread.php?408-ZFSGuru-features&p=1780&viewfull=1#post1780

 

[Tekkie]

I've got a AMD64 with 8GB RAM and see exactly the same problem when doing multiple transfers over Samba and iSCSI, kernel panics and the system becomes entirely unresponsive and I need to do a hard reset to get it back. :(

 

[Buhmi]

@protosd: Thanks for the advice, but I don't think ZFS is the problem. When writing an 8 gig testfile to my raid using dd (dd if=/dev/zero of=testfile bs=8192k count=1000) I get around 100mb/sec, so the speed is there. But when I copy large files via AFP from my Mac I'm stuck at 30-40mb/sec. But thats not the topic here anyway. I would even be happy with 40mb/sec if it wouldn't freeze randomly :(

 

[ProtoSD]

@Buhmi, I understand what you're saying, but I also had the same freezing during large copies from CIFS actually, but my system would hang and need a reboot also. Look at the ZFS / Kernel tuning, it really might help. It's not just ZFS, it's memory tuning... Of course you could see similar symptoms from bad RAM, but that's just my 2 cents. ;-)

 

[Buhmi]

Thanks again, maybe I'll look into that later. Does anyone have recommended settings for ZFS tuning on an Intel Atom A330 with 4GB? I'm a little afraid of destroying my data.

@protosd: did you manage do find out one single setting that caused the freezes? Or just try and error?

PS: Could anoyne tell me how do edit the loader.conf? nano tells me the file system is read only :(

 

[ProtoSD]

Buhmi

The settings can be a little confusing and it's more of a combination of several of them. I'll post the basic ones I think might help you and tell you how to edit loader.conf.

1) From the command line (SSH or console) type without the quotes 'mount -rw /'

2) cd /boot

3) cp loader.conf loader.conf.BAK

4) using your favorite text editor, edit loader.conf and add these lines, anywhere should be ok:

vm.kmem_size="1536M"
vm.kmem_size_max="2048M"
vfs.zfs.arc_min="256M"
vfs.zfs.arc_max="1024M"
#vfs.zfs.prefetch_disable="1"
#vfs.zfs.zil_disable="0"
vfs.zfs.txg.timeout="30"
vfs.zfs.vdev.max_pending="35"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.txg.write_limit_override=1073741824

4) save your file, and do 'mount -ro /'

5) reboot

Two of those are commented out, you can read about them and decided what you want to do with them.

None of them should destroy any of your data. If it works, you can read about them and adjust them yourself if needed. I'm not an expert, but I spent some time reading about them, I just don't remember off the top of my head what they all do. Actually the first two starting with 'vm_kmem' are probably your best ones to start with. If you want to comment out the others to start, those should help.