Debugging a crash - persist logfiles

[jedcred]

I'm getting this issue where at random intervals, my freenas 8 box drops off the network and is no longer available. However, the console screen is still active (the screen where you can select options 1-9), but selecting any of the options results in a hang.

I wanted to start at the debug logs, but unfortunately the read-only nature of the freenas image makes it so that logfiles do not persist through reboots.

My initial idea was to make the change to syslog.conf to put the same output in /var/log/messages in my zfs store (/mnt/tank/logs/messages, for example). I did find discussions around here about persisting changes by mounting / as writable using 'mount -uw /' and making the changes I wanted, then remounting as read-only and rebooting; however, my changes unfortunately do not seem to persist (my line in the syslog.conf is no longer there).

The system runs off a usb key. My initial thought was that the system was entering some power save mode after some time and not bringing the usb drive back up properly, but without log files to at least check, it's a bit difficult.

Any ideas, in terms of better ways to debug or maybe correctly configuring syslog.conf?

TIA.

 

[ProtoSD]

Hmmm, you could try a 'tail -f log-you-want-to-watch > /mnt/tank/persistent.log'

(/var/log/messages or /var/log/console)

The stuff on the USB key gets expanded into one or more RAM disks. If your idea about power save mode was true, I don't know if they get preserved in that state or not, you would think so, but ???

I think some of the config files in /conf? get preserved, but I don't know the details or if it'll help you.

 

[jedcred]

In the last week, the freenas box has gone down three times. I tried logging in and running tail -f /var/log/messages hoping to catch the last few log messages before the machine becomes responsive, but no dice. The only messages are from earlier in the day when it was working, not in the window of time where it failed.

EDIT: I should mention that most of the hardware in this box is rather well tested, as it passes memtest86 and (save for the video card) has served as my desktop machine for several years before being turned into a freenas box. No guarantee, mind you, but at least that's some more information.

 

[jedcred]

Wanted to follow up in case others should encounter this. Problem was simply that I had the 32-bit version of Freenas, not the 64-bit by accident. ZFS doesn't seem to play well with 32-bit address space, and attempts to allocate more memory result in a crash. 8.0.0 doesn't give you any output when this happens, but thankfully 8.0.1 did.

 

[deed02392]

I am also having problems with the FreeNAS box dropping off the network, yet clearly still 'running'. Surely someone has a solution to get damn persistent logging working. I would have thought that would be bare minimum of a Linux based machine.

 

[cyberjock]

You can setup a syslog if you want persistent logging.