Dates and times - kernel log messages

Status
Not open for further replies.

James S

Explorer
Joined
Apr 14, 2014
Messages
91
I get the daily security update from FN 11.1 I'm not clear when exactly (date, time) the events reported have occured. The mail normally comes through at 03.00 or so. Surely the dates/times of these events should be part of the report. Is something getting stripped out in transit here?

Example:
Code:
freenas.local kernel log messages:
> WARNING: 10.0.1.2 (iqn.1998-01.com.vmware:5b8bfc
4f-89f9-4faf-6621-38d547021ae0-19a48e7b): no ping reply (NOP-Out) after 5 seconds; dropping connection
> WARNING: 10.0.0.2 (iqn.1998-01.com.vmware:5b8bfc4f-89f9-4faf-6621-38d547021ae0-19a48e7b): no ping reply (NOP-Out) after 5 seconds; dropping connection
> WARNING: 10.0.1.2 (iqn.1998-01.com.vmware:5b8bfc4f-89f9-4faf-6621-38d547021ae0-19a48e7b): no ping reply (NOP-Out) after 5 seconds; dropping connection
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
No. These are, as it says, kernel log messages.

The kernel does not place timestamps on console output, so what's in the system message buffer is generally sans-timestamp. Historically, this is probably because the VT100 that you hooked up to your VAX had an 80-column display and you didn't want to waste precious screen real estate. The system message buffer is valuable because it has a number of unique qualities. For one, it requires nothing other than the kernel - it is not dependent on filesystem I/O, etc. - so anything that the kernel thinks an operator or a developer would need to be aware of is dumped here, and can be retrieved via the debugger or via a core file. Also, it is capable of surviving reboots, assuming the platform BIOS doesn't destroy it, so a panic caused by an I/O lockup can still be logged and analyzed when the system reboots, as the messages remain visible by the new running system.

syslogd picks up this data, and is responsible for timestamping it and depositing it into the system logs. You should find similar timestamped messages available in the system logs.

If you find the un-timestamp-ed messages distressing, you may suppress them by setting daily_status_security_kernelmsg_enable to false.
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
I kind of get the historic rational. I had seen this issue reported on IXsystems pages as a potential bug.
I'm that sensitive as to find the absence of time-stamped messages distressing :) A bit more coffee should get through it ...
It just not useful when, for example, trying to work out when a problem occurs (e.g., under / not under load).
Could you give a pointer for which log I should plow into? Thanks!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I kind of get the historic rational. I had seen this issue reported on IXsystems pages as a potential bug.
I'm that sensitive as to find the absence of time-stamped messages distressing :) A bit more coffee should get through it ...
It just not useful when, for example, trying to work out when a problem occurs (e.g., under / not under load).

UNIX has a strong history of being built out of smaller, simpler, more robust components that can be combined to accomplish functionality. Some other systems were historically designed so that the "print" program would format your text file and send it to the I/O ports in such a way as to make the manufacturer's provided printer print your output. This presents a problem if you maybe wanted to use a different kind of printer, or save the formatted text, or do anything else that the designers hadn't considered. UNIX, instead, makes text formatting a separate issue ("pr file"), queueing ("lpd"), multiple printer types ("printcap"), printing to a remote site ("teh interwebz"), etc., so that you can pin all these bits together in whatever way you need, or even replace some of them ("CUPS").

The system message buffer is intertwined with console output for historical reasons, probably because UNIX mainframe operators were typically on duty and "paying attention", and would hear the console teletype printer or see it on their VT. But it was also understood that these things needed to be stored, so you can pipe the stuff to syslogd, and you can even set syslogd to send logging messages across the network for central recording, reporting, and analysis, where you pick up the timestamping and other meta-capabilities.

The downside to the syslog-maintained files is that they can be relocated, they can be intermingled with other messages, they can be huge, and they can also fill the disk, resulting in a loss of messages sent while the disk is full. I imagine that the authors of the periodic cron scripts felt that the use of the "dmesg" output in the daily report was a pragmatic solution to these problems. The system message buffer is always available (even if the disk fills), it is size-limited (the buffer has a defined size), and it is not mixed in with other messages, meaning that there's no need to process a potentially huge log file. But it isn't time-stamped. You should probably regard the message as a clue that something went awry on your system that you need to investigate further, rather than expecting it to be a comprehensive and complete problem description.

Mind you, my opinion is flavored by three decades of BSD. :smile:

Could you give a pointer for which log I should plow into? Thanks!

syslogd reads the system message buffer as it is able, timestamps the messages, and handles the messages as specified by /etc/syslog.conf.

For FreeNAS, it would normally end up in /var/log/messages, except security.*, which ends up in /var/log/security
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
Well that made for an interesting read :) It is a bit a reminder of how far we have come - from rooms populated with operators on duty - to individuals remoting to their machines. I'm not technically trained so while the routine procedures can be worked through the troubleshooting requires more effort.

Thanks for the specifics on the log. It takes time to work out what the log is saying aside from working out which one to look at.
 
Status
Not open for further replies.
Top