Syslogd went bye bye. Will not start.

Status
Not open for further replies.
P

praecorloth

Contributor
Joined
Jun 2, 2011
Messages
159
Hey folks. So I've got FreeNAS 8.0.4 running on a box, syslog has gone away and will not start again.


# /etc/rc.d/syslogd start
Starting syslogd.
syslogd: child pid 90069 exited with return code 1
/etc/rc.d/syslogd: WARNING: failed to start syslogd

Obviously nothing more gets logged, so I'm kind of at a loss. We're going to try giving it a reboot, but due to the nature of the box, it may be a while before we can try that. Does anyone have any other suggestions?
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
P

peterh

Patron
Joined
Oct 19, 2011
Messages
315
start syslogd manually and add '-d' :

/usr/sbin/syslogd -s -C -d

this prevents syslogd to daemonize and will use stdout/stderr for messages. If needed add -v for more verbose.

Break with ^C when finished debugging.
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
Ok then:
Code:
[root@daniel] ~# /usr/sbin/syslogd -s -C -d
syslogd: bind: Address already in use
logmsg: pri 53, flags 4, from , msg syslogd: bind: Address already in use
Logging to CONSOLE /dev/console
syslogd: bind: Address already in use
logmsg: pri 53, flags 4, from , msg syslogd: bind: Address already in use
Logging to CONSOLE /dev/console
can't open /dev/klog (16)
off & running....
init
cfline("*.err;kern.warning;auth.notice;mail.crit		/dev/console", f, "*", "*")
cfline("*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	/var/log/messages", f, "*", "*")
cfline("security.*					/var/log/security", f, "*", "*")
cfline("auth.info;authpriv.info				/var/log/auth.log", f, "*", "*")
cfline("mail.info					/var/log/maillog", f, "*", "*")
cfline("lpr.info					/var/log/lpd-errs", f, "*", "*")
cfline("ftp.info					/var/log/xferlog", f, "*", "*")
cfline("cron.*						/var/log/cron", f, "*", "*")
cfline("*.emerg						*", f, "*", "*")
cfline("*.*						/var/log/ppp.log", f, "ppp", "*")
4 3 2 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 X CONSOLE: /dev/console
7 5 2 5 5 5 6 3 5 5 X 5 5 5 5 5 5 5 5 5 5 5 5 5 X FILE: /var/log/messages
X X X X X X X X X X X X X 7 X X X X X X X X X X X FILE: /var/log/security
X X X X 6 X X X X X 6 X X X X X X X X X X X X X X FILE: /var/log/auth.log
X X 6 X X X X X X X X X X X X X X X X X X X X X X FILE: /var/log/maillog
X X X X X X 6 X X X X X X X X X X X X X X X X X X FILE: /var/log/lpd-errs
X X X X X X X X X X X 6 X X X X X X X X X X X X X FILE: /var/log/xferlog
X X X X X X X X X 7 X X X X X X X X X X X X X X X FILE: /var/log/cron
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 X WALL: 
7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 X FILE: /var/log/ppp.log (ppp)
logmsg: pri 56, flags 4, from daniel, msg syslogd: restart
syslogd: restarted
logmsg: pri 6, flags 4, from daniel, msg syslogd: kernel boot file is /boot/kernel/kernel
Logging to FILE /var/log/messages
syslogd: kernel boot file is /boot/kernel/kernel

^Csyslogd: exiting on signal 2
syslogd: exiting on signal 2
logmsg: pri 53, flags 4, from daniel, msg syslogd: exiting on signal 2
Logging to CONSOLE /dev/console
Logging to FILE /var/log/messages
[root@daniel] ~#


And the more verbose one:
Code:
[root@daniel] ~# /usr/sbin/syslogd -s -C -d -v
syslogd: bind: Address already in use
logmsg: pri 53, flags 4, from , msg syslogd: bind: Address already in use
Logging to CONSOLE /dev/console
syslogd: bind: Address already in use
logmsg: pri 53, flags 4, from , msg syslogd: bind: Address already in use
Logging to CONSOLE /dev/console
can't open /dev/klog (16)
off & running....
init
cfline("*.err;kern.warning;auth.notice;mail.crit		/dev/console", f, "*", "*")
cfline("*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	/var/log/messages", f, "*", "*")
cfline("security.*					/var/log/security", f, "*", "*")
cfline("auth.info;authpriv.info				/var/log/auth.log", f, "*", "*")
cfline("mail.info					/var/log/maillog", f, "*", "*")
cfline("lpr.info					/var/log/lpd-errs", f, "*", "*")
cfline("ftp.info					/var/log/xferlog", f, "*", "*")
cfline("cron.*						/var/log/cron", f, "*", "*")
cfline("*.emerg						*", f, "*", "*")
cfline("*.*						/var/log/ppp.log", f, "ppp", "*")
4 3 2 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 X CONSOLE: /dev/console
7 5 2 5 5 5 6 3 5 5 X 5 5 5 5 5 5 5 5 5 5 5 5 5 X FILE: /var/log/messages
X X X X X X X X X X X X X 7 X X X X X X X X X X X FILE: /var/log/security
X X X X 6 X X X X X 6 X X X X X X X X X X X X X X FILE: /var/log/auth.log
X X 6 X X X X X X X X X X X X X X X X X X X X X X FILE: /var/log/maillog
X X X X X X 6 X X X X X X X X X X X X X X X X X X FILE: /var/log/lpd-errs
X X X X X X X X X X X 6 X X X X X X X X X X X X X FILE: /var/log/xferlog
X X X X X X X X X 7 X X X X X X X X X X X X X X X FILE: /var/log/cron
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 X WALL: 
7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 X FILE: /var/log/ppp.log (ppp)
logmsg: pri 56, flags 4, from daniel, msg syslogd: restart
syslogd: restarted
logmsg: pri 6, flags 4, from daniel, msg syslogd: kernel boot file is /boot/kernel/kernel
Logging to FILE /var/log/messages
syslogd: kernel boot file is /boot/kernel/kernel
^Csyslogd: exiting on signal 2
syslogd: exiting on signal 2
logmsg: pri 53, flags 4, from daniel, msg syslogd: exiting on signal 2
Logging to CONSOLE /dev/console
Logging to FILE /var/log/messages
[root@daniel] ~#

(looks the same to me though)

It says "can't open /dev/klog (16)". What does that mean?

Regards,
Memel
 
P

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
logmsg: pri 53, flags 4, from , msg syslogd: bind: Address already in use
Click to expand...

To me this seems like the real problem. It's trying to connect to an IP address that is either not a syslog server or is a duplicate IP address.

What do you have in your settings for the Syslog server (GUI System->Settings)?

What is the IP address of your FreeNAS system?
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
my system's IP address is 192.168.178.16
syslog server field in GUI is empty (I don't remember ever having changed that; my FreeNAS box always worked flawlessly until a few days ago).
 
P

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Hmmm, well it seemed like a good thing to check. It's time to sleep here, maybe I'll have a new idea tomorrow or someone else will think of something sooner!
 
P

peterh

Patron
Joined
Oct 19, 2011
Messages
315
you already have one syslogd running.
( "bind: Address already in use" )

confirm with 'ps -ax | grep syslog'
then kill the old syslog and start a new
 
StephenFry

StephenFry

Contributor
Joined
Apr 9, 2012
Messages
171
This may not be too helpful, but I have learned that it's often much quicker to do a fresh FreeNAS install than to troubleshoot an existing one.

It's usually possible to load the settings from the 'broken' installation, as well, without the problem popping up - so nothing is lost.

I recognize the urge to "know what's wrong" and to want to fix it in a neat manner, but a day only has 24 hours ;)
 
P

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
StephenFry said:
This may not be too helpful, but I have learned that it's often much quicker to do a fresh FreeNAS install than to troubleshoot an existing one.

It's usually possible to load the settings from the 'broken' installation, as well, without the problem popping up - so nothing is lost.

I recognize the urge to "know what's wrong" and to want to fix it in a neat manner, but a day only has 24 hours ;)
Click to expand...


Very good answer! I realize the hacker in us likes to know "why", but sometimes it's faster to just move on.
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
@peterh
Thanks! It indeed turned out that syslogd was already running. I've killed it and restarted it. Now wait and see what happens..

Could it by any chance be related to the error message "maxproc limit exceeded by uid 0"? Should I start a new thread for this particular problem?

Thanks everyone.,
Memel
 
P

peterh

Patron
Joined
Oct 19, 2011
Messages
315
If you reach any of
kern.maxproc:
kern.maxprocperuid:
then you have other problems to deal with. Are you running thousedns of cifs-clients ??
Find out what causing so many processes and stop it.
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
Ok, my results so far:

Killing and restarting syslogd seems to have worked; I now receive the daily and security run output in my mailbox.

What struck me as odd was the fact that when I rebooted my system, syslogd (always) is running as process 1070, but without any syslogd.pid file in /var/run. Manually killing & restarting does result in a syslogd.pid file though. Does anyone have any thoughts on this perhaps?

As for the "maxproc limit exceeded", I'm still waiting for it to happen again. I did some searching though, and found nothing conclusive as of yet, only that it happens sometimes: kernel: maxproc limit exceeded - The FreeBSD Forums. Some of my settings are:
Code:
[root@daniel] ~# limits
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize             33554432 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked         infinity kB
  maxprocesses             5547
  openfiles               11095
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
[root@daniel] ~# sysctl -a | grep kern.max
kern.maxvnodes: 138916
kern.maxproc: 6164
kern.maxfiles: 12328
kern.maxfilesperproc: 11095
kern.maxprocperuid: 5547
kern.maxusers: 384
kern.maxssiz: 536870912
kern.maxdsiz: 34359738368
kern.maxtsiz: 134217728
kern.maxbcache: 0
kern.maxswzone: 33554432

I am not running any cifs clients at all right now.

My system specs (for those interested):
Code:
FreeNAS Build 	FreeNAS-8.0.4-RELEASE-x64 (10351)
Platform 	AMD Athlon(tm) II X2 260 Processor
Memory 	3824MB
System Time 	Sat May 26 10:57:01 2012
Uptime 	10:57AM up 1 day, 7:56, 1 user
Load Average 	0.00, 0.00, 0.00
OS Version 	FreeBSD 8.2-RELEASE-p6

on an ASUS M4A88T-M motherboard.

Thanks for your advice & help so far.

Memel
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
Well, I think it happened again :-(
Code:
May 25 09:17:46 freenas[1798]: Calling: restart(cron)
May 25 09:17:46 freenas[1798]: Executing: /usr/sbin/service ix-crontab quietstart
May 25 09:17:46 freenas[1798]: Executed: /usr/sbin/service ix-crontab quietstart
May 25 09:17:46 freenas[1798]: Executing: /usr/sbin/service cron restart
May 25 09:17:46 freenas[1798]: Executed: /usr/sbin/service cron restart
May 25 09:48:45 daniel syslogd: exiting on signal 15

I don't get any response from the WebGUI, ssh or the console. Only ping works. That's going to be a hard reset.

I'm beginning to think I should do a clean install...

Memel
 
P

praecorloth

Contributor
Joined
Jun 2, 2011
Messages
159
Well my mystery is solved. So my friend went to restart a couple of services on his FreeNAS box due to unrelated issues. He was unable to turn the services back on. Reboot, FreeNAS gave some error about the kernel (don't blame me, I wasn't there to see this :) ) and that was that. He went to reinstall, but the FreeNAS installer threw errors when trying to install to the USB drive. Long story short, (too late) bad USB stick. The Microcenter brand USB sticks have really gone to shit, in case anyone is wondering. We've had so dang many failures of late it's not even funny.
 
memel.parduin

memel.parduin

Dabbler
Joined
Feb 13, 2012
Messages
42
Did a hard reset yesterday.

However, still having problems with syslogd. I receive an email message saying
Code:
newsyslog: can't notify daemon, pid 5348: No such process

It seems that syslogd is running as process 1070 (ps -ax), while /var/run/syslog.pid says 5348??

I do receive the daily run output in my email though.

Memel
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Syslogd went bye bye. Will not start."

Similar threads

morxy49
  • Locked
failed to start proftpd
Replies
8
Views
3K
dlavigne
D
T
  • Locked
UPS config problem - WARNING: $nut_upsshut is not set properly
Replies
12
Views
4K
keithg
K
nello
  • Locked
CrashPlan Will NOT Start
Replies
8
Views
4K
t3hG04T
t3hG04T
A
  • Locked
Issue with Transmission daemon in 8.0.4 Multimedia
Replies
0
Views
3K
almagest
A
A
  • Locked
Plex Media Server plugin will not start.
Replies
2
Views
2K
Jailer
Jailer
Top