/var keeps getting full and my inbox being spammed to death every 5 minutes

Status
Not open for further replies.

MrAST

Cadet
Joined
May 7, 2013
Messages
6
Hello.

  • Build: FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty
  • RAM: 16GB
  • 2x 1TB raid setup
  • Boots of an 8GB USB Stick
  • Joined to a Windows AD


Box been up and running for approx 6 months with no issues.

Last week i started getting emails every 5 minutes from root:



Code:
Traceback (most recent call last):
 
  File "/usr/local/bin/graph.py", line 480, in <module>
 
    GenerateInterfaceGraph()
 
  File "/usr/local/bin/graph.py", line 135, in GenerateInterfaceGraph
 
    'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'
 
rrdtool.error: Could not save png to '/tmp/graph/if-em0-1h.png'



Was pressed for time so just rebooted the server and off i went

All good again until this weekend.....

Again being sent emails every 5 minutes with the same output as above.

Reading around it because my /var is getting full apparently.

Here are my outputs

Code:
[root@MYNAS ~]# df /var                                                       
Filesystem 512-blocks  Used  Avail Capacity  Mounted on                       
/dev/md2      305583 305466 -24329  109%    /var 


Code:
1      /var/.snap                                                             
1      /var/heimdal                                                           
1      /var/at/jobs                                                           
1      /var/at/spool                                                         
3      /var/at                                                               
1      /var/cron/tabs                                                         
2      /var/cron                                                             
1      /var/rwho                                                             
8      /var/backups                                                           
1      /var/preserve                                                         
609    /var/db/collectd/rrd/localhost/swap                                   
1521    /var/db/collectd/rrd/localhost/memory                                 
305    /var/db/collectd/rrd/localhost/uptime                                 
1521    /var/db/collectd/rrd/localhost/cpu-0                                   
2129    /var/db/collectd/rrd/localhost/processes                               
873    /var/db/collectd/rrd/localhost/load                                   
7704    /var/db/collectd/rrd/localhost/df                                     
14218  /var/db/collectd/rrd/localhost/interface                               
1521    /var/db/collectd/rrd/localhost/cpu-1                                   
1521    /var/db/collectd/rrd/localhost/cpu-2                                   
1521    /var/db/collectd/rrd/localhost/cpu-3
33444  /var/db/collectd/rrd/localhost                                         
33445  /var/db/collectd/rrd                                                   
33446  /var/db/collectd                                                       
115    /var/db/fontconfig                                                     
2      /var/db/samba/smb_krb5                                                 
1      /var/db/samba/winbindd_privileged                                     
8775    /var/db/samba                                                         
1      /var/db/portsnap                                                       
1      /var/db/pkg                                                           
1      /var/db/ipf                                                           
1      /var/db/freebsd-update                                                 
1      /var/db/ports                                                         
1      /var/db/entropy                                                       
2      /var/db/dbus                                                           
1      /var/db/nut                                                           
484    /var/db/graphs/hourly                                                 
411    /var/db/graphs/daily                                                   
457    /var/db/graphs/weekly                                                 
473    /var/db/graphs/monthly                                                 
473    /var/db/graphs/yearly                                                 
2299    /var/db/graphs                                                         
44647  /var/db                                                               
1      /var/run/wpa_supplicant
1      /var/run/ppp                                                           
1      /var/run/named                                                         
2      /var/run/proftpd                                                       
1      /var/run/samba/nmbd                                                   
5      /var/run/samba                                                         
2      /var/run/dbus                                                         
2      /var/run/avahi-daemon                                                 
50      /var/run                                                               
2      /var/crash                                                             
1      /var/tmp/vi.recover                                                   
1      /var/tmp/.cache/.snap                                                 
138    /var/tmp/.cache/.samba                                                 
169    /var/tmp/.cache/.query                                                 
121    /var/tmp/.cache/.ldap/.activedirectory/.local/.users/MYDOMAIN           
122    /var/tmp/.cache/.ldap/.activedirectory/.local/.users                   
145    /var/tmp/.cache/.ldap/.activedirectory/.local/.groups/MYDOMAIN         
146    /var/tmp/.cache/.ldap/.activedirectory/.local/.groups                 
269    /var/tmp/.cache/.ldap/.activedirectory/.local                         
129    /var/tmp/.cache/.ldap/.activedirectory/.users/MYDOMAIN                 
130    /var/tmp/.cache/.ldap/.activedirectory/.users                         
145    /var/tmp/.cache/.ldap/.activedirectory/.groups/MYDOMAIN                 
146    /var/tmp/.cache/.ldap/.activedirectory/.groups                         
546    /var/tmp/.cache/.ldap/.activedirectory
547    /var/tmp/.cache/.ldap                                                 
856    /var/tmp/.cache                                                       
1      /var/tmp/.winbindd                                                     
1      /var/tmp/nginx/proxy_temp                                             
1      /var/tmp/nginx/fastcgi_temp                                           
1      /var/tmp/nginx/uwsgi_temp                                             
1      /var/tmp/nginx/scgi_temp                                               
5      /var/tmp/nginx                                                         
1      /var/tmp/firmware                                                     
1967    /var/tmp/graph                                                         
2847    /var/tmp                                                               
1      /var/empty                                                             
1      /var/audit                                                             
1      /var/mail                                                             
1      /var/named/etc/namedb/master                                           
1      /var/named/etc/namedb/slave                                           
1      /var/named/etc/namedb/working                                         
1      /var/named/etc/namedb/dynamic                                         
5      /var/named/etc/namedb                                                 
6      /var/named/etc                                                         
1      /var/named/var/stats                                                   
1      /var/named/var/log                                                     
1      /var/named/var/run/named
2      /var/named/var/run                                                     
1      /var/named/var/dump                                                   
6      /var/named/var                                                         
1      /var/named/dev                                                         
14      /var/named                                                             
1      /var/account                                                           
1      /var/agentx                                                           
1      /var/log/samba/cores/nmbd                                             
1      /var/log/samba/cores/smbd                                             
1      /var/log/samba/cores/winbindd                                         
4      /var/log/samba/cores                                                   
254736  /var/log/samba                                                         
1      /var/log/proftpd                                                       
255440  /var/log                                                               
1      /var/spool/lock                                                       
1      /var/spool/output/lpd                                                 
2      /var/spool/output                                                     
1      /var/spool/opielocks                                                   
1      /var/spool/lpd                                                         
1      /var/spool/samba                                                       
1      /var/spool/mqueue                                                     
8      /var/spool                                                             
1      /var/games
2      /var/msgs                                                             
1      /var/yp                                                               
161    /var/etc/private                                                       
162    /var/etc                                                               
12      /var/home/domainuser1                                                         
12      /var/home/domainuser2                                                       
12      /var/home/domainuser3                                                         
12      /var/home/domainuser4                                                       
12      /var/home/domainuser5                                                       
12      /var/home/domainuser6                                                       
12      /var/home/domainuser7                                                         
12      /var/home/domainuser8                                               
12      /var/home/domainuser9                                                         
12      /var/home/domainuser10                                                         
12      /var/home/domainuser11                                                     
12      /var/home/domainuser12                                                       
12      /var/home/domainuser13                                                     
12      /var/home/domainuser14                                                         
12      /var/home/domainuser15                                                   
12      /var/home/domainuser16                                                       
12      /var/home/domainuser17                                                         
205    /var/home                                                             
303403  /var 


So i can see that the Samba dir is the biggest of the lot, but when i du inside of that it doesnt shed any light

Code:
[root@MYNAS /var/log/samba]# du /var/log/samba                               
1      /var/log/samba/cores/nmbd                                             
1      /var/log/samba/cores/smbd                                             
1      /var/log/samba/cores/winbindd                                         
4      /var/log/samba/cores                                                   
254736  /var/log/samba 


Anybody help me determine whats filling my stick up and why please?

Thanks in advance
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
One could move /var to persistent storage. Do a Google search of the forums with:

site:forums.freenas.org /var full
 

MrAST

Cadet
Joined
May 7, 2013
Messages
6
Id be more interested in working out why its logging so much so quickly all of a sudden when its been fine for months prior.

Just dont know where to look next
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
If you search as gpsguy suggests you'll see many reasons why it might be logging a lot. ;)
 

MrAST

Cadet
Joined
May 7, 2013
Messages
6
So I have found the actual file that's the culprit

Code:
[root@MYNAS /var/log/samba]# ls -al                                       
                                                           
-rw-r--r--  1 root  wheel  88353175 Oct  9 15:04 log.nmbd                 


It has over 1 million lines in that file in just over 2 days!!!!

Most of the file contains these kind of entries

Code:
  process_name_query_request: Name query from 192.168.1.28 on subnet 192.168.1.$
[2013/10/07 11:56:35,  3] nmbd/nmbd_incomingrequests.c:456(process_name_query_r$
  process_name_query_request: Name query from 192.168.1.30 on subnet 192.168.1.$
[2013/10/07 11:56:35,  3] nmbd/nmbd_incomingrequests.c:456(process_name_query_r$
  process_name_query_request: Name query from 192.168.1.28 on subnet 192.168.1.$
[2013/10/07 11:56:35,  3] nmbd/nmbd_serverlistdb.c:397(write_browse_list



I have a 15 user Windows AD domain - each user has a mapped drive to the main General share.

Logging level for CIFS was set to Full so I have knocked that down to Minimal.

Is there anything else apart from moving to persistant storage (which I cant do as I would have to reconfigure the whole raid setup to free up a disk) to solve this please?
Any why is it behaving like this now when its been fine for months previous with the same amount of users and in the same infrastructure?

I have searched as was suggested but all I find is people moaning about the same problem with no real solutions apart from moving /var

Thanks
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Well, you did do one thing, turned the logging down(which is a good choice). I do the minimal unless I'm troubleshooting. But you shouldn't normally be encountering millions of entries in just a few days. That's a dead giveaway that you have too much logging or you have something wrong with your setup. As the admin for your server its your responsibility to fix either/both.

If you've properly fixed your server then you shouldn't have a need to move /var. The people that are being told to move /var or that recommend it are taking the lazy approach and instead of fixing the problem(their configuration) they're fixing the symptom(making the log space bigger). There are plenty of users here that have no intention of doing things "right" and instead go for "easy". That might bite them in the booty later. And that will be totally at their peril.
 
Status
Not open for further replies.
Top