File system full - now what???

[TravisT]

I've been copying files over to my newly configured freenas server. I've been rsync-ing everything over, and to keep an eye on things, have used the --progress and --stats options. Because of the large amount of data I'm copying over, I think the logs have filled up the partition on my USB drive.

I ssh'd to the server and deleted the messages.* logs and a couple other files that were really large. Now I don't get any messages in the GUI. I tried creating a new blank messages file, but that didn't seem to fix it. Have I screwed something up by deleting the files this way? The error emails have stopped, so I think the original problem is fixed, as long as I didn't create new ones.

Also, what can I do to prevent this from happening again?

 

[William Grzybowski]

Try restarting syslog...

# service syslogd restart

 

[TravisT]

Tried a restart, but even before that I still received a few emails every 5 minutes. Looks like there are still some issues with disk space I assume. None since restart, so maybe that is the key?

If anyone can comment as to how to prevent this in the future, that would be appreciated.

 

[peterh]

did you examine which filesystems that got full ? ( df -h )
And if it was your /var ( located on usb ) what was the messages that filled it up ?
Chances are that messages contained information about malfunctioning stuff ..

 

[TravisT]

Here's my df -h output:

Code:

Filesystem               Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs1a      927M    353M    499M    41%    /
devfs                    1.0K    1.0K      0B   100%    /dev
/dev/md0                 4.6M    1.8M    2.4M    43%    /etc
/dev/md1                 824K    2.5K    756K     0%    /mnt
/dev/md2                 149M    149M    -12M   109%    /var
/dev/ufs/FreeNASs4        20M    1.7M     17M     9%    /data
tank                      42G    292K     42G     0%    /mnt/tank
tank/Files               109G     67G     42G    62%    /mnt/tank/Files
tank/Jail                 42G    444M     42G     1%    /mnt/tank/Jail
tank/Movies              1.5T    1.4T     42G    97%    /mnt/tank/Movies
tank/Plugins              42G    182M     42G     0%    /mnt/tank/Plugins
moose                    556G    213K    556G     0%    /mnt/moose
moose/Databases          574G     19G    556G     3%    /mnt/moose/Databases
moose/Files              616G     60G    556G    10%    /mnt/moose/Files
moose/Media              3.1T    2.6T    556G    82%    /mnt/moose/Media
moose/Reference          590G     34G    556G     6%    /mnt/moose/Reference
moose/Software           913G    357G    556G    39%    /mnt/moose/Software
moose/Sort               563G    6.6G    556G     1%    /mnt/moose/Sort
/dev/md3                 1.9G    544K    1.7G     0%    /var/tmp/.cache
/mnt/tank/Plugins         42G    182M     42G     0%    /mnt/tank/Jail/transmission/mnt/plugins
/mnt/tank/Plugins/pbi     42G    182M     42G     0%    /mnt/tank/Jail/transmission/usr/pbi
devfs                    1.0K    1.0K      0B   100%    /mnt/tank/Jail/transmission/dev
procfs                   4.0K    4.0K      0B   100%    /mnt/tank/Jail/transmission/proc

I spent the weekend on the river kayaking and camping. Returned to over 600 emails with this subject line:

Code:

Cron <root@freenas> /usr/local/bin/python /usr/local/bin/graph.py

and this body:

Code:

Traceback (most recent call last):
  File "/usr/local/bin/graph.py", line 480, in <module>
    GenerateInterfaceGraph()
  File "/usr/local/bin/graph.py", line 135, in GenerateInterfaceGraph
    'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'
rrdtool.error: Could not save png to '/tmp/graph/if-em0-1m.png'

I had this problem once in the past when I enabled snapshots. Some part of the filesystem filled up and had similar problems. I ended up reinstalling freenas and turning off snapshots. I would really like to figure out a long term solution to this, but otherwise I will be re-installing tomorrow.

 

[carl0s]

yikes..

Code:

Filesystem               Size    Used   Avail Capacity  Mounted on
/dev/md2                 149M    149M    -12M   109%    /var

 

[TravisT]

Exactly. Seems like this could pose serious problems in stability for a system that logs a lot of messages. That is assuming the message logs caused this...