bzip2: I/O or other error, bailing out. Possible reason follows. bzip2: No space left on device

[Bobuser]

FreeNAS-8.3.1-RELEASE-p2-x64


Ive been using freenas for about a year - with no issues and amybe 2 reboots. All of a sudden Im getting these arrors every hour

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No space left on device
Input file = /mnt/VOLUME.3TB/DS.3T/logs/messages.0, output file = /mnt/VOLUME.3TB/DS.3T/logs/messages.0.bz2
bzip2: Deleting output file /mnt/VOLUME.3TB/DS.3T/logs/messages.0.bz2, if it exists.
newsyslog: `bzip2 -f /mnt/VOLUME.3TB/DS.3T/logs/messages.0' terminated with a non-zero status (1)

The mount in teh error message has plenty of space on the physical filesystem. /mnt is 103% used...??

here is the storage layout.

[bob@DIME521] /mnt/VOLUME.3TB/DS.3T/logs# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs1a 926M 386M 466M 45% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/md0 4.6M 3.3M 959k 78% /etc
/dev/md1 823k 783k -25k 103% /mnt
/dev/md2 149M 18M 119M 13% /var
/dev/ufs/FreeNASs4 19M 1.3M 17M 7% /data
VOLUME.1TB 673G 409G 263G 61% /mnt/VOLUME.1TB
VOLUME.1TB/jail 267G 3.8G 263G 1% /mnt/VOLUME.1TB/jail
VOLUME.1TB/nas 499G 236G 263G 47% /mnt/VOLUME.1TB/nas
VOLUME.1TB/software 263G 410M 263G 0% /mnt/VOLUME.1TB/software

/dev/ufs/disk3 2.7T 1.4T 1.1T 57% /mnt/disk3
Im very confusted by this error. Any insight or help would be appreciated!
Thanks

 

[dlavigne]

Does "du /mnt | sort -nr | head" show any large files?

 

[Durkatlon]

Where in that output of "df -ah" does it actually show the VOLUME.3TB mounted? I only see shares on VOLUME.1TB.

 

[Bobuser]

Where in that output of "df -ah" does it actually show the VOLUME.3TB mounted? I only see shares on VOLUME.1TB.

Thanks for your reply
your command is looking through the entire filesystem where there are alot of large files

so, the big question is - my 2 physical disks have plenty of free space - why is /mnt showing 103% used and Im getting these errors



/mnt/disk3 is a 3tb disk
/mnt/VOLUME.1TB is 1tb
so, there is only those 2 physical disks
and I have a 4g usb disk that freenas is on - thats it




[bob@DIME521] /mnt/# du /mnt | sort -nr | head
2202265344 /mnt
1520930708 /mnt/disk3
681333852 /mnt/VOLUME.1TB
534475332 /mnt/disk3/Images
374469692 /mnt/disk3/Music
310605114 /mnt/VOLUME.1TB/nashare
296283115 /mnt/VOLUME.1TB/nashare/gcopys
265662884 /mnt/disk3/Images/TBI
247163883 /mnt/VOLUME.1TB/nas
176337240 /mnt/disk3/Video



[bob@DIME521] /mnt/# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs1a 926M 386M 466M 45% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/md0 4.6M 3.3M 959k 78% /etc
/dev/md1 823k 784k -26k 103% /mnt
/dev/md2 149M 18M 119M 13% /var
/dev/ufs/FreeNASs4 19M 1.3M 17M 7% /data
VOLUME.1TB 673G 409G 263G 61% /mnt/VOLUME.1TB
VOLUME.1TB/jail 267G 3.8G 263G 1% /mnt/VOLUME.1TB/jail
VOLUME.1TB/nas 499G 236G 263G 47% /mnt/VOLUME.1TB/nas
VOLUME.1TB/software 263G 410M 263G 0% /mnt/VOLUME.1TB/software
/dev/ufs/disk3 2.7T 1.4T 1.1T 57% /mnt/disk3



[bob@DIME521] /mnt/# mount
/dev/ufs/FreeNASs1a on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /mnt (ufs, local)
/dev/md2 on /var (ufs, local)
/dev/ufs/FreeNASs4 on /data (ufs, local, noatime, soft-updates)
VOLUME.1TB on /mnt/VOLUME.1TB (zfs, NFS exported, local, nfsv4acls)
VOLUME.1TB/jail on /mnt/VOLUME.1TB/jail (zfs, local, nfsv4acls)
VOLUME.1TB/nas on /mnt/VOLUME.1TB/nas (zfs, local, noatime, nfsv4acls)
VOLUME.1TB/software on /mnt/VOLUME.1TB/software (zfs, local, nfsv4acls)
/dev/ufs/disk3 on /mnt/disk3 (ufs, local, soft-updates, nfsv4acls)



My message log is filled with this...
Sep 9 06:00:00 DIME521 rsync: rsync error: error starting client-server protocol (code 5) at main.c(1517) [Receiver=3.0.9]
Sep 9 06:01:00 DIME521 kernel: pid 1373 (syslogd), uid 0 inumber 87 on /mnt: filesystem fulls
Sep 9 06:14:00 DIME521 last message repeated 10 times
Sep 9 06:24:00 DIME521 last message repeated 10 times
Sep 9 06:34:00 DIME521 last message repeated 10 times
Sep 9 06:35:00 DIME521 kernel: pid 1373 (syslogd), uid 0 inumber 87 on /mnt: filesystem full
Sep 9 06:35:49 DIME521 su: bob to root on /dev/pts/0
Sep 9 06:36:00 DIME521 kernel: pid 1373 (syslogd), uid 0 inumber 87 on /mnt: filesystem full
Sep 9 06:37:00 DIME521 kernel: pid 1373 (syslogd), uid 0 inumber 87 on /mnt: filesystem full
Sep 9 06:39:00 DIME521 last message repeated 2 times
Sep 9 06:49:00 DIME521 last message repeated 10 times
...
Thanks for looking!!
Bob

 

[Bobuser]

also a screen shot... attached confirming the unix df

output

 

[Durkatlon]

My question regarding "df -ah" was, what is mounted on /mnt/VOLUME.3TB?

It looks from both the df output and the mount output, that nothing is mounted there. This means that this location (/mnt/VOLUME.3TB) is simply part of your root mount ("/"), which is a RAM disk.

This also means that the file that it is trying to zip in your original post (/mnt/VOLUME.3TB/DS.3T/logs/messages.0) is on your RAM disk, not on any big disk that you have attached to this system.

Maybe I'm misunderstanding, but that's what it looks like to me.

 

[Bobuser]

Hi I missed your originga question ...

"Where in that output of "df -ah" does it actually show the VOLUME.3TB mounted?"

VOLUME.3TB is a disk that was deconfigured (destroyed, reformatted) and presented as teh new /mnt/disk3

Somehow freenas did not do house keeping and there are still remains of that old configuration

WOW!!

So this may unroot the problem....

/mnt/VOLUME.3TB/DS.3T/VOLUME.3TB/DS.3T/log

is NOT a mount...
its an old path on the root of /mnt (not sure how that happened...)

Ive placed the system logs there....

hummm... Im going to clean that up :)

Very excellent eye!!



[bob@DIME521] /mnt/VOLUME.3TB/DS.3T/logs# ls -ltr
total 782
-rw------- 1 root wheel 63 Aug 29 20:24 xferlog
-rw-r--r-- 1 root wheel 63 Aug 29 20:24 ups.log
-rw------- 1 root wheel 63 Aug 29 20:24 security
-rw-r----- 1 root network 63 Aug 29 20:24 ppp.log
-rw-r--r-- 1 root wheel 63 Aug 29 20:24 lpd-errs
drwxr-xr-x 3 root wheel 512 Aug 29 20:24 ../
-rw-r--r-- 1 root wheel 130 Sep 1 05:30 monthly.log
-rw-r----- 1 root wheel 121 Sep 2 00:00 maillog.7.bz2
-rw-r----- 1 root wheel 1136 Sep 2 00:00 daily.log.7.bz2
-rw-r----- 1 root wheel 119 Sep 3 00:00 maillog.6.bz2
-rw-r----- 1 root wheel 1142 Sep 3 00:00 daily.log.6.bz2
-rw-r----- 1 root wheel 121 Sep 4 00:00 maillog.5.bz2
-rw-r----- 1 root wheel 1136 Sep 4 00:00 daily.log.5.bz2
-rw-r----- 1 root wheel 121 Sep 5 00:00 maillog.4.bz2
-rw-r----- 1 root wheel 1142 Sep 5 00:00 daily.log.4.bz2
-rw-r----- 1 root wheel 122 Sep 6 00:00 maillog.3.bz2
-rw-r----- 1 root wheel 1140 Sep 6 00:00 daily.log.3.bz2
-rw-r----- 1 root wheel 104 Sep 7 00:00 weekly.log.0.bz2
-rw-r----- 1 root wheel 62 Sep 7 00:00 weekly.log
-rw-r----- 1 root wheel 123 Sep 7 00:00 maillog.2.bz2
-rw-r----- 1 root wheel 1145 Sep 7 00:00 daily.log.2.bz2
-rw------- 1 root wheel 3550 Sep 7 23:00 cron.3.bz2
-rw-r----- 1 root wheel 124 Sep 8 00:00 maillog.1.bz2
-rw-r----- 1 root wheel 1142 Sep 8 00:00 daily.log.1.bz2
-rw------- 1 root wheel 3791 Sep 8 06:00 cron.2.bz2
-rw------- 1 root wheel 3516 Sep 8 13:00 cron.1.bz2
-rw------- 1 root wheel 3391 Sep 8 20:00 cron.0.bz2
-rw-r----- 1 root wheel 124 Sep 9 00:00 maillog.0
-rw-r----- 1 root wheel 62 Sep 9 00:00 maillog
-rw-r----- 1 root wheel 2551 Sep 9 00:00 daily.log.0
-rw-r--r-- 1 root wheel 716854 Sep 9 03:00 messages.0
-rw-r--r-- 1 root wheel 7598 Sep 9 03:01 security.log
-rw-r----- 1 root wheel 2526 Sep 9 03:01 daily.log
-rw-r--r-- 1 root wheel 0 Sep 9 06:35 bob
-rw------- 1 root wheel 1119 Sep 9 06:35 auth.log
-rw------- 1 root wheel 1119 Sep 9 06:36 auth.log.bob
drwxr-xr-x 2 root wheel 1024 Sep 9 06:45 ./
-rw-r--r-- 1 root wheel 3532 Sep 9 20:05 messages
-rw------- 1 root wheel 19952 Sep 9 20:06 cron



[bob@DIME521] /mnt/VOLUME.3TB/DS.3T/logs# df -h .
Filesystem Size Used Avail Capacity Mounted on
/dev/md1 823k 784k -26k 103% /mnt