[Errno 28] No space left on device: '/var/db/system/tmp00NMSH'

[BDMcGrew]

Hello!

I have two identical systems running FreeNAS-9.3 and one of them is throwing this error when I check for updates. Screenshot and df output attached below. Not sure why they are different nor how to fix it.

Help greatly appreciated :) Both are hosting iSCSI volumes for VMware and have run beautifully for quite some time.

[root@nas24] ~# df -h
Filesystem Size Used Avail Capacity Mounted on
freenas-boot/ROOT/Pre-FreeNAS-9.3-STABLE-201602031011-295290 26G 552M 26G 2% /
devfs 1.0k 1.0k 0B 100% /dev
tmpfs 32M 6.2M 25M 19% /etc
tmpfs 4.0M 8.0k 4M 0% /mnt
tmpfs 16G 245M 15G 1% /var
freenas-boot/grub 26G 6.1M 26G 0% /boot/grub
zdev_1 219k 219k 0B 100% /mnt/zdev_1
zdev_1/jails 219k 219k 0B 100% /mnt/zdev_1/jails
zdev_1/.system 433M 433M 0B 100% /var/db/system
zdev_1/.system/cores 3.7M 3.7M 0B 100% /var/db/system/cores
zdev_1/.system/samba4 292k 292k 0B 100% /var/db/system/samba4
zdev_1/.system/syslog-0dc2ca1e7fa9464d8c4d7c4fd81f6855 766k 766k 0B 100% /var/db/system/syslog-0dc2ca1e7fa9464d8c4d7c4fd81f6855
zdev_1/.system/rrd-0dc2ca1e7fa9464d8c4d7c4fd81f6855 219k 219k 0B 100% /var/db/system/rrd-0dc2ca1e7fa9464d8c4d7c4fd81f6855
zdev_1/.system/configs-0dc2ca1e7fa9464d8c4d7c4fd81f6855 9.6M 9.6M 0B 100% /var/db/system/configs-0dc2ca1e7fa9464d8c4d7c4fd81f6855
[root@nas24] ~#

 

[Spearfoot]

This thread may help, it deals with exactly your problem.

 

[BDMcGrew]

I only have 2 boot env's on each box...

 

[Mirfster]

What is the output of zpool list ? Curious to see the amount of free space on the boot devices...

Example:

Code:

 ~# zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ASC-JBOD01    43.5T  1.51M  43.5T         -     0%     0%  1.00x  ONLINE  /mnt
ASC-V01       5.44T  95.9G  5.34T         -     0%     1%  1.00x  ONLINE  /mnt
freenas-boot  31.8G  1.21G  30.5G         -      -     3%  1.00x  ONLINE  -

 

[BDMcGrew]

[root@nas24] ~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 27.8G 578M 27.2G - - 2% 1.00x ONLINE -
zdev_1 57T 5.27T 51.7T - 26% 9% 1.00x ONLINE /mnt
[root@nas24] ~#

((( How can I reduce that fragmentation percentage? 26% seems very high. )))

This box has a single zvol which is used as an iSCSI device. Last night I found some random article in the gargler that mentioned reducing the size of the zvol so on a whim I did that and now all the sudden I have free space in /var again. But... I also found on reboot that I have two bad hard drives. DOH!!!

So that begs another interesting question... I did not get an email from FreeNAS. I get the daily security run output mails and I get the mails from cron when it starts scubs but I didn't get any mail about disk failures. I'm very concerned as to why now...

thanks,

-b

 

[Mirfster]

So that begs another interesting question... I did not get an email from FreeNAS. I get the daily security run output mails and I get the mails from cron when it starts scubs but I didn't get any mail about disk failures. I'm very concerned as to why now...

Did you have e-mail properly configured?
What FreeNas version are you running?
I believe that there was some concerns with an earlier version that was not properly sending e-mails; which is fixed in a recent update.

Sticking this in CODE Tags for easier reading:

Code:

[root@nas24] ~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 27.8G 578M 27.2G - - 2% 1.00x ONLINE -
zdev_1 57T 5.27T 51.7T - 26% 9% 1.00x ONLINE /mnt
[root@nas24] ~#

 

[BDMcGrew]

Well, I think mail is configured correctly. I do receive the test message, I get the daily security run outputs and I get the monthly mail from cron when a scrub is started. This would imply email is correctly, configured, right?

 

[danb35]

What FreeNas version are you running?

 

[Mirfster]

Yes, that should signify that you do have that setup correctly.

Please see @danb35 's post "Why didn't I get a warning on this?" for more info.

/Lol, he just replied... Speak of the Devil... ;)

 

[BDMcGrew]

FreeNAS-9.3-STABLE-201602031011

I know it's a bit old but I was trying to do the updates last night :)

 

[danb35]

I don't know in which version the email bug first appeared, but it is fixed in 9.10.1.