Freenas 9 - GUI down - /var/log full

[Gaspard]

Hello Community!

I have an old freenas 9 that has been running for years without a problem but today the GUI stopped working, I can still access the SSH fine.

When I try to restart nginx or django it is complaining that :

touch: /var/log/nginx-error.log: No space left on device
touch: /var/log/nginx-access.log: No space left on device

When i try do delete these files or to put them to zero I have the same error.

When I run df I see that there is still plenty of space for /var but I also see some .system files full and that /var/log is pointing to /var/db/system/...

Filesystem Size Used Avail Capacity Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511040813 28G 534M 28G 2% /
devfs 1.0k 1.0k 0B 100% /dev
tmpfs 32M 5.3M 26M 17% /etc
tmpfs 4.0M 8.0k 4M 0% /mnt
tmpfs 10G 130M 10G 1% /var
....
stor01/.system 321M 321M 0B 100% /var/db/system
stor01/.system/cores 792k 792k 0B 100% /var/db/system/cores
stor01/.system/samba4 176k 176k 0B 100% /var/db/system/samba4
stor01/.system/syslog-d646df02c1474ae3a4f2283ac4ea185f 46M 46M 0B 100% /var/db/system/syslog-d646df02c1474ae3a4f2283ac4ea185f
stor01/.system/rrd-d646df02c1474ae3a4f2283ac4ea185f 96k 96k 0B 100% /var/db/system/rrd-d646df02c1474ae3a4f2283ac4ea185f
stor01/.system/configs-d646df02c1474ae3a4f2283ac4ea185f 141M 141M 0B 100% /var/db/system/configs-d646df02c1474ae3a4f2283ac4ea185f

I am quite newbie to freenas and I don't know where to look?

Thanks in advance for any help!

Gaspard

 

[sretalla]

I am quite newbie to freenas and I don't know where to look?

I have an old freenas 9 that has been running for years

I guess you're not so new... just didn't pay much attention for some years.

Since we're talking about FreeNAS 9.3, I think we're still on UFS for boot... zpool list -v should give us a better idea of that.

If you're on ZFS and actually 100% full, you may have no alternative than to grab a copy of your DB and wipe/reinstall/restore the db. (the db is in /var/db/system/configs-d646df02c1474ae3a4f2283ac4ea185f )

If it's UFS, maybe there's something that can be done to free the space with some deletions.

 

[Gaspard]

I guess you're not so new... just didn't pay much attention for some years.

Since we're talking about FreeNAS 9.3, I think we're still on UFS for boot... zpool list -v should give us a better idea of that.

If you're on ZFS and actually 100% full, you may have no alternative than to grab a copy of your DB and wipe/reinstall/restore the db. (the db is in /var/db/system/configs-d646df02c1474ae3a4f2283ac4ea185f )

If it's UFS, maybe there's something that can be done to free the space with some deletions.

Hello and thank you for your pointers.

Here is the result of the zpool list -v

I am not sure if I am on UFS boot, it is Freenas 9.3 yes.

 

[sretalla]

That just told us that you're on ZFS... freenas-boot is your boot pool.

So it must be a quota problem as you are consuming only 2% of your 29.8G and there's plenty of space on the stor01 pool where you have your system dataset.

Have a look in /var/db/system/cores and see if there are large files there that can all be deleted, they are only process crash dumps.

 

[sretalla]

Also, have a look at zfs get quota stor01/.system/syslog-d646df02c1474ae3a4f2283ac4ea185f and zfs get quota stor01/.system/cores

Or even zfs get quota | grep .system if the ones above provide nothing useful.

 

[Gaspard]

Thank you for helping out @sretalla

Here is the output of the following commands, I don't see any quota being set. Could it be a pb with the snapshot? But the snapshot should fill up the disk space (so showing up in DF).

 

[sretalla]

OK, so we still haven't really found what's causing the problem yet.

Maybe try this:

ncdu -xq /var/db/system

And se if you can find a file that looks out of place there.

 

[Gaspard]

OK, so we still haven't really found what's causing the problem yet.

Maybe try this:

ncdu -xq /var/db/system

And se if you can find a file that looks out of place there.

Hello @sretalla thanks again for your help, unfortunaltey this command is not available for this freenas version apparently.

Is it normal that the size of the stor01/.system files are small and not the siez of the /var partition?

I have rebooted the unit, I have acces to the GUI again but still some menu are not displaying and I cannot still delete anything from /var/db/system....

 

[sretalla]

Is it normal that the size of the stor01/.system files are small and not the siez of the /var partition?

Yes.
The contents of your /var/system... are mounted/coming from your system dataset, which is on your stor01 pool.

I don't know if it will work, but moving the system dataset to the boot pool might be something that could move the situation forward... otherwise plugging in a reasonable-sized USB stick and making a pool on that and moving the system dataset to that pool... you would then want to move it back to a more permanent location fairly quickly, but it may solve the situation of being stuck with some kind of limit as you are now.

 

[sretalla]

There may be some thinking to do about the way that 9.3 used to do the root FS in RAM... maybe using mount -uw / would put you in a position to delete some of those files?

 

[Gaspard]

I don't know if it will work, but moving the system dataset to the boot pool might be something that could move the situation forward...

Yes that fixed the pb, I can now remove the files and the logs are filling up as they should. Thanks a lot! (sorry for the late reply but we were moving everything off this NAS in order to be able to try things and upgrade it).

However I still not have access to all the menu in the GUI:

The update menu is not showing up, I will try to do an update via the CLI and see how it goes.

EDIT: I updated through CLI and get on 9-10 train, got the GUI fully working yeah ;-)

What is the best practice for the dataset?

Thanks again @sretalla