/var Full Issues

[Kelly Shutt]

OK, so I'm new to FreeNAS but not Linux; been doing administrator work for 15+ years now. So, I have already done some troubleshooting on my own and I'm thinking maybe I just need a little guidance here.

I am running 8.2-RELEASE at this time, which in fact was just installed fresh a few days ago. I'm in the process of moving a TON of data onto a single RAIDZ volume using FTP transfers; this is 1.5TB of data in about 500K files. I'm getting great throughput on my gigabyte network BTW, which makes me very happy.

The issue I'm running into seems to have cropped up before, from what I see on the forums, where /var will suddenly fill up. I am receiving the email about RRD files not being written because the drive is full. In this case I have already checked all the common things folks have reported and can't figure it out.

From what I can tell, it seems like some deleted file descriptors are stuck; since my DF and DU output is completely different. If i run DF it shows var at like 109% usage, on a 149Mb parition, but when I DU on the folders I can only account for about 5-10Mb of usage. Unfortunately, I have rebooted since to clear the problem and don't have actual output to give you right now.

The other thing I noticed is that it's happened twice now in two days. I rebooted the first time and /var usage went back to normal, showing only 5-10Mb of actual usage. Now after having on less than a day it's done it again. I have tried getting lsof to install, to see what's going on in the filesystem, but for some reason I can't do that either. It tells me the package is installed but then when I look for /sbin/lsof it says the file doesn't exist.

Any advice would be helpful here. It seems to me like something is holding files open when it shouldn't, but I can't figure that out if I can't even make a simple tool work.

Or maybe letting me just make /var bigger since I have FreeNAS installed on it's own dedicated SSD with plenty of space. There's no reason I should be forced to use RAM disks when I have enough space, etc. to work with. Why can't I just make my own partitions and then modify /etc/fstab?

Overall, I am very happy but I think there's some fundamental tools missing here.


Peace,
Kelly Shutt

 

[yaneurabeya]

1. 3rd party packages install into /usr/local/ by default.
2. Try fstat instead of lsof.
3. /var (and several other mounts) are memory backed because [CF/USB] flash (the default assumed install media) is inherently slow and it would wear down the media quickly because the MTBFs are generally lower than HDDs depending on the number of operations performed (and there are a few...).

 

[Kelly Shutt]

So this is interesting... It seems the issue is related to the fact that I was doing a large transfer. Once this was complete I stopped seeing /var fill up and it's been running fine since then, in fact it's gone more than a week now without so much as a reboot. Maybe something with FTP was creating temporary files that were not closing properly?

I understand the reasoning behind the memory backed disks. However, it seems to me that SSD's are just becoming more and more affordable and popular, so at some point you need to consider accommodating them. In my case I bought new SSDs for all my machine to act as boot devices, and in the case of my FreeNAS machine it's basically going to waste. I would much rather have the option of just moving the partitions onto the SSD.


Peace,
Kelly Shutt

 

[yaneurabeya]

So this is interesting... It seems the issue is related to the fact that I was doing a large transfer. Once this was complete I stopped seeing /var fill up and it's been running fine since then, in fact it's gone more than a week now without so much as a reboot. Maybe something with FTP was creating temporary files that were not closing properly?

I understand the reasoning behind the memory backed disks. However, it seems to me that SSD's are just becoming more and more affordable and popular, so at some point you need to consider accommodating them. In my case I bought new SSDs for all my machine to act as boot devices, and in the case of my FreeNAS machine it's basically going to waste. I would much rather have the option of just moving the partitions onto the SSD.

I would file an enhancement ticket then at support.freenas.org . What you're probably looking at is samba filling up your memory disk with oplock related data, but that's just a WAG.

 

[Alan Johnson]

Hi Kelly,

Did you get anywhere with this?

I just had the same issue. I was able to give myself some elbow room by running this command(s) from in /var:

Code:

mv -v db/ /mnt/vol0/system/var/; ln -s /mnt/vol0/system/var/db

/mnt/vol0/system is just a zvol in my main zpool. That cleared up a few MB.

After that, I restarted CIFS based on yan's guess and found the used space on /var went back to a few hundred kB. (Thanks, yan!)

Just for good measure I moved /var/logs over to disk the same way. I don't expect these symlinks will survive a reboot, but at least the data will.

Did you file that feature request? Did you find any way to make such changes permanent? I don't see anything useful to edit in /etc/fstab even if I could, but that might be a bad idea anyway. I could boot off some other media, mount the / block device and add a symlink for /var entirely there to /mnt/vol0/system/var, but that would require down time and I can imagine that getting messy as well.

Then I wonder, does samba choke if it runs out of oplock space? I didn't check that before I bounced it. So, another help would be if we could samba to use some other place for oplocks, but I don't know anything about that either.

Anyone?