Space is not released.

[CraXyOW3]

Well, as the title says.
I have a disk where all misc content is downloaded to, later it get sorted and then deleted from this "dumpdisk".
The sole purpose is just to be a workhorse for temporary content, be it something shared or downloaded/uploaded.
The problem I have is that once a download is completed, moved and deleted from the disk, the space that file occupied is not released, thus "filling" the drive.

A restart of the NAS solves it, bit that imo is not a solution.
I have checked the snapshots, nothing, even made sure the disk will not be snapshoted.

Been checking the internets for a solution and many have had similar but not the same problem, none of the other solutions worked for me.

Any help/advise is greatly appreciated.

 

[jgreco]

In UNIX, files can be held open by a process even after the file has been unlinked, and the blocks backing the file do not get freed until the last reference to the file is no longer in use. The directory entry for the file is removed from the directory but the inode and data blocks remain allocated until the file is closed by all processes using the file. This is colloquially known as an "anonymous file".

In a NAS context, it is possible for a remote process, such as a Samba or NFS client, to have opened a file, and for you to locally remove the file on the NAS, but the running NAS server process is still "using" the file. Rebooting the NAS kills the NAS server process, which causes the space to be actually freed (even though the remote client might still think it has the file open).

There are three fixes to this problem, one of which you discovered:

1) You can reboot the NAS, which is sort of a bludgeony fix, and which can leave the remote client accessing the file in a strange state. This does force the closure of the file locally.

2) You can find the process that is accessing the file. The "fstat" command will happily provide information on what process on the local system is accessing an inode, so you can use "ls -ali your/file/name" to identify the inode, and then use "fstat" to find it. This will work for processes such as Samba and jailed processes, but probably not for NFS, which is serviced by the kernel. Killing the process(es) holding the file open will close the file and release the space.

3) You can shorten ("truncate") the file prior to removing it. So if you are looking at /mnt/yourpool/your/file/name and saying "oh heck that's a terabyte" but some process has the file open for reading, doing an "rm /mnt/yourpool/your/file/name" does not release the space because a process is still using the file. You can do "truncate -s 0 /mnt/yourpool/your/file/name" followed by "rm /mnt/yourpool/your/file/name" which will zero out the file and probably cause the process with the file open to see "end of file". However, if the process has the file open for writing, and you truncate it, this will merely free the written blocks so far, but the process may still continue to write additional blocks.

So the real solution here is to make sure that all consumers of the file close the file, and the space will automatically be freed.

 

[sretalla]

I'm not sure if this is the same thing, but I have noticed in a jail that moves a lot of files around that it doesn't release the space on the mount that has the transitory/temp files. I can correct it by restarting that jail and there was a thread talking about a bug that was slated to be fixed in some version around 11... I never saw what happened to it, but it still seems to be there.

 

[CraXyOW3]

Awesome answers! Will dig into a bit more now.
In my current usecase that I have most problems with is, transmission downloads, application seeds, stops at ratio. Moves file to another location which is where I would think the space should be freed.

Restarting the jail with transmission released it, bit only restarting transmission inside the jail did not solve it.
Will test more and thanks for a good explanation!

 

[sretalla]

Since it seems we're talking about the same thing here, I should probably note in addition that the jail mounts that are impacted by this are usually nested datasets. Apparently it wasn't impacting the jail/mount if you mount each dataset directly.

 

[CraXyOW3]

Since it seems we're talking about the same thing here, I should probably note in addition that the jail mounts that are impacted by this are usually nested datasets. Apparently it wasn't impacting the jail/mount if you mount each dataset directly.

My setup is what you describe.
Jail is on another drive than the dumpdisk, which is just a dumpdisk, no other purpose.
Also one of the mounts is a nested dataset, but mounted by "rootfolder".
Going to double check my fstab. Tyvm