SOLVED Disk full - Can't delete any files. Please Help!

[an303042]

Hello,

First let me say - I am very new to freenas. This is my first machine, and I have done very little with it so far.
I have two 3TB disks (mirrored) and its used for Acronis true image making backups from 3 computers on my home network.
Acronis was a little hungry and didn't delete old backups, and now the nas volume is totally full. As a result, i can't delete any of the files on the nas. this is pretty funny imo, and makes me think that zfs might not really be ready for general use.
But, before i jump to conclusions, i need to get myself out of this sticky situation.
i found this thread: http://forums.freenas.org/showthread.php?10246-8-3-Out-of-space-cannot-delete and even thought i didn't really understand everything that's written there, i tried it out myself, but kept getting the "No space left on device" error, when i tired executing the commands mentioned through the shell in the freenas webgui.
So, any help would be very appreciated! first to delete some files, and second to prevent this from happening in the future.
Thanks!

 

[cyberjock]

this is pretty funny imo, and makes me think that zfs might not really be ready for general use.

I'm sorry, but I LOL'd at that. Enterprises use this, so I'd say it's been ready for "general use" for years...

AFAIK there is no recovery from a zpool that is 100% full. That's a drawback with CoW file systems. I believe your only fix is to destroy the zpool and start over(obviously copy any data you want first).

I don't use Acronis anymore(too many problems with reliability over networks) but Acronis has the feature built-in to delete old snapshots. As an alternative you could modify a script I use from http://forums.freenas.org/showthrea...te-old-backup-file-(Acronis-OO-DiskImage-etc) to help solve your problem. If you use the script though you will need to always perform a full backup(no incremental or differential) and Acronis will complain that backups are mysteriously disappearing.

 

[an303042]

I'm sorry, but I LOL'd at that. Enterprises use this, so I'd say it's been ready for "general use" for years...

No need to apologize, and I'm always happy to entertain! What i meant is that it's pretty funny to need empty space on a disk in order to delete a file. I stand by that. Also, enterprise use isn't really "general use", but I'm beginning to feel that maybe going with FreeNAS for my home use (I'm not an IT guy, but do consider myself very tech savy) wasn't the greatest idea. I'll stick with it for now, though, mostly cause I can't find a really suitable alternative.

Regardless of humor issues, I kept looking around for a solution to my problem while waiting for my post to show up on the board (which took foreverrrrrr), and some awesome people on the FreeNAS IRC channel managed to help me out. So in case anyone gets to this thread with a similar problem, here's what i did (written for noobs, by a noob):
open a shell on the freenas machine (i used the one in the webgui)
type "echo > /path/to/file/you/want/to/delete"
problem solved. :)

Thanks for your advice, cyberjock, and thanks to everyone on the IRC channel!

Edit: Oh, and as for preventing this from happening in the future - After people suggested "self restraint" (HA!) I went with creating a dataset with reserved space. Haven't been able to test this yet, but i hope it will work.

 

[jgreco]

Worth noting that ZFS may need more than a trite amount of space, so make it a large file if you can. ZFS pools should never be filled to capacity.

 

[cyberjock]

No need to apologize, and I'm always happy to entertain! What i meant is that it's pretty funny to need empty space on a disk in order to delete a file. I stand by that. Also, enterprise use isn't really "general use", but I'm beginning to feel that maybe going with FreeNAS for my home use (I'm not an IT guy, but do consider myself very tech savy) wasn't the greatest idea. I'll stick with it for now, though, mostly cause I can't find a really suitable alternative.

You are kinda-sorta correct about needing space to free up more space. It has to do with how CoW file systems work. It is a natural consequence of CoW. As for your feeling with FreeNAS, it isn't for the faint at heart. If you aren't really into IT stuff you may find you are in far over your head. Many people that "work in IT" have lost their data from not knowing what to do(and what not to do). Hopefully you won't fall into the category of losing your data.

open a shell on the freenas machine (i used the one in the webgui)
type "echo > /path/to/file/you/want/to/delete"
problem solved. :)

WOW! I didn't think of that as a solution. Very interesting idea indeed. I'm glad you got it figured out and I learned something. :P

 

[jgreco]

WOW! I didn't think of that as a solution. Very interesting idea indeed. I'm glad you got it figured out and I learned something. :P

Yeah, that's been the canonical solution (fsvo "solution") to the problem for some time now. Another poster ran across this late last year. I was a bit surprised that such a failure mode had been allowed in the ZFS design, but I did some followup on it and it seems that the "nuke a large file or a snapshot" approach has been the only available fix for about as long as ZFS has been around. It makes a little more sense when you consider that ZFS's designers didn't intend for a ZFS pool to exceed ~~80% capacity, and that things such as snapshots make for additional complications. While a reserved allocation of some sort could make this better for some people in some situations, it isn't comprehensive. On the other hand, I'm not entirely certain that the "nuke a large file" solution works in every scenario either, so ... this is a bit of a shortcoming and I wish that the designers of ZFS had addressed this a bit better.

 

[an303042]

Another poster ran across this late last year.

I found that thread while looking for a solution, but didn't really understand what they were saying there... :o

 

[jgreco]

Yeah, sorry, William's too terse, I'm too wordy. Sometimes when there's no great answer, the unsatisfactory answer is buried in a bunch of mostly-unhelpful words. I was kind of hoping to see if I could tease a response out of William about possible solutions.

 

[Fregor]

im having the same issue, but im afraid im not understanding what to do properly.

In the shell command in the GUI im writing as follows

echo < X:\Film\K\Klovn

but its not reacting at all :/

 

[warri]

Don't use the windows mount path - it does not exist on the FreeNAS system. The files will be somewhere in your /mnt folder, where your pool is mounted.

Also you might want to read some more about output redirection, your command does not make sense at all: http://www.codecoffee.com/tipsforlinux/articles2/042.html

 

[fracai]

im having the same issue, but im afraid im not understanding what to do properly.

In the shell command in the GUI im writing as follows

echo < X:\Film\K\Klovn

but its not reacting at all :/

Your bracket is pointing in the wrong direction. You want ">" not "<". Pointing right means write; pointing left means read. Your command is trying to read the file instead of write to it.
You're also using the Windows path, as warri, stated. Don't do that.

Code:

echo > /mnt/poolname/path/to/Film/K/Klovn

Hah, just noticed that warri did cover the output redirection as well.

 

[Fregor]

Hey Thanks to both of you for the replys, Yeah i noticed i went it the wrong way i usually do > sorry for that one.

i found out the fautly reaction came from using IE, it didnt even show me pressing enter ^^..

Well im getting closer any way but im quite not there yet, so please bear with me and have a bit more patience while im asking :)..

So my pool name is in this case /mnt/film3

so the command should be like this right..

Echo > /mnt/film3/folder name for example like this

Echo > /mnt/film3/Film/K/Klovn

im finding the pool name understorage in the GUI.. "Film3 /mnt/Film3 2.7 TiB (100%) None 2.7 TiB HEALTHY"

Your bracket is pointing in the wrong direction. You want ">" not "<". Pointing right means write; pointing left means read. Your command is trying to read the file instead of write to it.
You're also using the Windows path, as warri, stated. Don't do that.

Code:

echo > /mnt/poolname/path/to/Film/K/Klovn

Hah, just noticed that warri did cover the output redirection as well.

 

[warri]

Is /mnt/film3/Film/K/Klovn a file or folder? It has to be a file you want to delete - not a folder.
Also linux commands are case-sensitive. Lowercase echo exactly like shown in fracais post.

 

[Fregor]

Thanks, that was the missing link..

so heres the final work... echo > /mnt/Film3/Replace

but it dont work it says theres no space left on the drive. DAMMIT >_<

 

[warri]

Did it show any error message? Can you please post the output of

Code:

ls -al /mnt/Film3/<additional path here>

 

[William Grzybowski]

You must have snapshots then...

zfs list -t snapshots

 

[Fregor]

here you go

http://tinypic.com/view.php?pic=2vm62r4&s=5
no snappies

 

[William Grzybowski]

sorry, zfs list -t snapshot, not snapshotS

 

[Fregor]

http://i42.tinypic.com/2qvtf06.jpg

no snaps taken never set it for that :o

 

[benjidad]

So my pool name is in this case /mnt/film3

so the command should be like this right..

Echo > /mnt/film3/folder name for example like this

I have similar problem...

It works! now I can save some space that previously impossible to delete (as the files/folders deletion process complete, the files/folders eventually come back there once I refresh the file manager)

what is the explanation by the way?