undeletable file in ZFS volume

[parpar]

I have a problem deleting/opening a file in Freenas 8.3.0 on a raidz-1 volume with lzjb compression:

This is what ls says:

Code:

ls
ls: 02 Walking By Myself.m4a: No such file or directory
./            ../ 

and find lists it normally:

Code:

find . 
.
./02 Walking By Myself.m4a

I cannot get rid of the "02 Walking By Myself.m4a" file, nor its enclosing directory. Touch, rm -rf, chmod, getfacl all claim the file does not exist and do nothing. The file system is nowhere full (nor has it ever been).

I did a scrub on the volume, didn't find nor repaired any errors.
zdb on its inode gives:

Code:

zdb -dddddd zdata/SharedData 42456
<snip>
   Object  lvl   iblk   dblk  dsize  lsize   %full  type
zdb: dmu_bonus_hold(42456) failed, errno 2

Any suggestions how I can rid of that darned file?

 

[HolyK]

I guess the whitespace could be the "problem". Use single quote arround the filename or escape the space, so..

rm '02 Walking By Myself.m4a'
or
rm 02\ Walking\ By\ Myself.m4a

BTW: Get rid of the spaces in filenames on unix ;)

 

[parpar]

That is exactly what I did HolyKiller. I also checked the name for any other UTF8 hidden characters and performed a

Code:

find . -exec rm -rf {} \;

to no avail.
It is also not a permission problem: I am root and working from an ssh session

I believe this is a bug in ZFS or undetected corruption by scrub (which is why Im posting it here)

Any other suggestions?

 

[HolyK]

Do you have any quota set for that dataset?

Or you can try this:

Code:

> 02 Walking By Myself.m4a

or maybe a bit better is this:

Code:

cp -f /dev/null '02 Walking By Myself.m4a'

and then try to remove it again.

 

[parpar]

There is no quota set (0) on that dataset, which has 2.7 TB free. The results of your other two suggestions are:

Code:

cp -f /dev/null 02\ Walking\ By\ Myself.m4a 
cp: 02 Walking By Myself.m4a: No such file or directory

and

Code:

> 02\ Walking\ By\ Myself.m4a 
Invalid null command.

but also

Code:

echo > 02\ Walking\ By\ Myself.m4a 
02 Walking By Myself.m4a: No such file or directory.

No luck..

 

[William Grzybowski]

Well, I don't see any other way around other than create a new dataset, move everything and then destroy the old dataset.

What FreeNAS version are you running?

 

[parpar]

Thanks William,
I'm on FreeNAS 8.3.0 (FreeBSD 8.3-RELEASE-p5 64bit on an AMD64). Any idea how this mishap can occur? Although it is not a problem to recreate the dataset (I have a backup + numerous snapshots), I'd like to know what could cause it and/or prevent it from happening again. Any suggestions on how to best move everything (clone a snapshot and promote it? rsync the dataset, zfs send | receive?)

Thanks for the support,

Paul

 

[parpar]

Just discovered that the error replicated to my backup server as well (synced with autorepl.py), so

Code:

zfs send | zfs receive

may not be a good idea to get rid of the file..

 

[William Grzybowski]

No idea how that happened, some kind of obscure bug I'd think.

I would use rsync to another dataset, as you figured b yourself send + recv might replicate the bug as well.

 

[joeschmuck]

Could you just share that directory as an FTP share and then use something like Filezilla, then delete the file that way?

 

[parpar]

@joeschmuck: I doubt whether ftpd has a different method to unlink a file than rm/unlink. Just to make sure I tried your suggestion on the dataset that contained the error and tried to remove the containing directory and the following error was reported:

Code:

550 /mnt/zdata/oldShared/..../oo: Directory not empty.

the oo directory contained only the undeletable file, which is not shown in the FTP list (just as 'ls' doesn't do it but reports an error).

I meanwhile followed William Grzybowski's advice and rsynced the dataset over to a fresh one. As expected rsync didn't copy over the hidden and undeletable file, so I expect the new dataset is error free. Fingers corssed it won't happen again to more important files..

Thanks all for the support!

Paul