CIFS & MAC problems with special characters in folder names

[ovizii]

I have folders on my freenas which I share via CIFS. On my MAC I go to finder, CMD+K mount via smb://freenas/Pictures

Finder then shows folders with special characters but once I browse into them and try refreshing they disappear and they show as size 0 anyway. Then they pp up again and disappear.

http://screencast.com/t/eiZzXo13saSc
http://screencast.com/t/mdDLEgRx
http://screencast.com/t/fGo9E1Hu

Via SSH all looks good:

Code:

freenas# pwd
/mnt/fiver/Pictures/2006
freenas# ls
.DS_Store        2006 panoramas      
2006 balea        2006 thailand       
2006 fatboyslim        2006 holland jürgen  
freenas#

Via finder I can create folders with special characters though.
When I try to delete such a test folder created via finder from within ssh:

Code:

freenas# rmdir tänö

WHen I try to delete an existing folder it looks different. I think initially I transferred these problematic folders via rsync:

Code:

freenas# rmdir 2006\ holland\ ju<0308>rgen/

Any advice how to fix this? I can't transfer this data again, as I just deleted it locally, the copy on the freenas is my only one at this moment. I mean apart from go in and browse hundreds of folders and manually rename them???

###edit###
my CIFS settings:
DOS Charset: CP437
UNIX Charset: UTF-8

also found this option in the rsync man page but unsure on how its being used:
--iconv=CONVERT_SPEC request charset conversion of filenames

Also, this wouldn'T fix existing problems...

 

[dlavigne]

Which version of Mac? Also, which version of FreeNAS?

 

[ovizii]

Which version of Mac? Also, which version of FreeNAS?

Sorry, didn't think it'd be relevant:
OS X Yosemite Version 10.10.3 (14D136)
FreeNAS-9.3-STABLE-201505130355

 

[ovizii]

Ok, here is the whole situation, hopefully someone can help untangle it.

1. I copied most of my music library onto the NAS (from my MAC to a CIFS share on the NAS) - this should all be ok, no character display problems
2. In Between I deleted music from my MAC which now only resides on my NAS
3. At a later stage, I acquired more music on my MAC and used rsync to copy everything that was on my MAC but not on my NAS over to the NAS.
4. Step 3 apparently messed up and created character display problems with this newer part of my library

Basically everything on my MAC is fine and everything from step 1 on the NAS is fine, I just need a way to sync both onto either my MAC or my NAS, whichever is easier and delete the messed up part then copy everything back.

Manually this would be a nightmare :-(

 

[dlavigne]

Were you able to figure out a solution for this?

 

[ovizii]

Nope.
I manually fixed my picture collection as its fewer folders:

i.e.
2001
-country1
--event1
--event2
-country2
--event1

so I manually traversed and renamed every folder with a special character. file names are simply named dsc_xxx.nef or .jpg so no issues here.

But how to fix my music collection is beyond me as its about 200GB and the problem persists not only with folder names but also with file names.
Is there a way to search via command line for folders/files containing this type of characters?

Code:

ju<0308>rgen/

that should read

Code:

jürgen/

so basically "ü" becomes "u<0308>"

u<0308> seems to be unicode. can I search and replace unicode? Any other suggestions? Is that even advisable? It seems only my MAC can't "read" this unicode via a CIFS share. But If I login via ssh and simply rename this folder to "jürgen" I can read it just fine from my MAC via the same CIFS share.

 

[cyberjock]

There is probably a way to script something that can look for patterns of <xxxxx> and then convert it, but other than that I have no clue how to fix it yourself. When I moved my music collection to FreeNAS the first time I checked to make sure the names were right (they weren't) and I ultimately decided to copy my files over CIFS so I wouldn't have that problem. So I never had to fix anything, nor was I about to spend an inordinate amount of time trying to fix them. ;)

 

[ovizii]

Not sure why I can't find anything via Google either, I've tried searching along the lines of "rsync, character encoding problem special cifs" any other ideas? I'm sure I'm not the only one with this problem

 

[ovizii]

And just as I spoke up, I found something: http://serverfault.com/questions/39...ilenames-to-utf-8-nfc-in-either-rsync-or-afpd but that solution is not for fixing the problem but for proper usage of rsync + MAC + CIFS apparently there are two different UTF charsets: UTF-8 NFC & UTF-8 NFD

 

[ovizii]

So that thread has 2 options, any advice on what is better?

a) rsyncing between the source directory and the mounted remote file system (SMB, NFS, AFP), which rsync will just treat as local file system.

b) rsync -a --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

as far as I understand, a) would copy my music with the MAC UTF8 format, right, or does it get converted while writing to a CIFS share?
b) would convert to the the Linux UTF-8 NFC format but I'm not even sure if that's ok with freeNAS's BSD not being Linux?


###edit###
would this rsync command sync copy everything over even if its newer on the NAS? My intention is to force it to copy over everything, just to make sure there's nothing left of my previous attempts.

Code:

rsync  -P -r -t --ignore-times /Users/ovi/Music/iTunes/iTunes\ Media/ /Volumes/Music/

On the other hand, I only wanted to use rsync instead of a plain and simple copy job because I can resume but if I am ignoring differences and force-copying everything I can as well do a plain copy as I cannot resume this rsync copy anyway.

So all I need to know is if you guys suggest method a) or b)

 

[everyman]

A bit late to this party, but I've been having the same problem: everything was fine until we turned off AFP and switched to CIFS. Then folders everywhere started breaking due to special characters in the names ("/" and "." and "*" are particular favorites).

The solution is simple: temporarily enable AFP to copy the offending folder(s) to a client desktop; disable AFP, mount the CIFS share, and copy the folders back.

 

[trommegutten]

Special characters question mark ? \ * etc are problematic in smb samba share and I try to get rid of them. In the meantime it's frustrating, but a workaround is to manually map these "invalid" characters to other characters.

I use FreeNAS-11.3-U1 and SMB/Samba with MacOS Catalina and have found that my temporary solution to the character problem is:
Goto Sharing>SMB>Edit
1. Choose "advanced mode" and choose "VFS objects". Checkmark "catia"
2. in the auxiliary parameters paste in the following:

Code:

catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xa5,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6

 

[everyman]

Special characters question mark ? \ * etc are problematic in smb samba share and I try to get rid of them. In the meantime it's frustrating, but a workaround is to manually map these "invalid" characters to other characters.

I've taken a couple of different steps:

1. Admonish my users: Don't use characters like * : \ / & %
2. Create folders (and at least some documents) programmatically to avoid naming problems.
3. Many folder names are based on mailing addresses, so things like "c/o" must be replaced with "c-o" (and not, please, "%"). Automatic/programmatic folder creation helps with this.

I occasionally get a call about an older folder that is "missing." It is always due to names that were OK in in MacOS back in 1995 but not now. It's easier for me to deal with these as they crop up -- ssh to the server and rename the file or folder -- than to script it. There are thousands of folders going back more than 20 years.

A bigger problem, lately, is people being careless when they create Finder bookmarks for shares and moving the entire share instead. So far "Be careful!" is the only fix.

 

[trommegutten]

https://www.ixsystems.com/community...b-mangling-problem-on-macos.57865/post-579109

To other readers: forget my previous suggestion! Instead do as follow to make these windows-illegal characters work flawlessly between MacOS and FreeNAS SMB:
In FreeNAS webUI, goto Sharing>SMB>Edit
- Choose "advanced mode" and checkmark both catia and fruit in "VFS objects"
- Set auxiliary parameter to fruit:encoding = native
Then goto Services and restart SMB

I'm not sure why this encoding isn't by default.

 

[leafer]

Hello all:

Newbie here. Imported ext3 1TB HDD with Music and special characters. Looked ok at first, but lots of folders showing empty from WIN10 SMB if files in those folders have special characters. The 1TB HDD was never a problem under NASLite ... the special characters were probably created during the CD-ripping process (Song Titles) under Windows Media Player several MS-Windows iterations ago.

Has this issue been conclusively solved for Windows10 ? I would like to play those files via Groove Music or WMP.

TIA.
:) leafer

 

[ovizii]

https://www.ixsystems.com/community...b-mangling-problem-on-macos.57865/post-579109

To other readers: forget my previous suggestion! Instead do as follow to make these windows-illegal characters work flawlessly between MacOS and FreeNAS SMB:
In FreeNAS webUI, goto Sharing>SMB>Edit
- Choose "advanced mode" and checkmark both catia and fruit in "VFS objects"
- Set auxiliary parameter to fruit:encoding = native
Then goto Services and restart SMB

I'm not sure why this encoding isn't by default.

I moved away from freenas a while ago but this issue came up again and I was wondering if anyone has any input on which solution is better:

a) fruit:encoding = native

By setting fruit:encoding = native, all mapped characters are converted to native ASCII characters.

https://www.samba.org/samba/docs/current/man-html/vfs_fruit.8.html

b) catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6

SERVER_HEX_CHAR specifies a 0x prefixed hexedecimal character code that, when included in a Samba server-side filename, will be mapped to CLIENT_HEX_CHAR for the CIFS client.

The same mapping occurs in the opposite direction. Multiple character mappings are separated by a comma.

vfs_catia

 

[Constantin]

I had similar issues in FreeNAS but was able to overcome them in TrueNAS (12.xxx) when I switched all of my shares from AFP to SMB. The issue is with the protocol/implementation, not ZFS, as I had no issues whatsoever with umlauts or filename length in AFP that made SMB mangle away.

In TrueNAS, you can specify "Apple Style Character Encoding" in the Share setup page under "Other Options" (I.e. Sharing -> Windows Sharing (SMB) -> (select the triple dot next to the share name you want to edit on the far right) -> "Edit"). As best as I can tell, I have no more issues with mangled names - either due to length or due to unusual characters. Thank you @anodos and the rest of the iXSystems team for making that possible.

 

[ovizii]

I had similar issues in FreeNAS but was able to overcome them in TrueNAS (12.xxx) when I switched all of my shares from AFP to SMB. The issue is with the protocol/implementation, not ZFS, as I had no issues whatsoever with umlauts or filename length in AFP that made SMB mangle away.

In TrueNAS, you can specify "Apple Style Character Encoding" in the Share setup page under "Other Options" (I.e. Sharing -> Windows Sharing (SMB) -> (select the triple dot next to the share name you want to edit on the far right) -> "Edit"). As best as I can tell, I have no more issues with mangled names - either due to length or due to unusual characters. Thank you @anodos and the rest of the iXSystems team for making that possible.

thanks but as I've said I don't run freenas/truenas anymore. Would you mind sharing what the "?" reveals next to that "Apple Style Character Encoding" option? I guess in the background it will set one of the 2 smb options I have mentioned earlier.

 

[anodos]

Those above catia mappings are incomplete. Depending on whether vfs_fruit is enabled it will either set fruit:encoding = native, or add the catia mappings that fruit sets.

Code:

FRUIT_CATIA_MAPS = [
    "0x01:0xf001,0x02:0xf002,0x03:0xf003,0x04:0xf004",
    "0x05:0xf005,0x06:0xf006,0x07:0xf007,0x08:0xf008",
    "0x09:0xf009,0x0a:0xf00a,0x0b:0xf00b,0x0c:0xf00c",
    "0x0d:0xf00d,0x0e:0xf00e,0x0f:0xf00f,0x10:0xf010",
    "0x11:0xf011,0x12:0xf012,0x13:0xf013,0x14:0xf014",
    "0x15:0xf015,0x16:0xf016,0x17:0xf017,0x18:0xf018",
    "0x19:0xf019,0x1a:0xf01a,0x1b:0xf01b,0x1c:0xf01c",
    "0x1d:0xf01d,0x1e:0xf01e,0x1f:0xf01f",
    "0x22:0xf020,0x2a:0xf021,0x3a:0xf022,0x3c:0xf023",
    "0x3e:0xf024,0x3f:0xf025,0x5c:0xf026,0x7c:0xf027"
]

 

[anodos]

MacOS string replace maps are here: https://github.com/samba-team/samba/blob/master/source3/lib/string_replace.c
(see bottom of file).