APFS to SMB filename issues with rsync, what a mess!

[Sean Hafeez]

1. rsyncd (current working solution) is removed in Cobia
2. want to rsync from local machine to SMB mounted share to replace #1
3. all systems clients are MacOS with files on APFS systems
4. TrueNAS shares are SMB with "Enable Apple SMB2/3 Protocol Extensions" checked.
5. mount SMB on Mac. run rsync which wants to delete / replace 1000+ files (same with or without utf-8-mac,utf-8 set)
6. okay, let it do it. run again to see if things are synced. wants to delete and replace same files again.
8. remove everything it wants to delete by hand (just in case it is having issue removing files).
9. run again, files copied up. run again to check. same issue.

Okay, so I did a scrub of the community post and there is not a working answer for this as far as I can tell. Setting fruit:encoding = native in SMB results in shares that can be seen in Finder but disappear when you try to mount them.

Here is an example of the file/folder names:

deleting Wolfgang Amadeus Mozart/Cosi Fan Tutte (Concerto Köln, René Jacobs)/
deleting Wolfgang Amadeus Mozart/Don Giovanni (Ghiaurov, Janowitz, Burrows, Zylis‐Gara, Evans, Herbert Von Karajan)/51 Don Giovanni - Atto Secondo - Ah Dove È Il Perfido.m4a

 

[Patrick M. Hausen]

Have you tried running rsync over SSH instead?

 

[Sean Hafeez]

That is an option, but I was hoping to be able to workout why rsync has a different view of the world when it is used on the locally mounted SMB share and why it can never be made consistent.

 

[winnielinnie]

Have you tried running rsync over SSH instead?

I second this. You bypass the "middleman", which is SMB, and do a straight transfer between two native filesystems from client to server.

You won't face the same issues with filenames and timestamp granularity.

 

[Patrick M. Hausen]

There are a couple of components at work here, obviously

- Samba and its treatment of UTF-8
- TrueNAS configuration of Samba plus possibly additional VFS modules
- Apple's SMB client, specifically its treatment of some special characters - I cannot find the reference right now, but there's some hardwired weirdness in it
- Apple's implementation of rsync

Check if Apple finally updated rsync to a current version on Sonoma or whatever you are running. If not, install a current one via Homebrew or Macports and try something like rsync --iconv=utf-8-mac,utf-8.

All failing try the rsync mailing lists.

 

[Sean Hafeez]

As I said, I have been using the rsyncd service on TrueNAS and I know just running with ssh should work, however the issue is why is there an inconsistent view of the file system when using SMB? In playing with this I find that rsync and cp will create the same folder / file from the point of view of the standard ls when ssh's to the server. For example there will be to "versions" of Björk. Yes I know if I look harder they will show different encoding. When viewing that share from Finder on MacOS I will only see one "version" of that folder. The inconsistent view is the issue. The creation of 2 version because of the encoding means that I do not have a source of truth for the filesystem. I now never know if something done with cp will be correct if I rsync stuff later.

 

[Sean Hafeez]

As I said - rsync --iconv=utf-8-mac,utf-8 does not change things when the rsync is to local mounted SMB share. I am running rsync from homebrew rsync version 3.2.7 protocol version 31.

The reality is I cannot trust my tools to be consistent and that is the larger issue I would like to figure out.

 

[morganL]

1. rsyncd (current working solution) is removed in Cobia
2. want to rsync from local machine to SMB mounted share to replace #1
3. all systems clients are MacOS with files on APFS systems
4. TrueNAS shares are SMB with "Enable Apple SMB2/3 Protocol Extensions" checked.
5. mount SMB on Mac. run rsync which wants to delete / replace 1000+ files (same with or without utf-8-mac,utf-8 set)
6. okay, let it do it. run again to see if things are synced. wants to delete and replace same files again.
8. remove everything it wants to delete by hand (just in case it is having issue removing files).
9. run again, files copied up. run again to check. same issue.

Okay, so I did a scrub of the community post and there is not a working answer for this as far as I can tell. Setting fruit:encoding = native in SMB results in shares that can be seen in Finder but disappear when you try to mount them.

Here is an example of the file/folder names:

deleting Wolfgang Amadeus Mozart/Cosi Fan Tutte (Concerto Köln, René Jacobs)/
deleting Wolfgang Amadeus Mozart/Don Giovanni (Ghiaurov, Janowitz, Burrows, Zylis‐Gara, Evans, Herbert Von Karajan)/51 Don Giovanni - Atto Secondo - Ah Dove È Il Perfido.m4a

rsyncd is not removed..... it has been moved to apps

Rsync Daemon

Installation and basic usage instructions for the Rsync Daemon application.

www.truenas.com