After snapshot replication, the source and destination dataset contents are different.

[SwisherSweet]

Hi folks,

I am making room on my TrueNAS Core server, so I replicated a "backup" dataset to a new TrueNAS Scale machine I setup. The replication task completed without error. The dataset is one level deep (no child datasets).

However, before I deleted the old dataset, I did a sanity check and compared the the source with the destination with tools and manual checks. I found that they are indeed different. The differences are because of new or changed data as no one is writing to the original share. Some files and folder have been renamed to short names in the destination. There are also missing files in the destination.

For example, here's a few different files on the source:

Here are the same files with different names on the destination:

Perhaps these were renamed to a shorter name because the original names were too long, but why?


Another strange example is I have a PDF doc with a relatively short name already on the source:

And when I find it on my target NAS, it's there:

But when browse the containing folder using macOS over SMB on the source (TrueNAS Core), it looks like this (note the / between SE and 30):

And this is what it looks like on macOS over SMB on the destination (TrueNAS Scale):

In my final example, there are actually files missing:

Source:

Destination:

So it appears I have a least two problems:

1. Replicated snapshots are not replicating all files.
2. Metadata on destination NAS is not consistent with source NAS.

Anyone have ideas on how I could correct these behaviors?

Thank you.

 

[Patrick M. Hausen]

Login to both systems via SSH and examine the filenames from the Unix command line. I suspect something with the share configuration at the destination, because these look like MS-DOS file name mangling.

 

[winnielinnie]

To build on what @Patrick M. Hausen noted (about the filenames), the "missing" files are likely due to beginning with a "dot" (i.e, a "period".)

In order to truly compare the source and destination snapshots, you should be using the same tools via the same protocols; or preferably, via SSH with the same shell.

Snapshots are immutable. There's no way that the same snapshots, with the same GUID, would have differences in filenames, content, permissions, etc.

If the replication succeeded, then it means mypool/data@snap001 is exactly the same as newpool/data@snap001

 

[SwisherSweet]

Login to both systems via SSH and examine the filenames from the Unix command line. I suspect something with the share configuration at the destination, because these look like MS-DOS file name mangling.

Here's the console output (don't have SSH setup on these servers, but I assume console is just as good) of the first example:

Source:

Code:

root@trooper[...len Home Folder/Desktop/Clean Desktop]# find . -name "*Forums.pd
f"
./Re- MI424WR-GEN2 Rev E Configuration Thread - Verizon FiOS | DSLReports Forums.pdf
./How-to- Make Actiontec MI424WR Revision...ork - Verizon FiOS | DSLReports Forums.pdf

Destination:

Code:

admin@overkill[...len Home Folder/Desktop/Clean Desktop]$ find . -name "*Forums.pdf"
./Re- MI424WR-GEN2 Rev E Configuration Thread - Verizon FiOS | DSLReports Forums.pdf
./How-to- Make Actiontec MI424WR Revision...ork - Verizon FiOS | DSLReports Forums.pdf

Even if the share configuration is jacked up in some way, why are some files just missing in the destination?

 

[winnielinnie]

Even if the share configuration is jacked up in some way, why are some files just missing in the destination?

See my post above. Nothing is actually "missing".

In the directory with the images, while on the backup server:

Code:

stat ._IMG_0046.MOV

 

[SwisherSweet]

See my post above. Nothing is actually "missing".

Thank you. I used snapshot replication because my understanding is the same is yours, that there should be 0 difference between the source and destination.

When logging into the consoles on the respective TrueNAS, there is in fact differences when using the "ls" command (see my final example in the original post).

Is the console not as reliable as using SSH?

the "missing" files are likely due to beginning with a "dot" (i.e, a "period".)

Do you suspect that the console is hiding files that start with a dot?

 

[winnielinnie]

In order to truly compare the source and destination snapshots, you should be using the same tools via the same protocols; or preferably, via SSH with the same shell.

"Shell" in this case means sh, csh, bash, zsh, fish, etc.

SCALE and Core use different default shells. Some shells have "modern" ways to deal with directory listings, and will hide "dot" files without intervention.

So, in your case on SCALE, invoke the "-a" flag with the "ls" command to view a listing of everything.

Sidenote: Cropping images and hiding the original command makes it harder for people to help you. I understand if you want to hide personal information. Just keep in mind we only know what you provide in the post.

 

[SwisherSweet]

"Shell" in this case means sh, csh, bash, zsh, fish, etc.

SCALE and Core use different default shells. Some shells have "modern" ways to deal with directory listings, and will hide "dot" files without intervention.

So, in your case on SCALE, invoke the "-a" flag with the "ls" command to view a listing of everything.

Sidenote: Cropping images and hiding the original command makes it harder for people to help you. I understand if you want to hide personal information. Just keep in mind we only know what you provide in the post.

@winnielinnie thank you. Using the -a switch on ls allowed me to see the files that I not could see before.

Anyone have any thoughts on why the files and folder names are getting mangled when browsing them with macOS over SMB?

As a test, I copied a subset of these the files I'm having problems with using Finder on macOS over SMB from Core to Scale and I don't have the same problems. That is, all the file name and folder names are consistent.

Thanks again!

 

[Patrick M. Hausen]

Please post the output of testparm -s on both systems. There must be a crucial difference.

 

[SwisherSweet]

Please post the output of testparm -s on both systems. There must be a crucial difference.

Here's the output of the source (Core):

Code:

Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@trooper[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
        aio max threads = 2
        bind interfaces only = Yes
        disable spoolss = Yes
        dns proxy = No
        enable web service discovery = Yes
        kernel change notify = No
        load printers = No
        logging = file
        max log size = 5120
        nsupdate command = /usr/local/bin/samba-nsupdate -g
        registry shares = Yes
        restrict anonymous = 2
        server multi channel support = No
        server role = standalone server
        server string = Trooper TrueNAS Server
        unix extensions = No
        workgroup = CYBERTRON
        idmap config *: range = 90000001-100000000
        fruit:nfs_aces = No
        rpc_server:mdssvc = disabled
        rpc_daemon:mdssd = disabled
        idmap config * : backend = tdb
        directory name cache size = 0
        dos filemode = Yes


[backup]
        comment = General Backup
        ea support = No
        level2 oplocks = No
        mangled names = no
        oplocks = No
        path = /mnt/tank/backup
        read only = No
        smbd max xattr size = 2097152
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas zfs_core aio_fbsd
        streams_xattr:store_stream_type = no
        streams_xattr:prefix = user.
        fruit:locking = netatalk
        fruit:resource = file
        fruit:metadata = netatalk
        fruit:encoding = native
        nfs4:chown = true

Note, I removed details of the other shares for brevity and privacy. The [backup] share is the one I'm sourcing from.

Here's the output of the target (Scale):

Code:

Linux overkill 6.1.74-production+truenas #2 SMP PREEMPT_DYNAMIC Wed Feb 21 20:30:38 UTC 2024 x86_64

        TrueNAS (c) 2009-2024, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com

Welcome to TrueNAS
Last login: Fri Mar 15 10:52:38 CDT 2024 on pts/27

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

admin@overkill[~]$ sudo testparm -s
[sudo] password for admin:
Load smb config files from /etc/smb4.conf
lpcfg_do_global_parameter: WARNING: The "syslog only" option is deprecated
Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

Server role: ROLE_STANDALONE

# Global parameters
[global]
        bind interfaces only = Yes
        disable spoolss = Yes
        dns proxy = No
        load printers = No
        logging = file
        max log size = 5120
        passdb backend = tdbsam:/var/run/samba-cache/private/passdb.tdb
        printcap name = /dev/null
        registry shares = Yes
        restrict anonymous = 2
        server multi channel support = No
        server string = Overkill TrueNAS Scale Server
        winbind request timeout = 2
        workgroup = CYBERTRON
        idmap config * : range = 90000001 - 100000000
        fruit:zero_file_id = false
        fruit:nfs_aces = false
        rpc_server:mdssvc = disabled
        rpc_daemon:mdssd = disabled
        idmap config * : backend = tdb
        create mask = 0775
        directory mask = 0775

[backup-from-trooper]
        ea support = No
        path = /mnt/rusty/backup-from-trooper
        posix locking = No
        read only = No
        smbd max xattr size = 2097152
        vfs objects = streams_xattr shadow_copy_zfs acl_xattr zfs_core io_uring
        tn:vuid =
        fruit:time machine = False
        tn:home = False
        tn:path_suffix =
        fruit:time machine max size = 0
        tn:purpose = DEFAULT_SHARE
admin@overkill[~]$

@Patrick M. Hausen I forgot that I needed to run the command on Scale via sudo, and once I did, it worked.

 

[SwisherSweet]

The mangled file and folder names mystery has been solved. The SMB share at the source (Core) uses "Use Apple-style Character Encoding" and the destination share (Scale) did not have this checked in the share configuration. After I enabled "Use Apple-style Character Encoding" in the share configuration on the destination server, all the strange folder and character names cleared up.

Since we mostly use Macs, with a few Windows machines mostly for development, which rarely access the servers, does it make sense to use this setting on new shares? Here's what the tool-tip help has to say about it:

Help: Use Apple-style Character Encoding
By default, Samba uses a hashing algorithm for NTFS illegal characters. Enabling this option translates NTFS illegal characters to the Unicode private range.

It would seem writing data to shares with this turned on is a one-way street. If you use it, you always have to use it.

Thanks again for everyone's help!

 

[winnielinnie]

I use it on all my shares, and I don't have any Mac computers. I think the "Apple-style" bit is a slight misnomer. It's quite useful for Windows clients too.

Do understand a caveat: Any filenames with "illegal" characters will appear different when viewed directly on the server as opposed to over SMB. If you only ever access them via SMB, it's no big deal.