Windows Robocopy with FreNASS incorrectly reports files as modified

[Zaaphod]

I have a windows CIFS share on FreeNAS 9.3 where I have all my work related files, Occasionally I will need to go out of town, so I need to bring a backup copy of this share with my on my Windows 10 laptop, then when I return, I need to restore my changes back to FreeNAS. Being a windows computer, I am using robocopy for these backup and restore functions, then problem I am having is that whenever I either backup or restore to a CIFS windows share on FreeNAS, it reports every single file as 'modified' even if it has not been modified. This completely nullifies the reason I was using robocopy in the first place, which is to only transfer the changes, thus saving me a LOT of time and writes on my laptop's SSD.

I cannot figure out why this is happening, I am copying the timestamps, as well as file attributes, and ignoring security. I have removed the attribute bit from all files as I have read that cause problems. Here is the exact robocopy command I am using:

Code:

robocopy J:\ D:\ /XD .recycle .Trash-1000 $RECYCLE.BIN /fft /MIR /V /NP /NDL /copy:DAT /dcopy:T /R:10 /W:5 /log:~Backup2Laptop.txt /tee

If I run this command and let it finish, then simply run it again, ALL files should be exactly the same, and it should do nothing, but it instead reports every single file as modified and actually copies every single file over again. This ONLY happens when J:\ is a FeeeNas network share. If it is a removable drive for example, then it correctly skips files that have not changed. I have looked at the files on FreeNas and on my laptop drive and I simply cannot see what is triggering it to think that it is modified. I recently added the /fft argument in case there was some microscopic difference in the timestamps, but it did not help. using /XO is NOT an option as it will most certainly corrupt the entire process. I have a LOT of GIT repositories on this drive, and if I am on one branch when I backup, but happen to be on an older branch when I restore, then the entire repo will be corrupted... I need to the destination to match the source in all ways when I am done, but I do not want to re-copy identical files

Does anyone know what might be happening and how to get it to work? Is this just something that can't be done with robocopy and FreeNas, and if so, how can I accomplish this with a Windows 10 Laptop?

Any help would be greatly appreciated!

 

[anodos]

Post count contents of /usr/local/etc/smb4.conf

Make sure atime is disabled for the dataset you're sharing.

 

[Zaaphod]

Below is smb4.conf how do I disable atime?

Code:

[global]  
  username map = /usr/local/etc/smbusers  
  server max protocol = SMB3_00  
  interfaces = 127.0.0.1 192.168.0.222  
  bind interfaces only = yes  
  encrypt passwords = yes  
  dns proxy = no  
  strict locking = no  
  oplocks = yes  
  deadtime = 15  
  max log size = 51200  
  max open files = 1883432  
  load printers = no  
  printing = bsd  
  printcap name = /dev/null  
  disable spoolss = yes  
  getwd cache = yes  
  guest account = games  
  map to guest = Bad User  
  obey pam restrictions = yes  
  directory name cache size = 0  
  kernel change notify = no  
  panic action = /usr/local/libexec/samba/samba-backtrace  
  nsupdate command = /usr/local/bin/samba-nsupdate -g  
  server string = FreeNAS Server  
  ea support = yes  
  store dos attributes = yes  
  lm announce = yes  
  hostname lookups = yes  
  unix extensions = no  
  null passwords = yes  
  acl allow execute always = true  
  acl check permissions = true  
  dos filemode = yes  
  multicast dns register = no  
  domain logons = no  
  local master = no  
  idmap config *: backend = tdb  
  idmap config *: range = 90000001-100000000  
  server role = standalone  
  netbios name = FREENAS  
  workgroup = WORKGROUP  
  security = user  
  pid directory = /var/run/samba  
  create mask = 0666  
  directory mask = 0777  
  client ntlmv2 auth = yes  
  dos charset = CP437  
  unix charset = UTF-8  
  log level = 1  
  
  
[8]  
  path = /mnt/Vol_8TB/8  
  printable = no  
  veto files = /.snapshot/.windows/.mac/.zfs/  
  writeable = yes  
  browseable = yes  
  vfs objects = zfs_space zfsacl aio_pthread streams_xattr  
  hide dot files = yes  
  guest ok = no  
  nfs4:mode = special  
  nfs4:acedup = merge  
  nfs4:chown = true  
  zfsacl:acesort = dontcare  
  
[8TB]  
  path = /mnt/Vol_8TB  
  printable = no  
  veto files = /.snapshot/.windows/.mac/.zfs/  
  writeable = yes  
  browseable = yes  
  vfs objects = zfs_space zfsacl aio_pthread streams_xattr  
  hide dot files = yes  
  guest ok = yes  
  nfs4:mode = special  
  nfs4:acedup = merge  
  nfs4:chown = true  
  zfsacl:acesort = dontcare  
  
[Audio]  
  path = /mnt/Vol_8TB/Audio  
  printable = no  
  veto files = /.snapshot/.windows/.mac/.zfs/  
  writeable = yes  
  browseable = yes  
  vfs objects = zfs_space zfsacl aio_pthread streams_xattr  
  hide dot files = yes  
  guest ok = yes  
  nfs4:mode = special  
  nfs4:acedup = merge  
  nfs4:chown = true  
  zfsacl:acesort = dontcare  
  
[docsjames]  
  path = /mnt/Vol_8TB/Documents/James2nd  
  printable = no  
  veto files = /.snapshot/.windows/.mac/.zfs/  
  writeable = yes  
  browseable = yes  
  recycle:repository = .recycle/%U  
  recycle:keeptree = yes  
  recycle:versions = yes  
  recycle:touch = yes  
  recycle:directory_mode = 0777  
  recycle:subdir_mode = 0700  
  vfs objects = zfs_space zfsacl aio_pthread streams_xattr recycle  
  hide dot files = no  
  guest ok = yes  
  nfs4:mode = special  
  nfs4:acedup = merge  
  nfs4:chown = true  
  zfsacl:acesort = dontcare

 

[dlavigne]

Edit the dataset in Storage -> Volumes -> name_of_dataset.

 

[anodos]

Here's the powershell script I use on my W10 client to perform a quick and dirty backup of a few shares:

Code:

# Volume Name of External Hard Drive
$VolumeName = 'OFFSITE2'

# Make sure you have enough space on the Hard Drive for the shares you're backing up
# This is an array. Separate shares by commas and enclose each share name with quotation marks
$SharesToBackup = "Share 1","Share 2","Share 3"
$ServerName = 'storage'

# Make sure you put in a valid log path
$LogPath = "\\storage\logs\OFFSITE2"

# Messy way of getting the volume's drive letter. I should make this a one-liner.
$disk = Get-WmiObject -query "SELECT * from win32_logicaldisk where VolumeName = '$VolumeName'"
$DriveLetter = $disk.DeviceID

# Loop through robocopy command for each share in the array
foreach ($ShareName in $SharesToBackup){
	$SourcePath = "\\$ServerName\$ShareName"
	$LogFile = "$LogPath\$ServerName.$ShareName.$(get-date -f yyyy-MM-dd).log"


	# Test if disk is actually connected.
	if ($disk.DeviceID -eq $null){
		msg * "Backup task for $ShareName not run - $VolumeName is not connected to the computer"
		break
	}

	else {
		robocopy "$SourcePath" $DriveLetter\"$ShareName" /MIR /COPY:DT /Z /W:5 /R:15 /FFT /XF ntuser.* *.dat *.db *.tmp /LOG:$LogFile
	}

}
msg * "Backup task complete"

As you can see, the primary difference between the two robocopy commands is /dcopy:T. If killing atime doesn't resolve the issue, perhaps try removing this robocopy switch. File timestamps will be preserved (by /copy:DT or DAT). There was also some reason why I restricted /COPY to "DT". but I can't remember what it was.

 

[darkwarrior]

Hi,

per the robocopy command line reference, the 'A' in 'DAT' stands for the file attributes (aka hidden, archived and such)
http://ss64.com/nt/robocopy.html

By the way Zaaphod since you're using a Windows 10 laptop and a CIFS share, did you thought about using offline files ?
The offline filesync could take care of all the hassle automatically for you (if you schedule it)

cheers,
DW

Edit: typo

 

[anodos]

Hi,

per the robocopy command line reference, the 'A' in 'DAT' stands for the file attributes (aka hidden, archived and such)
http://ss64.com/nt/robocopy.html

By the way Zaaphod since you're using a Windows 10 laptop and a CIFS share, did you thought about using offline files ?
The offline filesync could take care of all the hassle automatically for you (if you schedule it)

cheers,
DW

Edit: typo

I've found 'offline files' to unreliably handle samba shares. Things appear to work fine for a few days / weeks. Then synchronization mysteriously fails leaving the share in an inconsistent state. It's better to use robocopy or a 3rd party tool. I've recently tested 'syncback free' and it seemed quite nice.

 

[Zaaphod]

Thank you to everyone very much for helping with this. I disabled atime and at first I didn't think it worked because I was still seeing 'modified' files that I knew were not, but then I noticed that even though it was reporting 'modified' it was not copying the file. I made a new backup on another drive, and now everything is working properly and unchanged files are being reported as 'same' I need /copy: DAT because without the A it does not copy the attributes, and then when I try to update the backup, it says all the files are 'tweaked' I also have the hidden .git folders that I wish to remain hidden.

Thanks for the recommendations on 'offline files' and 'syncback free' At the moment robocopy is working well, and I like controlling what direction my backup is going.

The only issue I'm having now is that /dcopy:T which is suppose to preserve the timestaps on directories is not working, but that's not really all that important, and I think it's an issue with the way that Robocopy works.

 

[anodos]

Thank you to everyone very much for helping with this. I disabled atime and at first I didn't think it worked because I was still seeing 'modified' files that I knew were not, but then I noticed that even though it was reporting 'modified' it was not copying the file. I made a new backup on another drive, and now everything is working properly and unchanged files are being reported as 'same' I need /copy:DAT because without the A it does not copy the attributes, and then when I try to update the backup, it says all the files are 'tweaked' I also have the hidden .git folders that I wish to remain hidden.

Thanks for the recommendations on 'offline files' and 'syncback free' At the moment robocopy is working well, and I like controlling what direction my backup is going.

The only issue I'm having now is that /dcopy:T which is suppose to preserve the timestaps on directories is not working, but that's not really all that important, and I think it's an issue with the way that Robocopy works.

@dlavigne - this appears to be a rather odd behavior caused by atime being turned on. What do you think? Should the SMB share wizard automatically turn off atime? Is this worth documenting?