'Modified' timestamp issue on 12-U6.1 on SMB share

[hrana]

On 12.0-U6.1, using Syncovery, FreeFileSync, etc. has become challenging on SMB shares. The 'modified' timestamp defaults to whenever the file is last touched on the SMB share. In my use case, any time a sync event occurs, the modified timestamp gets updated to the date of the event.

Whereas in earlier versions of TrueNAS 12.0, these applications could set those timestamps to facilitate file synchronization. Without this capability, the synchronization algorithm flags the file on the SMD share as newer than the original file and copies it over. This perpetuates the problem.

Files created/copied prior to the upgrade are unaffected until they are modified on the source system. After that point, they are locked in a perpetual update cycle on the SMB share due to the above behavior.

It appears this bug was solved a while ago but has resurfaced in the latest version: https://jira.ixsystems.com/browse/NAS-107005

I have checked the release notes for U6 and U6.1 but don't see anything about a change in behavior. Did I miss something or is this a regression?

 

[KarstenL680]

Same issue at my side, makes a lot of trouble keeping systms in sync so hopefully it is fixed soon.

 

[jsylvia007]

Dear god I thought I was going crazy... I'm getting ready to go on a trip and have been having some issues with Syncthing, so I decided to start from scratch. Everything went perfectly fine, until I started to see a "Local Additions" issue. It appears that this is related to the fact that there is something wrong with the "last modified" date not sticking when the files are copied.

I noticed that nobody has created a bug for this with the new version, so I did.

https://jira.ixsystems.com/browse/NAS-113513

 

[MobiusOne]

Same here! I wasn't sure if it was the update or other tinkering that I've done, but this confirms it for me. In the process of reinstalling the old version (that I conveniently didn't note which one it was, so I'm going off the save date of the last config file I had saved).

 

[hrana]

Has U7 fixed this bug for any of you?

 

[jsylvia007]

Has U7 fixed this bug for any of you?

I've been gunshy with doing the update because of the other issues with Chrome and the dashboard.

 

[PhoenixOne]

Has U7 fixed this bug for any of you?

Still the same problem for me.
Reverted to U6, which works ok.

 

[hrana]

U7 doesn't work for me either. Reverting to U6 appears to be the fix for now.

 

[anodos]

Issue won't be fixed until I have data requested in the ticket. I need a solid simple way to reproduce issue along with packet capture of session where this happens. I have been unable to reproduce using several tools.

 

[jsylvia007]

Issue won't be fixed until I have data requested in the ticket. I need a solid simple way to reproduce issue along with packet capture of session where this happens. I have been unable to reproduce using several tools.

PCAP Uploaded. Sorry about that. Just now got the time to perform the capture.

 

[ixis]

Dear god I thought I was going crazy... I'm getting ready to go on a trip and have been having some issues with Syncthing, so I decided to start from scratch. Everything went perfectly fine, until I started to see a "Local Additions" issue. It appears that this is related to the fact that there is something wrong with the "last modified" date not sticking when the files are copied.

I noticed that nobody has created a bug for this with the new version, so I did.

https://jira.ixsystems.com/browse/NAS-113513

Thank you so much for opening the ticket!

My robocopy backups were taking ages (up to 20-30 minutes, instead of normal 1-2 min) and it was getting progressively worse since middle of November. I had no idea what was happening. I noticed even increased activity in December, when I upgraded to TrueNAS 12.0 U7, so I assumed that was an issue introduced at that time, but it is clear now that it was happening since upgrade to U6.1.

I made copy tests in latest Windows 10 with normal copy (drag&drop), robocopy and powershell copy in TrueNAS 12.0 U6, U6.1 and U7 and only U6 preserves the modification date correctly. Both U6.1 and U7 puts the current date and time as modification date - that causes backups to re-scan all files uploaded since the upgrade and as files are added the process takes more and more time. After reverting to U6 and re-running robocopy with /TIMFIX switch all is fine now.

I hope the developers can find a solution for that regression. If you need some more debug info, let me know. Love the TrueNAS, keep up the good work!

 

[ZFS-2012]

Hey gang!

Figured I would chime in and share some things I figured out today, after going down some deep rabbit holes related to all things timestamp related, with TrueNAS, FreeBSD, and Samba.

TL:DR version:

I found reverting to stock Samba support for ZFS ACLs fixes timestamp issues, add these to aux parameters for your shares:

vfs objects = zfsacl zfs_space streams_xattr shadow_copy_zfs aio_fbsd
nfs4:chown = no

Not sure if this is required, but I bounced the SMB service after making these changes (full reboot not required). After which ALL timestamps worked the way I have been accustomed to. Which for me is almost a decade now on ZFS under OpenIndiana (Solarish).

* NOTE! It appears one of the motivations behind creating a custom "ixnas" to replace "zfsacl" was to add some additional features, such as user quotas with AD integration. I am not using these. BUT, if you are, reverting to "zfsacl" likely breaks this. Based this comment on:

12. Sharing — FreeNAS®11.3-U1 User Guide Table of Contents

www.ixsystems.com

Verbose version:

From what I gather "ixnas" only just came out of the "experimental" stage (during 11.3 U1 ish?) maybe like a year ago. Might not yet be fully ready for prime time? In my case, was chasing after show stopper issues with creation timestamps. But would also seemingly apply to modify timestamps.

While going down various rabbit holes and reading various forums, I can across the following which I consider relevant:

Default VFS objects for a Windows SMB shares have changed/are different in 11.3-U1

To avoid hijacking another thread where this question was originally asked, but has not been answered at this time (Original question in "question on ACL for SMB share" thread), I am starting a new thread to see if someone can answer this question reliably: FreeNAS 11.3 was released and I’m a...

www.truenas.com

12. Sharing — FreeNAS®11.3-U1 User Guide Table of Contents

www.ixsystems.com

Reddit - Dive into anything

www.reddit.com

https://jira.ixsystems.com/browse/NAS-102985

I guess some "supporting arguments" for ixnas being relatively new and perhaps not fully ready for prime time. This forum thread being among that, and actually one of the earlier discussions I cam across. Which is why I came back to share what I have learned thus far. That last link either shows this issue is a regression, or was never actually fully fixed? The bug ID just kind of trails off to no where, with out confirmed resolution.

I am also concerned with the fact that official Samba documentation suggests using "nfs4:chown" is a bad idea:

vfs_zfsacl

One would assume the iX team took care to fully test and vet the use of "nfs4:chown" in this use case, but given the other bugs I keep running into, not sure I personally am willing to make that assumption.

Since I am a "newb" to these forums, if someone with more clout wants to take it from here? I would suggest some of this could be shared with the dev team in Jira for bug tracking.

Hope that helps some of you. Honestly not sure if I am going to stick around long enough to see where this goes? Really only signed up here so I could share this info. Because I spent hours chasing after this, in an attempt to determine how many "show stoppers" remain on my list when it comes to doing an eval of TrueNAS 12 as an option for OS on my new ZFS server build. Of which this was one, which I seem to have found an acceptable work around for. I rarely post in forums, but figure in this case I should try and give back.

No offense to the iX team, I love me some FreeNAS / TrueNAS! Have been a fan for a loooong time (like 2009?). Even run it at our office for some light weight storage needs. But I am struggling to reach an acceptable comfort level with TrueNAS 12, as a potential replacement to a decade old OpenIndiana + Napp-It build I am running at home. To be fair, my personal needs are far more demanding than most home NAS users (more demanding than some small biz too).

Since this post is not the appropriate place to elaborate on that, might stick a comment as to the back story in my profile.

Take care all, and Happy New Year!

 

[ZFS-2012]

Update: IGNORE my previous post.

I had rolled back to 12.0 U5-1 based on comments about this issue, and several others. So my previous testing with creation timestamp issues and "ixnas" were under that version.

After upgrading to 12.0 U7, I am seeing the same modify timestamp issue reported here. Even with using "zfsacl" instead of "ixnas".

So that tells us two things:

- This modify timestamp issue has nothing to do with the creation timestamp issue, and is not related to the use of "ixnas"
- The creation timestamp issue in "ixnas" exists in both 12.0 U5-1 and 12.0 U7. (I had originally encountered this on U7 and rolled back to U5-1 falsely believing it related to the modify timestamp issue that exists in U7).

I'm sorry, but my previous advice is of no help. Other than to highlight that there are in fact TWO timestamp bugs as of 12.0 U7.

Again, if those with more clout care to comment to devs. List of show stoppers is getting too long, think I might be switching gears here.

Best wishes for 2022!

 

[retznas]

Good evening, I'm having the same issue with modified times on SMB shares. The timestamp on the truenas side is set to the last modified time, not the original modified timestamp.

I recently upgraded to 12u7 (from 11), where I've been successfully running robocopy nightly for years, so this isn't a robocopy issue.

There seems to be a cut-off for file size as small files, like txt files are getting the correct date, but larger files (anything over 5GB) get the wrong modified timestamp.

 

[PhoenixOne]

Just to add some more mystery to the issue... or it might even help someone figure out a solution.

I have 2 pools on my main TrueNAS rig, and the issue is happening only on one of the pools, my main/older one.
Since I have around 10TB of data on it, it's a bit difficult to recreate it to see if it fixes the issue. But I have all the data already backed up, and may go this route if it's a viable solution.

The other pool is for temp storage with random disks I have. I end up recreating it every now and then. It is not showing the issue in U6.1/U7.

To be sure, I just now re-checked both the pools and there is no "Update pool" option available, so I'd say I'm on the latest "version"/features..?

Any thoughts?

 

[anodos]

If anyone who is able to reproduce this issue at will is willing to do a teamviewer session to allow me to investigate further, please send me a private message and we can arrange a time. I have made a similar request in the jira ticket with no response.

 

[jsylvia007]

If anyone who is able to reproduce this issue at will is willing to do a teamviewer session to allow me to investigate further, please send me a private message and we can arrange a time. I have made a similar request in the jira ticket with no response.

Sent you a direct message.

 

[retznas]

Hi again, just wanted to share an update. My new Truenas 12u7 with a new pool does not exhibit this behaviour, it was only on an pool created in 11. I'm assuming this has been fixed now (I can see the commit in the bug link above) but this could be helpful for anyone else that has this problem.

I appreciate the activity around this, glad to know the community is there.

 

[anodos]

Thanks to community help I was able to see issue in U6.1 / U7 (edge case where file created and client writes mtime that is older than birthtime). We were dropping the request to avoid unexpectedly setting the file's birthtime. I dropped this protection to ease compatibility concerns. c.f. https://github.com/truenas/ports/commit/abbc24624922243bffa85dcc89da921c81e760eb

I also added workaround in utimensat(2) system call (special AT_ flag) for TrueNAS 13 that will allow us to explicitly specify birthtime, atime, and mtime. c.f. https://github.com/truenas/os/commit/0fb42105d901c5f2f0d50df608851fa3583f4a5c

 

[FindingFilene]

Thank you for your work on this, @anodos . I hope I can find this in a release on TrueNAS nightlies or something. Do you know any workaround that can be done by GUI until this is part of our official release?