TrueNAS Shadow Copy works intermittently

[Frikkie]

Hi happy forum people,

I recently upgraded from 11.3-U4 to TrueNAS 12.0 and have been very happy with the stability except for the intermittent working of shadow copies, which previously worked without a hitch.


Setup:
SMB Share with "Default share parameters" for dataset "Vinnig/Clarence" → shadow copy enabled
Periodic snapshot setup, creation of one, SMB service restart.

Result:
Most of the time the previous versions tab in windows explorer only populates when I connect to the share via the IP address and NOT the hostname.
Sometimes after an SMB service restart or the removal and subsequent re-adding of "enable shadow copy" the snapshots would appear for a time in the "hostname" previous versions tab but then disappear shortly thereafter.

If I make the .zfs snapshot directory visible, I can navigate to and use it properly via the "hostname".
Does anyone else find this super weird?!?

Side note:
This morning I saw two errors on the TrueNAS console but I am not sure if it relates at all.
"[...] freenas.home smbd 82989 ../../source3/modules/vfs_ixnas.c:2301(ixnas_ntimes)"
"[...] freenas.home smbd 82989 ixnas_ntimes: utimesat failed: No such file or directory"


Thanks in advance for any help. :)

 

[kiriak]

I also noted this.
Some times previous versions say there is nothing, after some time without doing anything, some but not all the previous versions are shown up.

I just have an old PC setup as a play and learn TrueNAS system (new install).
I haven't notice this behaviour in my previous 11.x test system.

 

[Frikkie]

I also noted this.
Some times previous versions say there is nothing, after some time without doing anything, some but not all the previous versions are shown up.

I just have an old PC setup as a play and learn TrueNAS system (new install).
I haven't notice this behaviour in my previous 11.x test system.

Thanks for the response!
Very spooky stuff indeed seeing as a different dataset has the previous versions accessible via hostname.
Oh well, doesn't really bother me an insane amount seeing as I still have access to it via the IP and reverting back to 11.3 is not an option for me.

 

[bmh.01]

It's broken, i've put a bug report in but i'm not sure how seriously it's be prioritised, I can't believe that TN12 was released with such a fundamental feature broken. And tbh I get the feeling it's going to remain broken for U1 as there seems to be a blind push to release it.

https://jira.ixsystems.com/browse/NAS-107722

 

[Frikkie]

That is indeed good to know. Thank you for clearing that up.

One of my users also "breaks" after a while and shows up as the full SID (unknown account) under security in an SMB share accessed from a Windows Client.
Maybe I should create a bug report for that before I start thinking there is something wrong with my configuration and completely create a new user and have to strip the ACLs and re-add them.

 

[bmh.01]

I've had that as well, although it was groups for me as it'd lost the group mapping somehow (net groupmap list will show it, it applies to users as well).

Removing the samba authentication option from the user/group then adding it back re-creates the mapping and 'fixes' it, I used powershell to loop through the groups via the API to do it as i've got a fair amount of users/groups. Restarting samba after that to clear the caches and it's been working OK since.

Disappointing, but it seems the old freenas upgrade adage still applies.

 

[Frikkie]

Legend! User is displayed correctly again.

I'll be following your bug report intensely... hopefully the support team does the same.

 

[bmh.01]

https://jira.ixsystems.com/browse/NAS-107567

I'm not super confident at this point, that's the issue that's been linked as the fix but it doesn't fix it. I've highlighted this but (https://www.ixsystems.com/blog/truenas-12-u1-is-scheduled/) makes me think it's going to be swept under the carpet for this release.

 

[bmh.01]

And right on cue, pushed back to U2 .

 

[Frikkie]

How they can give this Samba previous versions bug a "low" priority, let alone push it back to U2, makes me wonder if there are actually any tangible benefits for the end-user due to the newly merged FreeNAS and TrueNAS release cycles........
Didn't they "redefine" the release term U1 to apply to "mission critical" use cases? LOL. It is no where near production ready.
I am running this at home and have backups so if it just dies, no one will really care. I do however feel an incredible amount of sympathy for those who are running this OS in a business environment and decided to upgrade.

 

[seanm]

Frikkie, as you can see in the ticket, the code change seems to be done already. But thoroughly testing software also takes time. If every bug held up a release, then no releases would ever get out. It might be better to get U1 out sooner, to get other fixes in people's hands. That doesn't mean U2 can't be out soon after.

PS: most in business environments probably haven't upgraded, especially since U1 was explicitly stated as the 'mission critical' release, so they won't jump before that.

 

[ude6]

Hi. Got the same problem. In 12 SMB support is really buggy. I also have a major isse with user names and share names beeing identical. :-(

 

[bmh.01]

Frikkie, as you can see in the ticket, the code change seems to be done already. But thoroughly testing software also takes time. If every bug held up a release, then no releases would ever get out. It might be better to get U1 out sooner, to get other fixes in people's hands. That doesn't mean U2 can't be out soon after.

PS: most in business environments probably haven't upgraded, especially since U1 was explicitly stated as the 'mission critical' release, so they won't jump before that.

The code change doesn't fix this bug, I've already tested it and it has the same issue. The changes that have been made are around efficiency only, when there are some changes reference libzfs or similar I'll give it a try again.

 

[anodos]

The code change doesn't fix this bug, I've already tested it and it has the same issue. The changes that have been made are around efficiency only, when there are some changes reference libzfs or similar I'll give it a try again.

It depends on the particular changeset you're looking at. One of them changes the value being looked at from USED to also evaluate WRITTEN zfs property. There is an additional changeset that fixes a bug in evaluation of samba's connectpath / cwd when dealing with shadow copies. This is more likely the issue affecting some users. Samba's VFS in 4.12 / 4.13 was _significantly_ changed due to the continued modernization efforts of the core team members.

When you're dealing with shadow copies, you have to dynamically change the share's connectpath and user cwd in ways that can be complex because users may have shares that are directories inside ZFS datasets, there may be multiple nested ZFS datasets with different snapshots, etc, etc. So between the general complexity of doing shadow copies and the large-scale VFS changes upstream, we had a bug creep in that took a little time to track down. It's merged into 12-stable (for U1 release) but not master (nightlies) yet.

 

[bmh.01]

I've only seen the one around the USED change so apologies if there are more commits that I've not seen, that sounds more along the lines to fixing it as from what I could see it wasn't returning the snapshots in the first place and there were several times where it was looking in the wrong dataset as well.

I'll look forward to being able to test that out when it makes it into the nightlies. It feels like U1 is set to a hard deadline and that it's going to be pushed back so it would be fantastic if it would make it into U1.

 

[anodos]

I've only seen the one around the USED change so apologies if there are more commits that I've not seen, that sounds more along the lines to fixing it as from what I could see it wasn't returning the snapshots in the first place and there were several times where it was looking in the wrong dataset as well.

I'll look forward to being able to test that out when it makes it into the nightlies. It feels like U1 is set to a hard deadline and that it's going to be pushed back so it would be fantastic if it would make it into U1.

We made a hard push to get shadow-copy fixes into U1. Please let me know if you continue to have issues there.

 

[bmh.01]

We made a hard push to get shadow-copy fixes into U1. Please let me know if you continue to have issues there.

Was going to post here last night to say that it looked from GH like you'd managed to get it in, (I personally at least) really appreicate the effort made there.

I've posted on the bug tracker but from what I've tried it looks like it's 99% sorted at the very least, I think (need to test some more) empty snapshots are still getting ignored with ignore_empty_snaps set to false for the few shares that i've got it enabled on but other that it looks good so far.

 

[anodos]

Was going to post here last night to say that it looked from GH like you'd managed to get it in, (I personally at least) really appreicate the effort made there.

I've posted on the bug tracker but from what I've tried it looks like it's 99% sorted at the very least, I think (need to test some more) empty snapshots are still getting ignored with ignore_empty_snaps set to false for the few shares that i've got it enabled on but other that it looks good so far.

There's still one more PR for an issue I discovered after U1, so it will have to wait till next release. By default there are actually two criteria evaluated for whether to show a particular snapshot for a file.
1) whether `written` > 0
2) whether mtime for file differs from previous snap (if it's a file, not a dir). Goal is to limit to relevant snapshots because of protocol limitations.

 

[Scharbag]

Accidentally started another thread on this - but I can confirm that shadow copies are not working as expected on 12.0-U1 from a Windows 7 client.

Cheers,

 

[bmh.01]

There's still one more PR for an issue I discovered after U1, so it will have to wait till next release. By default there are actually two criteria evaluated for whether to show a particular snapshot for a file.
1) whether `written` > 0
2) whether mtime for file differs from previous snap (if it's a file, not a dir). Goal is to limit to relevant snapshots because of protocol limitations.

I guessed the push to remove the empties was to remove the overhead and it makes sense, as there's no point in returning it if there is nothing of value in the snapshot.

I've done some more testing and I agree with Scharbag it still doesn't work propetly it's not directly related to ignore empties. It returns more snapshots than 12-RELEASE did but it's still seemingly random which ones it does return. I've got a shares with ignore empties set to both true and false and they both exhibit the same behaviour where there are multiple snaps with USED=0 and it returns some empties but not others. I can't see a pattern to it.

On the other had I think it's returning all the ones with USED > 0 for me.