SOLVED CIFS shares become read-only to Windows clients

[mudshark]

This is a very strange, ongoing issue: All my CIFS shares eventually become read-only to Windows clients.

I've built and rebuilt this FreeNAS 9.3 server several times now (4 or 5, at least) and without fail, after some time (weeks or more) the mapped shares become read-only to the Windows clients.

Up until that point I can read/write/execute/modify etc w/o any issues at all then suddenly and w/o warning ALL CIFS shares become read-only to Windows... Even shares I wasn't touching!!! One minute everything is fine, then suddenly I cannot edit or delete anything on any share.

Only once I was able to correct this symptom by editing and re-saving the CIFS service but not this time.
One time it looked like it occurred after taking a FreeNAS upgrade but now I assume that was a coincidence.
Rebooting the client and the server does not help.

Here are current error and Windows permissions settings:

The "Unknown Account" (S-1-5-21-4173839866-1125406939-101818755-501) and the UNIX wheel group are both set to "Full control" but the Windows Everyone account suddenly loses "Full control" + "Modify" rights. It seems that "Unknown Account" perhaps seizes control from the Windows EVERYONE group for *all* shares, even the ones I never read or wrote to.

I do not know where that user came from and I cannot find any way in the FreeNAS GUI to fix this.

The CIFS service setting all look good and the share mounts and CIFS share settings all seem OK too.

Guidance via this forum would obviously be helpful but I am trying to roll out a server build to others and w/o knowing how or why this is happening or how to fix it I cannot move forward on this so time is becoming a big issue.

PM me if you can help directly via a remote session to figure this out once and for all.

Thanks all,
RichieG

 

[SweetAndLow]

What version of freenas? And a debug file please.

 

[mudshark]

Oh! I had to u/d my sig to:
Build: FreeNAS-9.3-STABLE-201505130355
Platform: AMD A6-6400K APU with Radeon(tm) HD Graphics
Memory: 7342MB

And you'll have to "remind" me how to generate a debug file. ;)

 

[FlynnVT]

This sounds very like something I saw here recently. This thread fixed it:
https://forums.freenas.org/index.php?threads/cifs-permissions.28813/#post-199285

 

[Ericloewe]

Unix group?

Are you using Windows permissions?

 

[Ericloewe]

Memory: 7342MB

Somewhat short on RAM, we are. Shouldn't be the cause here, though. If you have 8GB installed, see if there's a BIOS option to dedicate less RAM to the GPU.

And you'll have to "remind" me how to generate a debug file. ;)

System -» Advanced -» Save Debug

 

[mudshark]

OK I have a debug file saved... Attached.

The UNIX wheel group is straight from FreeNAS. It does have the full control permissions so that is not the problem but that the Windows group "EVERYONE" has lost some rights to this new "unknown" is the issue I am assuming!

The commands below ( from https://forums.freenas.org/index.php?threads/cifs-permissions.28813/#post-199285 ) are said to perhaps fix this issue. These I assume are run from the FreeNAS GUI shell screen.
If so then how are they going to change mappings on the Windows machines though?
Anything else I should know before running these lines?
Perhaps I'd best wait to see if anything in the debug files stands out to one of you FreeNAS experts before stopping / restarting the samba service?

service samba_server stop
rm -rf /var/db/samba4/*
rm -rf /var/etc/private*
net groupmap cleanup # this should blow away all mappings
service ix-pre-samba start # create config (and group mapping) with known SID
service samba_server start

Thanks folks. So much!
R

 

[Philip Robar]

I've also been bitten by this several times while running FreeNAS 9.3. I reported this to the FreeNAS test alias as I was running nightlies, but I and iXsystems eventually chalked it up to user error and/or a byproduct of the nightlies. (They clearly thought that is was the former.) I no longer think that either is the case. I've been running 9.3 stable for several weeks now and suddenly, after destroying a pool and it's share and then recreating them, both of my shares, an existing and the new one, turned read only to all of my clients—both OS X and Windows. Restarting the SMB service didn't help. Restarting my system fixed it temporarily, but after a little while the shares went back to read only. And any new shares I create, with or without the Wizard, all are read only.

 

[cyberjock]

I just want to clarify a few points on CIFS permissions:

• FreeNAS is designed to work in a combination (albeit limited number ) of ways with regards to CIFS and permissions.
• There are ways to do it with windows ACLs, and ways to do it with Unix permissions.
• For people that are pros with permissions, there are ways to do a hybrid of the two and still be functional, albeit with some CLI-fu. The real limitation here is what the FreeNAS server admin can actually handle based on their experience and knowledge. There are also certain limitations to what you can actually do with a hybrid of the ACLs and Unix Permissions. For example, you cannot do Unix permissions with 3 users. You can only specify one user and one group (plus world). So you cannot do 3 users, but you could do a group that includes those users.
• There isn't any particularly great documentation on how iXsystems designs the permissions model that FreeNAS uses.

I personally think that having a guide that is a "here's how iXsystems designed the permissions to work" with some examples of how to implement certain scenarios would be extremely useful for many users, and much of the confusion that users experience would probably go away. The manual covers permissions, but only on the most basic level. But something with a deeper level of detail is necessary (IMO). That's what my white paper on FreeNAS permissions was/is supposed to solve. Unfortunately it's very low on the priority in relation to other things going on, and with some big changes in my personal life coming its something that is a little hard to budget into my free time. It's been on an indefinite hiatus and right now I can't give an ETA. I do project that this year it will be finished. Things that are outside of my control are improving my ability to do things like a permissions guide and others that I have partially done to some extent, but it takes time. Time isn't something I have lots of right now, but if things continue down the path I am projecting, I should have it done before the end of the year, and likely before Halloween. After I write it, it has to go through quite a few hands to be approved since I want to be sure that the info being disseminated is complete and correct.

So yes, I understand why iXsystems is so quick to dismiss permissions as a "user configuration issue". I personally find that virtually every time the issue comes down to one of two scenarios:

1. The user doesn't understand permissions. This can range from "cyberjock's mom" clueless to "just needing some info on how FreeNAS makes all this stuff relate to each other properly". In both cases I'd call it a user configuration error because the software is performing as designed and expected, the user just isn't giving good input data to FreeNAS to get the expected result.
2. The user understands permissions, but wants to do something that is impossible (like do 3 users to a single file with unix permissions example I gave above). It's not the best example because if the admin really knew permissions they should know they can't have 3 users with permissions to a single file with unix permissions. But again, I'd somewhat argue that this is again a user configuration issue because they don't understand permissions well enough to know that they can't do what they are wanting to do.

Very rarely has it been a bug in the implementation of FreeNAS. Remember that permissions are a combination of how ZFS stores them plus how Samba interprets and enforces them. So the two have to work together to get the job done or everything is broken. Normally it would be your responsibility as the admin of the server to make sure these two things integrate well together. We get to shirk that responsibility because FreeNAS pretty much does all that hard work for us. But that also means that most users lack the knowledge to know how all of that integration actually works, so users blindly experiment and experiment some more until they figure it out or get frustrated and give up.

At the end of the day, FreeNAS doesn't unnecessarily limit your options. The options, however, are limited to the casual observer that doesn't understand how all this stuff works in the background. iXsystems has plenty of users with 100s+ of users and many departments, multiple datasets, quotas and reservations, many shares, etc. and it all works... properly. So there is no doubt in my mind that this stuff can and does work properly. The answer lies in understanding, for yourself, how all of this integrates and works. That doesn't come easy, and since FreeNAS is pretty much a push-button install it also negates the experience and knowledge that would come from being forced to integrate ZFS and Samba for yourself.

 

[Philip Robar]

There doesn't seem to be a way to delete a post. Perhaps a moderator can do it.

 

[cyberjock]

I'm not going to go troubleshooting the problem (sorry). I've learned years ago (and the other mods here also learned years ago) that there are too many variables to troubleshoot this stuff in a forum setting. But, you left out key details in your "simple" looking 7 steps. It should have been something like 500 steps because of things like:

1. Which user was being authenticated when your client attached? If they had been attached to the old pool that you destroyed in step 2, you'll be in for a surprise as to what is cached on step 4. Oh, and if its any consolation, Mac and Windows don't always cache the same thing at the same time for the same reason. One or both will discard the cached login credentials in certain circumstances (and those circumstances don't match). So unless you have debug logs, I'd argue you didn't know who you were really logging in as.

2. Traversing directories that have permissions that Windows tries to "fix" because it thinks something is wrong can take something that works and suddenly appear to "break" it thanks to various other things. Nothing shocking at all to the behavior, assuming the behavior is correct. All you actually did was observe and record this behavior, but again I see no debug log showing the actual cause. You included no getfacl output showing a directory that you should have been able to write to, and suddenly your permissions changed. Or that getfacl shows you should have full permissions even when you don't. No logs validating the user you are even authenticating as.

I could probably list a dozen more reasons why your 7 steps are far too simplified to really be useful for troubleshooting, but I hope you get my point.

The problem is that every time we hear that some new bug comes up, they generally provide a short list of steps to reproduce the problem. iX devs, as an almost guaranteed rule, are unable to duplicate the behavior. On many occassions a dev actually did a remote session with a user only to find the user had less of a clue than they thought and a quick getfacl would have made it immediately obvious that they shouldn't have a certain permissions level, despite what the user thought or the user wasn't logged in as the user they were sure they were logged in as. Yes, it is possible to put in a username and password that you think would authenticate as a user, and actually end up a guest with no outward sign that you are actually a guest.

I hope I don't sound condescending with this post, but devs have played this ball game plenty of times, and there are hundreds of TrueNAS (and FreeNAS certified systems) in large-scale production environments that don't have directories suddenly go read-only with no reason, and many of the other behaviors that many users have claimed in the past have turned into "user config errors". So I'd argue that unless you plan to show a debug log with massive amounts of output that include proof that "userX" should have full permissions to "folderY" and they are only being given read-only, or that a getfacl shows full permissions, and suddenly a getfacl shows read-only permissions with no log in Samba, expect the blame is going to be put on the user. If you want to make the devs feel like its a bug, provide the copious amounts of data that backup your claim. I don't mean say "I think I have a bug, what do you want". I mean provide the required information to unequivocally prove that you did everything right, that X was working properly and suddenly X was misbehaving with nothing to explain the behavior in the debug logs.

Personally, because of what a P.O.S. Windows is and how much stuff it caches all over the place, I don't believe in troubleshooting via Windows. It does *way* too much stuff automagically in the background with no clue presented to the end-user that it is doing what it is doing, but the debug logs of other systems over the years make this a pretty damning fact. I use Linux when I can to teach permissions. Linux doesn't play games like Windows does. Once Linux works then you can transfer that to Windows and see what Windows is doing. In fact, when I started writing my permissions guide, I had Windows in a VM, Linux in a VM, and an ssh session to the FreeNAS server with Samba logging on debug so I could watch it *very* closely to disassemble what was going on from what I thought was going on.

The real problems FreeNAS had with CIFS permissions were in early 9.2.1.x releases. 9.2.1 brought about Samba4, which was a wholly different beast from Samba3. Even myself, I was one of the last of the mods to migrate to 9.2.1.x. When 9.2.1.5 came out word got to iX that I was still on 9.2.0 because I didn't want to break a perfectly working server. I was literally asked to test out 9.2.1.6 nightlies and file bug tickets so that the release version of 9.2.1.6 would be "cyberjock approved". I'm not even joking. When the largest poster of the forums mentions everywhere that he's 7+ version behind the latest, on purpose, because he doesn't trust the latest version at all, that really hurts. Eventually I did do testing and eventually migrated to 9.2.1.6-RELEASE. Of course we all know there was multiple versions after that (haha) for various reasons.

I'm not going to pretend that permissions are easy. Windows doesn't help you either with the games it plays in the background that you may not even be aware of. Even some of the stuff I've seen Windows do makes me wonder who the idiot was that thought it was a good idea to go traversing directories for stupid crap like picture thumbnails and letting Windows Explorer addons take the file share on a guided tour of potentially millions (and sometimes billions) of files and directories, with no regard for how much load that might create on the server, the network, or the client.

I don't use a Mac, but I've seen some behaviors that make me question how well OSX has implemented things on the Mac. I know that when it comes to ACLs, we still recommend going to a Windows machine to change the permissions (optionally if you are a CLI guy you can do permissions from the FreeNAS CLI). Macs have no good alternative to setting ACLs, at all (one damn good reason to question how well OSX has implemented things).

Other than that I can't really help more. I deliberately avoid trying to help with permissions because it quickly turns into an argument and often remote sessions that are measured in days and not hours. Nobody here is paid to write posts, so spending multiple days explaining things and proving that FreeNAS doesn't do the things they are 100% adamant that FreeNAS is doing and leads to even more questions and more proofs. Nobody here really has that kind of time to help every user that thinks they found the next FreeNAS permissions bug. :(

If you think it's a bug, I challenge you to prove it. Show the devs with a debug log, quote the applicable entries, validate every little thing that many people would assume and prove that X is happening when it shouldn't. That's the best way to get the devs attention. They will often tune out immediately if you can't provide a solid proof of your situation because if you can't prove it to them in a bug ticket, you probably don't even know where the very things you should be using to troubleshoot are (like samba logs, how to set the logging to debug, etc.). I will tell you that a vague 7-step procedure is going to be shot down pretty quickly because any kind of permissions problem is either an edge case that is rarely hit and therefore extremely complex to prove, or is so simple to hit that everyone AND their mother is hitting it, which means it is also easy to reproduce.

Good luck.

 

[anodos]

OK I have a debug file saved... Attached.

The UNIX wheel group is straight from FreeNAS. It does have the full control permissions so that is not the problem but that the Windows group "EVERYONE" has lost some rights to this new "unknown" is the issue I am assuming!

The commands below ( from https://forums.freenas.org/index.php?threads/cifs-permissions.28813/#post-199285 ) are said to perhaps fix this issue. These I assume are run from the FreeNAS GUI shell screen.
If so then how are they going to change mappings on the Windows machines though?
Anything else I should know before running these lines?
Perhaps I'd best wait to see if anything in the debug files stands out to one of you FreeNAS experts before stopping / restarting the samba service?

service samba_server stop
rm -rf /var/db/samba4/*
rm -rf /var/etc/private*
net groupmap cleanup # this should blow away all mappings
service ix-pre-samba start # create config (and group mapping) with known SID
service samba_server start

Thanks folks. So much!
R

Are you still having problems?

A few things stood out:

• It's good to start from a blank slate. (1) Recursively set dataset permissions to be owned by the correct user / group through the GUI. (2) verify that permissions type is set to "windows". (3) In share config, check "apply default permissions" and click "OK" (4) Clear cached credentials on clients
• I noticed that you have mapped a directory within one of your shares and are attempting to modify its permissions. This isn't really the best way to go about your permissions. Set permissions at the share level. For instance: you should navigate to \\FREENAS, right-click on "Music" and set permissions through the security tab. Don't map "\\FREENAS\MUSIC" and change permissions on "Frank Zappa".
• I noticed you have your datasets owned by "nobody" (your guest user), but you are authenticating as "richie" this will effectively make your permissions level that of "everyone" which tends to be RO. You should make "richie" owner of the datasets, authenticate as "richie", then add read, write, modify ACE for "nobody" or "everyone". The way guest access in samba works is it maps bad users to the guest account (your guest users will appear as "nobody"). If you authenticate against the server you will no longer have guest-level access, and possibly less permissions than a guest would have.

 

[Noctris]

Ok.. so apparently, i reproduced this problem ( as in, i noticed this morning all of my shares are now read-write only, no edit , and i notice the SID guid instead of a name ) so as i'm willing to bite. Since this is a non production system
( well.. my kids might be missing out on some quality time with miffy but hey)

I'd like to know how similar our config is.

My current setup ( since this is a home box awaiting it's new motherboard, ram and cpu):

Last version freenas
Cifs running and joined in active directory domain, guest account = nobody
Alle cifs shares setup as guest only
Shared datasets are all 0777 with nobody as both group and owner, unix style permissions

When looking at the files, i see this:
("nieuw textdocument.txt = new document.txt"

"Iedereen" = "everyone")

• I can create a file or folder
• Delete a file or folder
• Cannot modify of rename.

I noticed that when i turn off guest only and change ownership to an Active directory user and group, it does work, but even when changing it back, i still don't see the normal "guest user"
I have no access to change permissions from a windows client.


Question for you: Are you running Active directory ? ( at this point, all my shares are "guest only" and permissions are set up like that too) i'm wondering if this has something to do with mixed up sid's as the article previously mentioned has led to believe ( although i tried that procedure to no effect)

 

[anodos]

Not that it's worth anything, but all of the shares in question were guest only, mapped to user "nobody". The datasets are owned by "nobody" and group "wheel". There are no added user accounts on the FreeNAS box.

So basically what you're saying is

1) Apple and Microsoft's SMB implementations are so crappy that it's a miracle that they appear to work at all, let alone allow you to do anything useful.

My above post was in reference to the OP, who had helpfully included a debug file. Unfortunately, this thread appears to be repeatedly thread-jacked and so it's hard to keep everything straight.

I wouldn't necessarily call it crappy. It's just there are lots of small and large factors at play that make it hard to troubleshoot permissions problems in a forum post. Just take a stroll through the samba mailing lists. Perhaps you should start a new thread and post a debug file.

2) Samba 4 and iXsystems' GUI are such paradigms of godlike perfection that users have to be able to provide evidence that only knowledgeable networking professionals would know how to gather to get an iXsystems developer to even deign to look at what might to be a bug. (This conflicts with my experiences interacting with iXsystems engineers on the test alias, but you're the expert so who am I to disagree.)

Actually, I think most people can agree that the UI is somewhere borderline terrible, which is why they're overhauling it for 10. I think most people who use samba regularly would be hesitant to use the term 'godlike perfection'.

3) iXsystems and other Samba vendors know that (1) is true, but even so they hand their customers a bomb that will eventually blow up in their faces. In effect they either don't give a damn and/or they're all laughing they're way to the bank to cash their giant support and consulting checks—thanking Apple and Microsoft all of the way.

It's not really a bomb, but it can have a steep learning curve (Not intentional. It's just complicated. You're essentially turning a Unix server into a windows server.).

4) Given all of this you'd have to be an incompetent engineer to use SMB in a business situation, and home and SOHO users should run away screaming as fast as they can and only use sneaker net to move files around. (Sssssh, don't tell them what a piece of crap the USB protocol is or they'll all just curl up into wailing balls of madness.)

5) I'm such a moron that even following iXsystems directions, with a pretty good understanding of how ownership and permissions work, and not going anywhere near the command line, I managed to screw up a working system—including parts of the system that I didn't even touch. Gosh, I must be both stupid and magical at the same time.

6) If you're not a big bucks paying customer, then you'd be an idiot to use FreeNAS for SMB file sharing, because given (1), it's only a matter of time before your network stops working and no one will be able and/or willing to help you figure out what went wrong.

It now occurs to me that, given how stupid I am, it's amazing that my OS X and Windows machines have no problems talking to each other—especially given what pieces of crap they are. Must be magic.

Anyway it seems then that I have a couple of alternatives:

1) Go back to FreeNAS 9.2, the solution for simpletons.
2) Switch to NFS and hope that at least one of the NFS Windows clients actually works. (NFS, because occasionally you might actually want to do something useful with your network.)

Start a new thread and post a debug file.

 

[Noctris]

Just to make sure: i was not trying to hijack the thread in any way either. I'm just saying i experienced the exact same behaviour and for th love of god cannot figure out WHY this occurs all of the sudden.

This box has been running for 2 weeks, and this morning, one minute i'm editing files, the next ALL my shares are read/create/delete only but no modify. (ironically the first post i saw here, before even able to enter a search string, was this one..)

 

[mudshark]

What version of freenas? And a debug file please.

Was that debug file at all helpful?
Any other ideas?

I suppose I could setup FreeNAS to only server via webdav and map drives that way...

 

[Noctris]

I read though screens and screens of my own samba debug logs but could not find anything to point at.

As a fix, for me, i did something the following: ( as i have this feeling somewhere along the line samba seems to had lost the connection between account nobody and the sid it gave it)

• I created a user samba_guest ( which will automatically create a group) ( no password, microsoft account checked)
• configured samba to use this samba_guest as guest account
• modified the dataset permissions to make samba_guest user and group owner on the data ( recursivly setting permissoins)

after this, i had normal access again. This has been running since this morning. I am very curious wether this will hold permanently but next to existing shares , i created a specific test share to keep looking at so i have a "non edited" sample later on, would this occur again.

 

[mudshark]

I read though screens and screens of my own samba debug logs but could not find anything to point at.
As a fix, for me, i did something the following: ( as i have this feeling somewhere along the line samba seems to had lost the connection between account nobody and the sid it gave it)

• I created a user samba_guest ( which will automatically create a group) ( no password, microsoft account checked)
• configured samba to use this samba_guest as guest account
• modified the dataset permissions to make samba_guest user and group owner on the data ( recursivly setting permissoins)

after this, i had normal access again. This has been running since this morning. I am very curious wether this will hold permanently but next to existing shares , i created a specific test share to keep looking at so i have a "non edited" sample later on, would this occur again.

That makes sense, but I have a question about the second step.
Can I configure "samba to use this samba_guest as guest account" in the windows GUI?
With my CIFS history I'm anxious that the "fix" won't stick w/o knowing why it's happening over and over again in the first place!
thanks.

 

[Noctris]

Sorry, i wasn't really clear. It's called CIFS in freenas. Samba is the name of the software handeling the cifs protocol.

All steps are to be done on freenas.

first, go create the user in Account -> Users like this:

Then go over to services, edit CIFS config and set it like this:

Finally,

set permissions on your datasets you share:

Concerning your "luck".. i'm afraid i can only advise you to buy a a six pack so IF there is a next time, you have something to drink while you do this again and ultimatly will not care anymore ( somewhere between the third and the fifth beer :p

 

[anodos]

That makes sense, but I have a question about the second step.
Can I configure "samba to use this samba_guest as guest account" in the windows GUI?
With my CIFS history I'm anxious that the "fix" won't stick w/o knowing why it's happening over and over again in the first place!
thanks.

I looked at your debug file and it indicated that you authenticated to your FreeNAS server with an account other than your guest account. Since the share is owned by your guest user, this would typically confer read-only access to the share. See reply #12 above.