SOLVED SMB Broken after update to 9.10.2-U3

[Saxito Pau]

So since my update from 9.10.2-U2 to 9.10.2-U3 about a week ago, I have found that I can no longer access my SMB shares from any of windows PCs.
If I reboot the server or restart the SMB service, the shares work again, but only for 2 to 3 minutes, then they become inaccessible.
Plex and Transmission works fine in spite of this (just can't access their storage folders directly).

It is a frustrating thing. I literally have two or 3 minutes to copy files off my freenas server.

I tried to boot from an earlier build. ( 9.10.2-U2, then 9.10.2) but in the latter case, I couldn't access the shares at all.

I'm not that big of a FreeNAS/FreeBSD/Linux person (especially in the command line area) but I am a techie, so I can follow any instructions given to investigate the issue further.

HELP!!

UPDATE - I can't even restart the service via GUI now

 

[joeschmuck]

Please provide your system specs.

If this was just an update from U2 to U3 then roll back to U2, I know you said you tried this but I need more details.

Also lets change your mindset just a bit and place it into "troubleshooting mode". Stop thinking it must be FreeNAS because it may not be. This could be anything else in your network, including your computer. Why do i say that you ask? Good question!

1) Plex and Transmission still work.
2) You installed the upgrade a week ago and didn't have immediate problems but now you can't get anything to work for any length of time.
3) You problem sounds like your computer, not FreeNAS.

So you need to start troubleshooting and think about all the possibilities, don't throw something out unless you have proven it first. For instance, use a differerent computer to access the files, does the same thing happen? If yes then connect the computer directly to the FreeNAS Ethernet port and see if you still have the issues. If Yes then repalce the Ethernet cable and try again. If you still have the problem and you have connected two different computers to FreeNAS directly then it does sound like FreeNAS as you have ruled out everything else.

So these are the first steps I'd try. It's not normal for you to update FreeNAS (or any software on any device) and it works fine for a while but then just craps out and never works fine again. If it were the FreeNAS upgrade then I would expect then when you rebooted the system that it would work fine for a while, not crap out in under 10 minutes. I think you would have noticed that after the initial upgrade.

 

[Saxito Pau]

Specs:

• SuperMicro X9SCM-F
• Intel Xeon E3-1225 v2
  
• Crucial 16GB Kit (2 x 8GB) DDR3L-1600 ECC
  
• 2 x 3TB Seagate NAS drives, mirrored
• 1 x 4TB Seagate NAS drive, single drive pool
  
• 1 x 3TB Seagate Desktop drive, single drive pool
  
• 1 x Sandisk U100 16GB SSD as the boot drive
• Fractal Design Define Mini case
  
• PC Power & Cooling Silencer Series 400 Watt

I will state that due an unusually busy week I had, I did not really have to time to do any real investigation until last evening.

To answer your troubleshooting questions:

Using 9.10.2-U3:
Accessing the shares from other computers yields the same result.
Plugged a known working cable directly via ethernet to the FreeNAS server. I am able to ping it via hostname and ip address, and log into it via web browser.
I cannot access any shares via hostname or IP address from Windows explorer, even after restarting SMB via GUI.
I can access Plex and Transmission.

Switched to 9.10.2-U2, rebooted
repeated the above steps. Same outcome.

Anything else I should look at?

 

[joeschmuck]

The permissions of the shares is likely not correct. Check those, but in your original posting you state that you can access the shares for 2-3 minutes before they stop. Is that true? Or are they not working from the time you boot the system up? These are two very different pieces of information. If they work initially and then fail after 2 minutes then it's not a permissions issue. If they fail right away then it could be a permissions issue. This is key data of course and makes a huge difference in the way we troubleshoot this problem.

Lastly,
You could also reinstall FreeNAS and restore your configuration file from before the upgrade to U3.

 

[Saxito Pau]

Initially

The permissions of the shares is likely not correct. Check those, but in your original posting you state that you can access the shares for 2-3 minutes before they stop. Is that true? Or are they not working from the time you boot the system up? These are two very different pieces of information. If they work initially and then fail after 2 minutes then it's not a permissions issue. If they fail right away then it could be a permissions issue. This is key data of course and makes a huge difference in the way we troubleshoot this problem.

Lastly,
You could also reinstall FreeNAS and restore your configuration file from before the upgrade to U3.

Originally, it worked for a few minutes before failing. Now, it fails outright, never works, even after reboot

I will go with the latter for future troubleshooting.

I may well have to reinstall FreeNAS, I need to access a 24GB zip file asap.

 

[joeschmuck]

Thats a big zip file. Good Luck.

 

[nojohnny101]

At this point, I echo @joeschmuck that a fresh install is in order. Then import config file.

The behavior seems strange, as was explained above, two very different issues if it works but then fails or if it doesn't work at all.

Just to be thorough, have you doubled check your SMB settings after the upgrade rather than just flipping the switch to turn it off and on? Highly unlikely, but something might have gotten messed up with your SMB settings during the upgrade process.

Also have you tried sharing with another service (FTP or NFS) and what was the result?

 

[Saxito Pau]

So no good news.

Clean FreeNAS install using 9.10.2 (a476f16),
Upload the most recent matching config backup.

Still no access to shares. Transmission now broken as that was setup in a newer build of FreeNAS. But this is minor in relative terms.
Plex still works.

At this point I will simply have to painstakingly rebuild the server and manually recreate all the shares.The one thing I'd like to do is preserve my Jails, so I won't have to reset/rebuild my Plex database. Any tips on that will be appreciated. I do know I will have to screenshot every single page of the most recent config I had before starting.

As for the 24GB file, I'm just going to re-download it on my main PC.

 

[Ericloewe]

It's vaguely possible that something on the clients is doing something funny and trying to force NTLMv1 authentication after it somehow detects the server. Google how to force Windows to use modern authentication.

 

[joeschmuck]

My last advice is to change all the permissions for the dataset to Unix and 777. In the SMB configuration ensure you have "Allow Guest Access" checked.

Also, replace your boot device with something else, could be a USB Flash drive, and try again.

Lastly, run a scrub to ensure all your data is in-tact.

I still feel it's a permissions issue. And turn off your firewall on your computer, if you have one, see if that helps.

Something happened but I'm skeptical that it's FreeNAS. You should have FreeNAS errors if something was going wrong.

Damn, I can't stop thinking about this... Do you run SMART tests on your drives?

Just to make sure I understand it, when you say you cannot access the shares, does that mean you cannot see the shares at all in Explorer or that you can see the shares but cannot access them? It may not sound like a big difference but it is.

I also don't feel your data is lost so I wouldn't wipe the hard drives. If you think it's your FreeNAS hardware then run the Memtest86 on it overnight to see if it works. This may help isolate a hardware issue.

 

[Saxito Pau]

Oooh, so many replies. Let's see how to reply to them all.

Please note that I will actually try any advice suggested in the morning (8 to 12 hours after time of this post). Too tired after battling a FB troll over a dash cam video of me nearly getting in a car accident Friday.

It's vaguely possible that something on the clients is doing something funny and trying to force NTLMv1 authentication after it somehow detects the server. Google how to force Windows to use modern authentication.

I'll have a look

My last advice is to change all the permissions for the dataset to Unix and 777. In the SMB configuration ensure you have "Allow Guest Access" checked.
- Most of my shares were already set to "allow guest access" as I am effectively the only person accessing them. Very handy when you're installing software on a pc you just installed Windows and lan drivers on.
I'll do that unix method.

Also, replace your boot device with something else, could be a USB Flash drive, and try again.
- will do and report

Lastly, run a scrub to ensure all your data is in-tact.
- will do and report. I did have scheduled scrubs on a monthly basis

I still feel it's a permissions issue. And turn off your firewall on your computer, if you have one, see if that helps.
- Current using a custom-built pfsense firewall router that works fine. That was setup about 2 months ago without issue. Will turn off firewall and report.

Something happened but I'm skeptical that it's FreeNAS. You should have FreeNAS errors if something was going wrong.

Damn, I can't stop thinking about this... Do you run SMART tests on your drives?
- SMART tests are set to run on a monthly schedule. Short tests early in the month, long tests on various days during the rest of the month (once a month for each drive)

Just to make sure I understand it, when you say you cannot access the shares, does that mean you cannot see the shares at all in Explorer or that you can see the shares but cannot access them? It may not sound like a big difference but it is.
- I cannot see the shares at all in Explorer.

I also don't feel your data is lost so I wouldn't wipe the hard drives. If you think it's your FreeNAS hardware then run the Memtest86 on it overnight to see if it works. This may help isolate a hardware issue.
- I am certain the data are intact myself. One thing I love about freenas is the fact that an Os reload does nothing to your data. just re-import drives in the worst case non-pool/data loss scenario....
I'll get memtest running in the morning

 

[Saxito Pau]

Ok, So I finally decided to take another crack at it.

Installed 9.10.3-U3 to the 16GB SSD again. Restored the last config I made just before calling on you guys for help.

An idea suddenly popped into my head to simply delete one of the SMB shares and re-create it.
All the shares worked after.

Checked back about 30 minutes later, still works.

BLOODY HELL.

I feel like I just wasted everyone's time with this issue and wonder if I could have done better trouble shooting this myself before making this thread.

Of course there is still the matter of the post-mortem analysis. How and why this happened in the first place (if it wasn't the update)?

I am officially taking this success with a grain of salt for now. Giving it 24 hours before I declare this "resolved."

 

[joeschmuck]

Could you explain what made you delete the SMB share? Did you have more than one SMB share to the same dataset/folder? And when you recreated the SMB share, was it to the same location? I'm trying to learn from what happened and see if there was either a flaw with FreeNAS or human error. I'd rather see human error to be honest with you but that is not always the case.

Thanks.

 

[Saxito Pau]

Could you explain what made you delete the SMB share? Did you have more than one SMB share to the same dataset/folder? And when you recreated the SMB share, was it to the same location? I'm trying to learn from what happened and see if there was either a flaw with FreeNAS or human error. I'd rather see human error to be honest with you but that is not always the case.

Thanks.

Was re-reading the entire thread in case I missed something. Looked at one of your posts and saw this:

"If they work initially and then fail after 2 minutes then it's not a permissions issue. If they fail right away then it could be a permissions issue. This is key data of course and makes a huge difference in the way we troubleshoot this problem."

..and that is what gave me the idea.

The share was recreated to the same location and the same name. It has guest access permissions.

I'm happy to help as I need to know what I did to cause this.

The only other thing thinking back (that I forgot to mention earlier) is that I recall is one of the shares being at full capacity, so I deleted a few files to recover space. This was before the mess started.

 

[joeschmuck]

Well lets hope this is resolved for you.

 

[Saxito Pau]

I guess we can consider this matter resolved.

17 hours later, everything is working like a charm. No further issues observed.

My thanks to all who posted to assist me with this.