FreeNAS 9.2.1.7 - Active Directory - CIFS - Disconnects

[latez]

Hello All -

I've recently replaced my aging Synology DS410 with a newly built FreeNAS server (8x4tb). Setup & Installation has been a relative breeze as has been configuring FreeNAS to use AD for authentication. I run a Windows 2008 Domain Controller at home and FreeNAS had no issue connecting and using it to authenticate users.

The problem however is stability. On a regular basis the CIFS just stops working... I get red X's next to the mapped drives. Once I click on the drives themselves it wakes up and reconnects - Ordinarily I *could* live with this until the next version of FreeNAS comes along and a newer hopefully better working version of Samba is available. However - I am having disconnects while playing media over CIFS as well. I have not yet seen a file transfer crash but its common for me to be watching a movie or tv show off of my FreeNAS and have it disconnect at some point.

Please note that NFS is working flawlessly and I have not seen it drop what so ever. Pouring over the smb.log even with log level set to debug has yet to yeild anything worthwhile. This is the very last issue I am having which is stopping me from being able to fully enjoy my NAS server.

My line of thinking was that if I can't get this working - I may have to resort to running Windows in a VirtualBox VM and serve CIFS this way however I am doing everything I possibly can to avoid this. Any insight or recommendations would be mostly humbly appreciated. Please note the Synology which was also using AD *never* disconnected.


FYI Specs -
ASRock C2750D4I
Intel Avoton C2750 Octa-Core
32GB (4x8gb Crucial 1600 DDR ECC)
8 x 4tb HGST drives
1 x 128g Samsung Evo (L2ARC)


PS - this is my first post on freenas.org (Although I've been a user and customer (TrueNAS)) for quite some time. Please keep up the fantastic work and I am super excited about 9.3 and 10!!

 

[cyberjock]

So the disconnecting of network shares is a 3-way streak. Pardon the detail, I'm including it for completeness....

1. Samba, by default, disconnects idle connections after 4 hours (pretty sure it's 4 hours, but might be 8 or 12). This typically isn't a problem as your clients almost always send keep-alive messages at much shorter frequencies than that.
2. Windows will automatically disconnect shares after a predetermined limit (the limit changes based on your Windows version). If I remember correctly the setting is changed via a registry edit, but I don't know more than that.
3. If the TCP stack on either end is wiped out (which happens on such things as your Wifi losing signal and reconnecting on the client) or your server gets disconnected from the network then you will lose the share.

There are some people that have had problems with random disconnects on the ASRock motherboards. It sounds like you might fall into that category. I'd recommend you watch the server more closely for signs of the NIC going offline and then online. If so you might need to contact ASRock to determine how to correct the issue via a RMA or whatever.

I will tell you that running Windows in a VM to serve CIFS is likely not going to solve this problem, but you will find the performance through a VM to be absolutely terrible even if you do try it. ;)

 

[latez]

So the disconnecting of network shares is a 3-way streak. Pardon the detail, I'm including it for completeness....

1. Samba, by default, disconnects idle connections after 4 hours (pretty sure it's 4 hours, but might be 8 or 12). This typically isn't a problem as your clients almost always send keep-alive messages at much shorter frequencies than that.
2. Windows will automatically disconnect shares after a predetermined limit (the limit changes based on your Windows version). If I remember correctly the setting is changed via a registry edit, but I don't know more than that.
3. If the TCP stack on either end is wiped out (which happens on such things as your Wifi losing signal and reconnecting on the client) or your server gets disconnected from the network then you will lose the share.

There are some people that have had problems with random disconnects on the ASRock motherboards. It sounds like you might fall into that category. I'd recommend you watch the server more closely for signs of the NIC going offline and then online. If so you might need to contact ASRock to determine how to correct the issue via a RMA or whatever.

I will tell you that running Windows in a VM to serve CIFS is likely not going to solve this problem, but you will find the performance through a VM to be absolutely terrible even if you do try it. ;)

Cyberjock thank you for your quick reply!!

1. I am not sure it does this by default - but I did notice the defautl deadtime is set to 15, which I find to be a little odd and yes windows *should* be sending keep alive messages far more frequently, as I said the Synology which was running Samba3 iirc did not have any issues like that.

2. I am aware of this windows issue and have already made the appropriate registry changes, even though they were not at all necessary with the Synology. Unfortunately this has not had any affect.

3. The server is currently connected to a gigE switch as is my windows machine which I am doing the majority of testing on, so its a full copper connection. Now I could suppose that both of the Intel i210 (onboard) nics could be dropping connections after an unspecified period of time in which case I can always throw in another NIC and test with that but this seems to be a problem specifically with Samba/AD. The reason I say this is that when using DLNA over the network to feed my media devices and/or Plex (running on a different machine but with media mounted via NFS) this never occurs. Also at the time the Samba/CIFS connection breaks I am usually connected via SSH and it never fails. I should also probably mention that I have FreeNAS setup on a failover lagg connection (both i210 nic's)

4. Regarding Windows in a VM - I suspected as much but wanted to throw the idea around and see what you guys thought. Its very unfortunate that this wont be possible.

My alternatives unfortunatley are bleek if I can't get it CIFS working reliably. The alternative may have to be to throw in a spare raid controller (I happen to have a Dell H700 in my bin somewhere) and run Windows Server directly on the machine. Use HyperV to run the Linux VM's for Sickbeard, Sabnzbd and so on. Although I would really really much rather not leave FreeNAS/ZFS.

I hope someone can help me fix this! I don't want to go Windows :(




This is what I currently see in logs -

smbd.log - As you can see it closes all the connections then reopens them - but i cant seem to figure out what is setting this off...

[2014/09/29 00:59:02.047620, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Photos
[2014/09/29 00:59:02.047940, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Apps
[2014/09/29 00:59:02.048196, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Video
[2014/09/29 00:59:02.048448, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Netbackup
[2014/09/29 00:59:02.048695, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Downloads
[2014/09/29 00:59:02.048949, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Games
[2014/09/29 00:59:02.049206, 1] ../source3/smbd/service.c:1130(close_cnum)
10.0.0.2 (ipv4:10.0.0.2:59194) closed connection to service Music
[2014/09/29 00:59:17.089987, 2] ../source3/param/loadparm.c:535(max_open_files)
max_open_files: increasing sysctl_max (11095) to minimum Windows limit (16384)
[2014/09/29 00:59:17.090177, 2] ../source3/param/loadparm.c:543(max_open_files)
rlimit_max: increasing rlimit_max (11090) to minimum Windows limit (16384)
[2014/09/29 00:59:17.091494, 1] ../source3/param/loadparm.c:3178(lp_do_parameter)
WARNING: The "acl check permissions" option is deprecated
[2014/09/29 00:59:17.092648, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Apps]"
[2014/09/29 00:59:17.093930, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Backup]"
[2014/09/29 00:59:17.095206, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Downloads]"
[2014/09/29 00:59:17.096493, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Games]"
[2014/09/29 00:59:17.097780, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Media]"
[2014/09/29 00:59:17.099049, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Music]"
[2014/09/29 00:59:17.100318, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Netbackup]"
[2014/09/29 00:59:17.101606, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Photos]"
[2014/09/29 00:59:17.102883, 2] ../source3/param/loadparm.c:3581(do_section)
Processing section "[Video]"
[2014/09/29 00:59:17.121463, 2] ../lib/util/modules.c:191(do_smb_load_module)
Module 'aio_pthread' loaded
[2014/09/29 00:59:17.122162, 2] ../lib/util/modules.c:191(do_smb_load_module)
Module 'streams_xattr' loaded
[2014/09/29 00:59:17.123575, 2] ../lib/util/modules.c:191(do_smb_load_module)

 

[cyberjock]

Well, now that you've mentioned you are doing LAGG I should mention that you should try without it. Far to often people think they know what they are doing with LAGG, but don't meet all of the network hardware and software setup to make it work properly. It's possible that LAGG is causing your problem. If you search the forums you'll find that for quite a few people, even those that claim to be an expert on LAGG, are unaware of the requirements.

You also have a lot of people that setup jumbo frames and do other things without actually understanding the potential consequences, so if you have anything that isn't default related to your network configuration you might want to get rid of those settings just to rule them out.

This issue *may* also be client-side. For example, your firewall software might be causing interference. It's not common, but it is well documented around here.

So I'd recommend you try disabling LAGG and use just one NIC and see how it works. If for no other reason but to rule it out as a cause for your problems.

Other than those I mentioned above, I don't have any recommendations. ASRock has had some networking problems based on a handful or so of users on the forums. I have that board in my Mini and all I can say is "it works for me". I know, not much consolation, but that's the best I can do unfortunately. :(

 

[latez]

So I just wanted to report back to you guys because maybe this could help somebody.

I've had a FreeNAS server setup at work for quite some time. Also have full active directory/cifs file shares and never experienced any issues of odd disconnects. I thought the setup was identical except for one particular option - 'Local Master'. While it was switched off at work for some reason it was on at Home. Since turning it off around 5pm yesterday I have not experienced a single disconnect. I even left 3 or 4 long DVD's play over night in sequence to make sure i didn't experience any disconnects and so far not a thing. I'll report back in a week and let you know how it went unless of course it stops working in which case I'll be back much sooner and a sad panda to boot.

 

[anodos]

So I just wanted to report back to you guys because maybe this could help somebody.

I've had a FreeNAS server setup at work for quite some time. Also have full active directory/cifs file shares and never experienced any issues of odd disconnects. I thought the setup was identical except for one particular option - 'Local Master'. While it was switched off at work for some reason it was on at Home. Since turning it off around 5pm yesterday I have not experienced a single disconnect. I even left 3 or 4 long DVD's play over night in sequence to make sure i didn't experience any disconnects and so far not a thing. I'll report back in a week and let you know how it went unless of course it stops working in which case I'll be back much sooner and a sad panda to boot.

AD member servers should not be set as local master.

 

[latez]

This became painfully obvious after I re-read the description of the service. This was likely the problem, and I am frankly a bit surprised I wasn't being disconnected more often.

 

[anodos]

This became painfully obvious after I re-read the description of the service. This was likely the problem, and I am frankly a bit surprised I wasn't being disconnected more often.

Well, workstations joined to an AD domain should be using DNS for hostname resolution. It might be worth your time to figure out why your workstation is falling back on NetBIOS name resolution (unless they're not joined to the domain), but that goes beyond the scope of FreeNAS forums. :)

 

[anodos]

*double-post*

 

[latez]

Well the issues have sadly re-appeared again and I cant think of a single damn way to fix it! It's driving me absolutely crazy. Yesterday and today just getting all these error when watching media.. :(


This is the only entries I see in log.smbd

[2014/10/11 12:30:13.739520, 2] ../source3/smbd/close.c:780(close_normal_file)
HOME\dave closed file TV/SomeTVShow/SomeTVshow.mkv (numopen=3) NT_STATUS_OK
[2014/10/11 12:30:13.740234, 1] ../source3/smbd/service.c:1130(close_cnum)
192.168.0.2 (ipv4:192.168.0.2:54287) closed connection to service Video
[2014/10/11 12:30:13.755048, 3] ../source3/smbd/server_exit.c:221(exit_server_common)
Server exit (NT_STATUS_CONNECTION_RESET)


This problem is driving me absolutely bat shit insane - sometimes it goes for a couple of days without appearing and some evenings I'll see it twice or three times in an hour. SO RANDOM!!

 

[jkh]

What CIFS client(s)? We use CIFS on FreeNAS daily and I haven't seen this kind of issue once, so there's something unique about your setup that's key here.

 

[latez]

Sorry my reply is a bit tardy -

I'm currently on the latest - 9.2.1.8

Hardware -
SilverStone DS380B Black Aluminum front door, SECC (Case)
ASRock C2750D4I Mini ITX Server Motherboard FCBGA1 (Intel 8-core Avaton)
FSP Group FSP300-60GHS-R 300W SFX12V 80 PLUS Certi
Crucial 8GB Kit DDR3L 1600MT/s (PC3-12800) DR x8 ECC UDIMM 240-Pin (4 for a total of 32GB)
HGST Deskstar NAS 4tb (8) - in RaidZ2
Samsung EVO 128G SSD (l2arc)


Domain - In order to double check everything I most recently did a vanilla installation of a Windows 2008 R2 Domain - it only has a couple of users in it. The FreeNAS had absolutely no problems joining the domain. Authentication is working perfectly.
CIFS Clients - I've now tried several. Specifically I've tried Windows Vista, Windows 7, Windows 8.1 and this weekend I've even tried Windows 10 technical preview.
As for my networking components - I use Intel NICs across the board and I'm using 3com/HP switches.

As stated previously - I've most recently (before the FreeNAS) was using a Synology DS411 (Which I believe runs Samba3) and had absolutely no problems with playing media off of shares nor did it at any time ever show a broken share map (image included).

I've tried to look for patterns in this kind of behavior but have yet to identify the issue. For example - I've tried playing video files at the exact same time on two different machines. One may crash while the other one keeps playing so its not as if all network connections are broken at the same time. Additionally - I use minidlna / plex (via nfs shares) to play my media content and they never drop or die like the CIFS shares so I think the problem is localized to Samba and/or my AD setup in some way.

If you have any additional questions I would be more than happy to answer it as long as I can get to the bottom of my problem and hopefully help others in the future who may be as unfortunate as me.

One of the more frustrating things that's occurring is when playing media via VLC on a Windows based CIFS client is these error which to me indicate the share dropping off but I could be mistaken.


File reading failed:

VLC could not read the file (Bad file descriptor).


---
Would it make sense to see if this is an issue specific to Samba by running a different version in a portjail?