SAMBA crashes while many reads occur

[crazybyte]

Hi community,

I'm using FreeNAS to share a directory over Samba to an amd64 device where a dockerized rTorrent instance is running.

rTorrent uses this share to either store downloaded or temporary files. When it's uploading (hence reading from the NAS) many files, the samba server close the client connection.
When it happens, I can see from the server the following logs (loglevel full):

Code:

[2019/03/16 13:38:23.047039,  1] ../source3/smbd/smb2_server.c:3918(smbd_smb2_io_handler)
  ignore NBT[0x49] msg
[2019/03/16 13:38:23.087477,  1] ../source3/smbd/smb2_server.c:3918(smbd_smb2_io_handler)
  ignore NBT[0x2A] msg
[2019/03/16 13:38:23.123200,  1] ../source3/smbd/smb2_server.c:3918(smbd_smb2_io_handler)
  ignore NBT[0xAF] msg
[2019/03/16 13:38:23.175463,  1] ../source3/smbd/smb2_server.c:3918(smbd_smb2_io_handler)
  ignore NBT[0x0F] msg
[2019/03/16 13:38:23.282583,  1] ../source3/smbd/smb2_server.c:3918(smbd_smb2_io_handler)
  ignore NBT[0x7E] msg
[2019/03/16 13:40:10.572218,  2] ../source3/smbd/close.c:802(close_normal_file)

While on the client, that runs Archlinux, it can be seen the following kernel message:

Code:

[Thu Mar 14 12:25:51 2019] CIFS VFS: Server 172.16.4.4 has not responded in 120 seconds. Reconnecting...

The mount options I use are:

Code:

sudo mount -t cifs -o credentials=/home/username/cifs_creds,uid=666,gid=666,rw,nounix,iocharset=utf8,file_mode=0755,dir_mode=0755,cache=none //freenas/Kodi/rtorrent/download /mnt/download

I have already tried to change the cache option to strict and loose, but it didn't help me. I also tried to change the protocol version using vers=xxx, but the issue persists.

I'm using Freenas 11.2-U2.1 with Samba 4.9.4 on the server, and CIFS 2.14 with Linux 4.20.12-arch1-1 on the client. I've initially wrote a post on the Archlinux forum, where I posted some more information about the client configuration, but then I understood that the problem reside on the server side.

The problem doesn't occur if I have only few torrents uploading at the same time. Does anybody experienced something similar? Or have some suggestions/tips/ideas about how to solve it?

Thank in advance to anyone will intervene!

 

[SweetAndLow]

Try using an ip for the sever. Remove that cache flag, remove the nounix flag. Simplicity is key when debugging stuff.

 

[anodos]

Try setting the following auxiliary parameters on the SMB shares and see if things stabilize:

Code:

aio write size = 0
aio read size = 0

 

[crazybyte]

Following both your suggestion I set on the server:

Code:

aio write size = 0
aio read size = 0

and mounted the share using only the following options:

Code:

credentials=/home/crazybyte/cifs_creds,uid=666,gid=666,rw,file_mode=0755,dir_mode=0755,vers=3.1.1

But the issue still persist :(...
I captured more logs using loglevel: full on the samba server (attached).
While on the client the kernel message is always the same:

Code:

[Tue Mar 19 12:45:34 2019] CIFS VFS: Server 172.16.4.4 has not responded in 120 seconds. Reconnecting...

 

[crazybyte]

I mistakenly cut out some lines in the attached log after the error smbd_smb2_request_error_ex. Here the right part:

Code:

[2019/03/19 12:41:48.773138,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
[2019/03/19 12:41:48.904591,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
[2019/03/19 12:41:49.259247,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
[2019/03/19 12:41:49.355925,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_OBJECT_NAME_NOT_FOUND] || at ../source3/smbd/smb2_create.c:296
[2019/03/19 12:41:49.355965,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[5] status[NT_STATUS_FILE_CLOSED] || at ../source3/smbd/smb2_server.c:2599
[2019/03/19 12:41:49.355981,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[9] status[NT_STATUS_FILE_CLOSED] || at ../source3/smbd/smb2_server.c:2599
[2019/03/19 12:41:49.358626,  2] ../source3/smbd/open.c:1447(open_file)
[2019/03/19 12:41:49.363722,  3] ../source3/smbd/trans2.c:3465(smbd_do_qfsinfo)
  smbd_do_qfsinfo: level = 1007
[2019/03/19 12:41:49.364591,  3] ../source3/lib/sysquotas.c:488(sys_get_quota)
  sys_get_vfs_quota() failed for mntpath[.] bdev[(null)] qtype[1] id[-1]: Operation not supported
[2019/03/19 12:41:49.364618,  3] ../source3/lib/sysquotas.c:488(sys_get_quota)
  sys_get_vfs_quota() failed for mntpath[.] bdev[(null)] qtype[3] id[-1]: Operation not supported
[2019/03/19 12:41:49.443969,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
[2019/03/19 12:41:50.350426,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
[2019/03/19 12:41:50.626486,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)

 

[anodos]

Can you post output of 'testparm -s'?

 

[crazybyte]

Can you post output of 'testparm -s'?

Code:

# Global parameters
[global]
    bind interfaces only = Yes
    deadtime = 15
    disable spoolss = Yes
    dns proxy = No
    dos charset = CP437
    interfaces = 127.0.0.1 172.16.4.4
    kernel change notify = No
    lm announce = Yes
    load printers = No
    logging = file
    map to guest = Bad User
    max log size = 51200
    max open files = 466967
    nsupdate command = /usr/local/bin/samba-nsupdate -g
    obey pam restrictions = Yes
    panic action = /usr/local/libexec/samba/samba-backtrace
    printcap name = /dev/null
    security = USER
    server min protocol = SMB2_02
    server role = standalone server
    server string = FreeNAS Server
    time server = Yes
    unix extensions = No
    idmap config *: range = 90000001-100000000
    idmap config * : backend = tdb
    acl allow execute always = Yes
    aio read size = 0
    aio write size = 0
    create mask = 0666
    directory mask = 0777
    directory name cache size = 0
    dos filemode = Yes
    strict locking = No


[Game]
    access based share enum = Yes
    comment = Game ROMs and BIOSes to use with emulators
    guest ok = Yes
    path = "/mnt/Main-Volume/Games"
    read only = No
    veto files = /.snapshot/.windows/.mac/.zfs/
    vfs objects = zfs_space zfsacl streams_xattr recycle
    zfsacl:acesort = dontcare
    nfs4:chown = true
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U


[homes]
    browseable = No
    comment = User's home directories
    path = "/mnt/Main-Volume/Home/%U"
    read only = No
    valid users = %U
    veto files = /.snapshot/.windows/.mac/.zfs/
    vfs objects = zfs_space zfsacl streams_xattr recycle
    zfsacl:acesort = dontcare
    nfs4:chown = true
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U


[Kodi]
    access based share enum = Yes
    comment = Dataset to store medias.
    guest ok = Yes
    path = "/mnt/Main-Volume/Kodi"
    read only = No
    veto files = /.snapshot/.windows/.mac/.zfs/
    vfs objects = preopen zfs_space zfsacl streams_xattr recycle
    zfsacl:acesort = dontcare
    nfs4:chown = true
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

 

[anodos]

remove preopen

 

[crazybyte]

remove preopen

Hi Anodos,

I followed your tip, but the issue still persists :(... Anyway, looking for samba logs this time, it seems the same behavior is triggered from a little bit different error! I sent you more detailed logs from my samba server via PM.
Can you please take a look at it?

Thank you

 

[pro lamer]

Hi, guys, were you able to resolve this?

Sent from my phone

 

[crazybyte]

not yet, I got a long private 3d with anodos. If we find a solution I will share it here. Is your case right the same of mine?

 

[pro lamer]

If we find a solution I will share it here

I'll be glad to see the outcome, hopefully a solution :)

Is your case right the same of mine?

Can't help here. Rather learning what type problems I might encounter. I'm considering a purchase o E5-26xx V3 based hardware.

Sent from my phone

 

[SAK]

I'm having a similar problem with lots of small random reads via smb. I cloned an Ubuntu VM, changed its hostname and IP address. TrueNAS was never crashing when just the 1 VM was doing lots of reads and a few writes. But when the second one starts reading, kaboom. I'm not sure how to access all of the info you will want to see from me, but I would definitely appreciate any help in resolving this.

Platform: Generic
Version:
TrueNAS-13.0-U3.1
AMD Ryzen 7 5800X 8-Core Processor
4 Threads (running as a VM in proxmox)

83.5GiB​

total available (ECC)

 

[anodos]

I'm having a similar problem with lots of small random reads via smb. I cloned an Ubuntu VM, changed its hostname and IP address. TrueNAS was never crashing when just the 1 VM was doing lots of reads and a few writes. But when the second one starts reading, kaboom. I'm not sure how to access all of the info you will want to see from me, but I would definitely appreciate any help in resolving this.

Platform: Generic
Version:
TrueNAS-13.0-U3.1
AMD Ryzen 7 5800X 8-Core Processor
4 Threads (running as a VM in proxmox)

83.5GiB​

total available (ECC)

If /var/log/samba4/log.smbd shows abort() followed by aio_cancel() in next line in the backtrace, then don't worry about it. The issue will be fixed in the next release.

 

[SAK]

Sorry, I posted in the wrong place actually. Someone can delete (I didn't see a way to delete or edit). The problem I was having was not samba crashing, but rather the entire VM. It was caused by too much vcpu core assignment to other VMs. It looks like if TrueNAS is waiting for CPU time, something can become unstable.