Intermittently getting EBADF & EINVAL over CIFS on Win2k8 and Win2k12 servers

[ttblum]

Hello,

I'm using a Sybase utility that backs up entire databases to a CIFS share on my FreeNAS 9.2.1.5 server every night. The database file sizes range between 60 GB and 500 GB, and I usually have about 4 servers backing up to it at the same time.

About once a week, the backup fails in the middle of it, outputting filesystem errors 'Bad file number' (EBADF) or 'Invalid argument' (EINVAL). From what I understand, these errors are coming the OS filesystem layer, not from the backup utility.

Has anyone else had EBADF or EINVAL errors over CIFS shares?

There doesn't seem to be anything else logged on either the FreeNAS server or Windows servers. The backups are being written to the 'mybackups' share.

Output of testparm:

Code:

Load smb config files from /usr/local/etc/smb4.conf
max_open_files: increasing sysctl_max (11095) to minimum Windows limit (16384)
rlimit_max: increasing rlimit_max (11095) to minimum Windows limit (16384)
WARNING: The "acl check permissions" option is deprecated
WARNING: The "idmap uid" option is deprecated
WARNING: The "idmap gid" option is deprecated
Processing section "[backup1]"
Processing section "[mybackups]"
Processing section "[mylogfiles]"
Processing section "[mylogrestore]"
Processing section "[app-backups]"
Loaded services file OK.
WARNING: You have some share names that are longer than 12 characters.
These may not be accessible to some older clients.
(Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.)
Server role: ROLE_DOMAIN_MEMBER
Press enter to see a dump of your service definitions

[global]
    dos charset = CP437
    workgroup = MYDOMAIN
    realm = MYDOMAIN.COM
    server string = FreeNAS1 Server
    server role = member server
    security = ADS
    allow trusted domains = No
    map to guest = Bad User
    obey pam restrictions = Yes
    smb passwd file = /var/etc/private/smbpasswd
    private dir = /var/etc/private
    max log size = 51200
    unix extensions = No
    deadtime = 15
    max open files = 11070
    load printers = No
    printcap name = /dev/null
    disable spoolss = Yes
    local master = No
    domain master = No
    dns proxy = No
    cache directory = /var/tmp/.cache/.samba
    pid directory = /var/run/samba
    panic action = /usr/local/libexec/samba/samba-backtrace
    template shell = /bin/sh
    winbind cache time = 7200
    winbind enum users = Yes
    winbind enum groups = Yes
    winbind refresh tickets = Yes
    winbind offline logon = Yes
    idmap config MYDOMAIN: range = 20000-20000000
    idmap config MYDOMAIN: backend = rid
    idmap config * : range = 10000-19999
    idmap config * : backend = tdb
    acl allow execute always = Yes
    create mask = 0666
    directory mask = 0777
    directory name cache size = 0
    kernel change notify = No
    map archive = No
    map readonly = no
    store dos attributes = Yes
    strict locking = No
    wide links = Yes
    dos filemode = Yes

[backup1]
    path = /mnt/volume1/dataset1
    read only = No
    veto files = /.snap/.windows/.zfs/
    vfs objects = zfsacl, streams_xattr, aio_pthread
    zfsacl:acesort = dontcare
    nfs4:chown = yes
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

[mybackups]
    path = /mnt/volume1/dataset1/myBackups
    read only = No
    veto files = /.snap/.windows/.zfs/
    vfs objects = zfsacl, streams_xattr, aio_pthread
    zfsacl:acesort = dontcare
    nfs4:chown = yes
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

[mylogfiles]
    path = /mnt/volume1/mylogfiles/myLogFiles
    read only = No
    veto files = /.snap/.windows/.zfs/
    vfs objects = zfsacl, streams_xattr, aio_pthread
    zfsacl:acesort = dontcare
    nfs4:chown = yes
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

[mylogrestore]
    path = /mnt/volume1/mylogfiles
    read only = No
    inherit acls = Yes
    veto files = /.snap/.windows/.zfs/
    vfs objects = shadow_copy2, zfsacl, streams_xattr, aio_pthread
    zfsacl:acesort = dontcare
    nfs4:chown = yes
    nfs4:acedup = merge
    nfs4:mode = special
    shadow:format = auto-%Y%m%d.%H%M-5w
    shadow:localtime = yes
    shadow:sort = desc
    shadow:snapdir = .zfs/snapshot
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

[app-backups]
    path = /mnt/volume1/dataset2
    read only = No
    inherit acls = Yes
    veto files = /.snap/.windows/.zfs/
    vfs objects = zfsacl, streams_xattr, aio_pthread
    zfsacl:acesort = dontcare
    nfs4:chown = yes
    nfs4:acedup = merge
    nfs4:mode = special
    recycle:subdir_mode = 0700
    recycle:directory_mode = 0777
    recycle:touch = yes
    recycle:versions = yes
    recycle:keeptree = yes
    recycle:repository = .recycle/%U

 

[ser_rhaegar]

Can you post the errors from FreeNAS or the full message of the error that you're getting?

 

[ttblum]

Sure, 3 of the servers had failed backups last night, here is the output from the backup utility. The utility is backing up to a mapped drive that is mapped to the 'mybackups' share.

Code:

Server1 did not back up - backup directory empty, no .db or .log
==============================================================
DBBackuping Database Wed 03/25/2015  1:20:30.08
SQL Anywhere Backup Utility Version 12.0.1.3152
Database backup failed
Error writing backup file "Bad file number"
Exit Code is 1
DBBackup finished Wed 03/25/2015  1:55:41.00

Server2 did not back up - backup directory empty, no .db or .log
==============================================================
DBBackuping Database Wed 03/25/2015  0:04:36.03
SQL Anywhere Backup Utility Version 12.0.1.3152
Database backup failed
Error writing backup file "Invalid argument"
Exit Code is 1
DBBackup finished Wed 03/25/2015  1:02:48.80

Server3 did not backup - 0 size .log, no .db file
======================================================
DBBackuping Database Wed 03/25/2015  0:01:04.85
SQL Anywhere Backup Utility Version 12.0.1.3152
Database backup failed
Error writing backup file "Invalid argument"
Attempted two active database requests
Attempted two active database requests
Exit Code is 1
DBBackup finished Wed 03/25/2015  1:03:16.34

 

[dlavigne]

Did you figure this out?

 

[ttblum]

I'm suspecting that latency on the local disks of the servers might be causing the issue, not from anything with the LAN, CIFS, or FreeNAS.

It's happening on all my servers, but more so on the ones with shared local storage, which are all running intensive backups at the same time.

I've staggered the backups more to see if the errors decrease.

 

[ttblum]

There errors decreased after staggering, but only somewhat.

Now I'm wondering if the issue might be related to running out of storage space on the volume.

My volume is usually between 60% - 75% full (currently 7.5 TB out of 10.5 TB). Every night, backup scripts on the servers move the previous night's backup to a different directory, overwriting older stored backups. Then they run a backup to the NAS. In a 6 hour span, about 2.5 TB of data is overwritten, and about 2.5 TB of new data gets copied in. I wonder if ZFS is occasionally runs out of working space during this time.

This same errors were intermittently happening when I was backing up to a share on a Windows server also, there wasn't a lot of free space there either.