Running database backup via CIFS intermittently bombs out

[ttblum]

Hello,

I am using the Sybase 'dbbackup' utility to make full backups of the Sybase databases on Windows servers. I am writing these directly from the Windows servers to my FreeNAS server via CIFS. Intermittently, these dbbackups are terminating with no error message at all.

It seems like it could be possible to run the dbbackup utility the other way around, in a FreeNAS linux jail to pull the data, but I don't have the linux binary for this.

The system load on the FreeNAS server goes up to about 4-5 one-minute averages at night during backups, and I have two quad-core CPUs and 32 GB of RAM, running FreeNAS 9.2.1.5.

I seem to be getting these messages around the time some of the backups 'bomb out', but I don't know if it's a coincidence:

Code:

May  5 01:00:15 freenas1 smbd[72062]: [2014/05/05 01:00:15.161568,  0] ../lib/util/talloc_stack.c:104(talloc_pop)                                             
May  5 01:00:15 freenas1 smbd[72062]:  Freed frame ../source3/smbd/open.c:3510,
expected ../source3/modules/nfs4_acls.c:936.                                 
May  5 01:00:15 freenas1 smbd[72062]: [2014/05/05 01:00:15.254734,  0] ../lib/util/talloc_stack.c:104(talloc_pop)                                             
May  5 01:00:15 freenas1 smbd[72062]:  Freed frame ../source3/smbd/open.c:3510,
expected ../source3/modules/nfs4_acls.c:936.    

Now, looking closer at my network usage graph, one of the interfaces is showing a peak of 923.2 Mbps, which is too close for comfort on a Gigabit connection (I'm using load balancing on two interfaces). I will try staggering the backups to try to avoid overloading the network connections.

 

[cyberjock]

Why is it "too close for comfort"? I ran my server for 7 solid days at 99% capacity on Gigabit with no ill effects.

 

[ttblum]

Ok, maybe this has to do with the Freed frame error messages then?

smbd[72062]: [2014/05/05 01:00:15.161568, 0] ../lib/util/talloc_stack.c:104(talloc_pop)
smbd[72062]: Freed frame ../source3/smbd/open.c:3510, expected ../source3/modules/nfs4_acls.c:936.
smbd[72062]: [2014/05/05 01:00:15.254734, 0] ../lib/util/talloc_stack.c:104(talloc_pop)
smbd[72062]: Freed frame ../source3/smbd/open.c:3510, expected ../source3/modules/nfs4_acls.c:936.

I'm thinking of turning off and on the 'Inherit ACLs' option on the dataset as suggested here:

https://bugs.freenas.org/issues/4476

 

[ttblum]

I turned off 'Inherit ACLs' on the CIFS share and now the Freed frame error seems to have gone away, but the backups are still stopping midstream intermittently.

The backup files are being overwritten on the share each night (before they go to tape), their filesizes are anywhere between 30GB and 500GB.

I'm writing this to a RAIDZ of 3 desktop drives. The graph shows the average write I/O for each drive is about the same: 33 MB/s, maxing at about 90 MB/s.

I'm not sure what the correct settings should be on a CIFS share in Active Directory, I have:

CIFS share - Inherit ACLs - not checked
CIFS share - Inherit permissions - not checked
Dataset - Type of ACL - Windows

The servers writing the backups are running Windows 2008.
Is there some other network protocol I should be trying, such as NFS or something else?