ZFS Replication Fails from time to time.

[mskenderian]

This usually happens less seldom but recently its happening all the time. I have noticed the snapshots that are due to expire in the next day or two fail.

Replication Data/Media/Apps -> sub.domain.net: Data/remote/garni failed: Failed: Data/Media/Apps (auto-20150620.1522-3m)
Replication Data/jails -> sub.domain.net: Data/remote/garni failed: Failed: Data/jails/syncthing_1 (auto-20150209.0121-6m)
Replication Data/Users -> sub.domain.net: Data/remote/garni failed: Failed: Data/Users/user (auto-20140909.2006-3m)
Replication Data/Media/Music -> sub.domain.net: Data/remote/garni failed: Failed: Data/Media/Music (auto-20150621.0643-3m)
Replication Data/Media/Video -> sub.domain.net: Data/remote/garni failed: Failed: Data/Media/Video (auto-20150620.1826-3m)
Replication Data/Syncthing -> sub.domain.net: Data/remote/garni failed: Failed: Data/Syncthing (auto-20150322.0121-6m)
Replication Data/Media/Audio -> sub.domain.net: Data/remote/garni failed: Failed: Data/Media/Audio (auto-20150621.1513-3m)

all of them are old snapshots, i dont know why its failing. Any idea on how i can fix this. if i leave it alone after a while they will all shows has "succeed".

 

[mskenderian]

On the remote server i just saw the Console spitting out some errors:

Sep 20 08:24:47 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:24:57 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:24:57 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:07 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:07 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:17 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:17 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:27 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:27 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:37 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:37 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:47 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:47 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:25:57 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:25:57 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:26:07 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:26:07 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:26:17 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:26:17 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:26:27 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:26:27 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory
Sep 20 08:26:37 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/.warden-template-standard--x64-20150802101348) failed: No such file or directory
Sep 20 08:26:37 server2 collectd[83461]: statvfs(/mnt/Data/remote/server1/jails/jails) failed: No such file or directory

 

[Robert Trevellyan]

What version of FreeNAS are you running?

 

[mskenderian]

FreeNAS-9.3-STABLE-201509160044

 

[Robert Trevellyan]

Unfortunately there are some issues with replication in the most recent updates (anything considered 9.3.1 as far as I know).

 

[mpfusion]

Looks like this issue has already been reported as #11550.

 

[rogerh]

Looks like this issue has already been reported as #11550.

I agree that trying to send an old snapshot and sending an email, both when the latest replication fails, is like bug 11550. That seems to be triggered, for instance, when the replicating server loses contact with the remote server. But in this case there seems to be another problem on the remote server triggering the problem when the replicated datasets can't be opened.

Edit: I wonder if it is anything to do with:

https://forums.freenas.org/index.php?threads/directories-inaccessible.38209/