Replication issues -- replications automatically deleting themselves...

[SnakeByte]

Something strange is going on with the mirroring of our data. First some info about the setup:

Here's our replication task:

Here's a screenshot of the source system:

and here's a screenshot of the destination server after successful transfer:

It looks like "MM" is mirroring just fine, with some small differences that I assume are related to our live system receiving file changes as the snapshot replica transfers.

However, when I ssh into the destination and manually verify that the files are there, I'm not seeing anything:

My original reason for creating this post was to ask if there was some additional step I needed to perform to reveal the files, however, what's more troubling is that as I was writing here all of the data on the destination vanished, and it looks like the replication is restarting from scratch:

Both FreeNAS servers are showing the friendly green "OK" in the GUI and I'm not getting emailed about any errors. Anyone know what could be going on here?

 

[SnakeByte]

Forgot the screenshot of the source server's Periodic Snapshot Tasks:

 

[Robert Trevellyan]

when I ssh into the destination and manually verify that the files are there, I'm not seeing anything:

I've seen some threads about datasets not being automatically mounted on the replication target.

 

[SnakeByte]

Wow, there are a lot of them, none with solutions...

I've tried deleting and recreating as was suggested in the other threads, but with the same results. The destination server shows disk space being used up, but no data is shown using "ls."

The docs are pretty straight forward-- create a replication task, point it to another freenas server, watch the replication complete, but no files to see once it's done.

At least now nothing is getting wiped in front of my eyes... that was weird.

Here's the info on the destination server ... I made TWO different replication tasks, one drops to trunk/Backup, the other to trunk, neither show files.

Code:

[root@freenas-2] ~# zfs list
NAME														 USED  AVAIL  REFER  MOUNTPOINT
trunk/Backup												1.29T  7.63T   128K  /mnt/trunk/Backup
trunk/Backup/MM											 1.29T  7.63T   140K  /mnt/trunk/Backup/MM
trunk/Backup/MM/applied									  992G  7.63T   499G  /mnt/trunk/Backup/MM/applied
trunk/Backup/MM/deploy									  10.5G  7.63T  2.58G  /mnt/trunk/Backup/MM/deploy
trunk/Backup/MM/home										 151G  7.63T  35.5G  /mnt/trunk/Backup/MM/home
trunk/Backup/MM/profiles									70.3G  7.63T  19.9G  /mnt/trunk/Backup/MM/profiles
trunk/Backup/MM/shared									  92.6G  7.63T  49.5G  /mnt/trunk/Backup/MM/shared
trunk/MM													1.29T  7.63T   140K  /mnt/trunk/MM
trunk/MM/applied											 994G  7.63T   499G  /mnt/trunk/MM/applied
trunk/MM/deploy											 10.5G  7.63T  2.58G  /mnt/trunk/MM/deploy
trunk/MM/home												152G  7.63T  35.9G  /mnt/trunk/MM/home
trunk/MM/profiles										   71.0G  7.63T  20.1G  /mnt/trunk/MM/profiles
trunk/MM/shared											 92.9G  7.63T  49.6G  /mnt/trunk/MM/shared

Code:

[root@freenas-2] ~# mount
trunk/Backup on /mnt/trunk/Backup (zfs, local, nfsv4acls)
trunk/Backup/MM on /mnt/trunk/Backup/MM (zfs, local, read-only, nfsv4acls)
trunk/Backup/MM/applied on /mnt/trunk/Backup/MM/applied (zfs, local, read-only, nfsv4acls)
trunk/Backup/MM/deploy on /mnt/trunk/Backup/MM/deploy (zfs, local, read-only, nfsv4acls)
trunk/Backup/MM/home on /mnt/trunk/Backup/MM/home (zfs, local, read-only, nfsv4acls)
trunk/Backup/MM/profiles on /mnt/trunk/Backup/MM/profiles (zfs, local, read-only, nfsv4acls)
trunk/Backup/MM/shared on /mnt/trunk/Backup/MM/shared (zfs, local, read-only, nfsv4acls)
trunk/MM on /mnt/trunk/MM (zfs, local, read-only, nfsv4acls)
trunk/MM/applied on /mnt/trunk/MM/applied (zfs, local, read-only, nfsv4acls)
trunk/MM/deploy on /mnt/trunk/MM/deploy (zfs, local, read-only, nfsv4acls)
trunk/MM/home on /mnt/trunk/MM/home (zfs, local, read-only, nfsv4acls)
trunk/MM/profiles on /mnt/trunk/MM/profiles (zfs, local, read-only, nfsv4acls)
trunk/MM/shared on /mnt/trunk/MM/shared (zfs, local, read-only, nfsv4acls)

Code:

[root@freenas-2] ~# cd /mnt/trunk/MM
[root@freenas-2] /mnt/trunk/MM# ls
./		../	   applied/  deploy/   home/	 profiles/ shared/
[root@freenas-2] /mnt/trunk/MM# cd applied
[root@freenas-2] /mnt/trunk/MM/applied# ls
./  ../
[root@freenas-2] /mnt/trunk/MM/applied# ls -la
total 13
drwxrwxr-x+ 2 root   20513  2 Apr 11 21:51 ./
drwxrwxr-x+ 7 28136  20513  7 Apr 11 21:51 ../

What am I missing?

 

[Robert Trevellyan]

Good question. What does ls -la MM/applied show on the source? What about du -h?

 

[SnakeByte]

Rebooting the destination was a suggestion in the other threads too, one I tried without successful results (It wasn't successful for that thread either).

Is there documentation that explains how to manually mount? One of the things I tried was to clone a replicated snapshot, but again, the contents were empty.

ls -la on the source shows all of the files.. here's a little cut:

Code:

[root@freenas] /mnt/trunk/MM/applied# ls -la
total 1730
d---------+ 165 MM\domain admins  MM\domain users	178 Apr 27 17:05 ./
drwxrwxr-x+   7 MM\ibrown		 MMIA\domain users	  7 Apr 11 21:51 ../
-rwxrwxr-x+   1 MM\administrator  MM\domain users	  0 Jul 27  2015 .windows*
d---------+   8 MM\domain admins  MM\domain users	  8 Aug 29  2003 03/
-rwxrwxr-x+   1 MM\administrator  MM\domain users	  9 Mar  1  2015 1A50004F-2074-4fce-8D48-06CDE7C23618.cpt*
d---------+   2 MM\domain admins  MM\domain users	512 Apr 26  2013 2LDL/
d---------+   5 MM\domain admins  MM\domain users	  5 May 11  2006 @vantage Archive/
d---------+   2 MM\domain admins  MM\domain users	286 Apr  8  2015 ACTNIPF/
d---------+   4 MM\domain admins  MM\domain users	 99 Apr  8  2015 ALPHALAN/
d---------+   2 MM\domain admins  MM\domain users	 36 Mar  1  2015 APBATCH/
d---------+  12 MM\domain admins  MM\domain users   1244 Apr 28 13:48 APPS/
d---------+   4 MM\domain admins  MM\domain users	 47 Sep 12  2005 ARROWH/
d---------+   2 MM\domain admins  MM\domain users	  6 Apr 15  2003 ASFAX/

du -h would take quite some time to complete on the source, but it was running away when I ran it.

Code:

[root@freenas] /mnt/trunk/MM/applied# du -h
12M	./PSQ5
512B	./PML/ILDATA
1.0K	./PML
1.7M	./adbanker/xtras
14M	./adbanker
512B	./PANV7/FORMCREA.TED
52K	./PANV7/FORM/8FORMS
136K	./PANV7/FORM/CCL
123K	./PANV7/FORM/SLIDES
320K	./PANV7/FORM/6BUSPLAN
157K	./PANV7/FORM/5TAX
643K	./PANV7/FORM/2INCOME
974K	./PANV7/FORM/7EXECBEN
435K	./PANV7/FORM/1BASIC
119K	./PANV7/FORM/3TERM
584K	./PANV7/FORM/DATA
512B	./PANV7/FORM/GRF
311K	./PANV7/FORM/PRESENT
55K	./PANV7/FORM/4INVEST
4.4M	./PANV7/FORM
8.8M	./PANV7

 

[SnakeByte]

In case this matters, both FreeNAS servers are iXsystems mini's with identical hardware (including drives). The only two differences are that the destination is not joined to the domain, and is not using volume encryption.

 

[Robert Trevellyan]

Might be time for a bug report.

 

[SnakeByte]

Filed on 4/29:
https://bugs.freenas.org/issues/23674

I put in a link to this thread, but maybe I should copy and paste to make it easier for them.