Files Invisible on Replication Computer

[HomeStorageManiac]

Hello,

I have been using FreeNAS for regular home storage for a while now. I recently decided to add a replication FreeNAS and following the instructions. It appears to be working. The source computer sends the Snapshots over to the destination. There are no errors and sometimes I even catch the source computers snapshot in the middle of transfer. The storage consumption is approximately the same on both computers....

Here is the problem. I cannot view any of the files on the destination computer. My method for checking is to open the terminal from the we interface, cd into the directory, and running ls. It comes back with nothing, as if the folder is empty. Is there something I am missing/ not doing correctly?

 

[Johnny Fartpants]

Hello,

I have been using FreeNAS for regular home storage for a while now. I recently decided to add a replication FreeNAS and following the instructions. It appears to be working. The source computer sends the Snapshots over to the destination. There are no errors and sometimes I even catch the source computers snapshot in the middle of transfer. The storage consumption is approximately the same on both computers....

Here is the problem. I cannot view any of the files on the destination computer. My method for checking is to open the terminal from the we interface, cd into the directory, and running ls. It comes back with nothing, as if the folder is empty. Is there something I am missing/ not doing correctly?

Reboot your replica system and boom!

 

[HomeStorageManiac]

Reboot your replica system and boom!

Could not have been a simpler solution I suppose. I can 'ls' through all the files and run CLI searches on it as well. Thanks!!

 

[pro lamer]

Reboot your replica system and boom!

Why did it work?

Sent from my mobile phone

 

[sretalla]

I think what happened at reboot is that the mountpoints of all the datasets are processed and mounted... I presume zfs mount -a would get us to the same result too...

 

[HugoMcKinley]

Hi,
the problem seems to exist up to FreeNAS-11.3-U1.
I rebooted the system and the files got visible, but the next replication hides the files again.
The only file I can see ist the .windows file.
I doesn't matter if it s a local replication(USB HDD) or a remote replication(Offsite Backup).
Is there any fix for this problem?

Edit: zfs mount -a doesn't help

 

[tomasin]

Hi, Just started using FreeNAS a few weeks ago so I'm complete novice to this environment. I have been experiencing the exact same issue with version FreeNAS-11.3-U3.2. I have deleted all replications, snap-shots on both systems and starting with a clean pool on the backup system several times. Each time the first replication works fine, all snapshots look good, all ACL's are replicated over and file shares are visible. When the second replication runs it looks fine, all snapshots look fine however, the shares have no ACL's and the shares are not visible. When I try to edit the ACL of a share on the replicated system I get an "[ENOENT] Path not found" error. As per above a reboot of the server fixes the issue and all ACL's and shares are visible again.

 

[MikeyG]

I have the same issue. For me I replicate a dataset recursively with one task with child datasets. Some are browseable, some are not. Doesn't seem to matter if they are set to read only or not. Maybe file a bug report?

 

[adrianwi]

This sounds a lot like my similar issue here.

 

[anodos]

Are the problematic datasets mounted? (zfs get mounted <pool/dataset>) ACL manager returning an error with ENOENT is expected if the path doesn't exist or isn't accessible.

 

[MikeyG]

@anodos In my case yes, all the datasets in question show up as mounted.

 

[anodos]

@anodos In my case yes, all the datasets in question show up as mounted.

When you're talking about some datasets being browsable and others not, are you talking about through a filesharing protocol or locally?

 

[adrianwi]

I've just checked this on my target device and the jail folder is showing as mounted.

Code:

root@freenas1:~ # zfs get mounted APEpool1/iocage/jails
NAME             PROPERTY  VALUE    SOURCE
APEpool1/iocage/jails  mounted   yes      -

The dataset appears to be empty though a SSH connection, but not via the WUI

 

[MikeyG]

When you're talking about some datasets being browsable and others not, are you talking about through a filesharing protocol or locally?

Locally:
root@freenas:/mnt/BackupPool/Mike/Files/Backup # zfs get mounted BackupPool/Mike/Files/Backup NAME PROPERTY VALUE SOURCE BackupPool/Mike/Files/Backup mounted yes - root@freenas:/mnt/BackupPool/Mike/Files/Backup # ls root@freenas:/mnt/BackupPool/Mike/Files/Backup #

 

[anodos]

Locally:
root@freenas:/mnt/BackupPool/Mike/Files/Backup # zfs get mounted BackupPool/Mike/Files/Backup NAME PROPERTY VALUE SOURCE BackupPool/Mike/Files/Backup mounted yes - root@freenas:/mnt/BackupPool/Mike/Files/Backup # ls root@freenas:/mnt/BackupPool/Mike/Files/Backup #

What about zfs list output? is that a dataset or a directory?

 

[MikeyG]

Definitely a dataset:

BackupPool/Mike/Files/Backup 311G 10.1T 212G /mnt/BackupPool/Mike/Files/Backup

 

[tomasin]

I left it alone to see if the next nights replication would be OK or if it would revert to the same issue again. Checking today we have the same issue. Suspect the issue on the backup system is somehow related to the ACL’s. After rebooting the backup system all ACL’s are good and all datasets and smb shares are all visible again. More of a detail below.

Everything on the primary system looks good and no errors.
- Primary System Snapshots all ran as normal.
- Primary System Replication Task ran as normal.
- All datasets are visible on the primary system (APOLLO)

Code:

root@APOLLO[/]#  ls /mnt/DATA_POOL
.recycle        Backups         ClipArt         eBooks          Manuals         Other           Pictures        Templates       Videos
AudioBooks      Business        Documents       Knowledgebase   Music           Photographs     Software        Users
root@APOLLO[/]#

On the secondary/backup system (HELIOS) the following.
- all snapshots match the primary system’s.
- no smart errors and zpool shows everything is online.
- verified any changed files and they all match those on primary system.
- smb service is running and only showing the .recycle folder .
- no datasets are visible on the backup system.

Code:

root@HELIOS[/]# ls /mnt/DATA_POOL
.recycle
root@HELIOS[/]#

Console Output on backup system from mid-night when the replication runs.

Code:

Jun 12 00:00:00 HELIOS syslog-ng[1153]: Configuration reload request received, reloading configuration;
Jun 12 00:00:00 HELIOS syslog-ng[1153]: Configuration reload finished;
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Backups) failed: No such file or directory
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/ClipArt) failed: No such file or directory
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Documents) failed: No such file or directory
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/AudioBooks) failed: No such file or directory
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Knowledgebase) failed: No such file or directory
Jun 12 00:00:15 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Business) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Manuals) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Software) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Photographs) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Music) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Other) failed: No such file or directory
Jun 12 00:00:25 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Pictures) failed: No such file or directory
Jun 12 00:00:35 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/eBooks) failed: No such file or directory
Jun 12 00:00:35 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Templates) failed: No such file or directory
Jun 12 00:00:35 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Users) failed: No such file or directory
Jun 12 00:00:35 HELIOS collectd[867]: statfs(/mnt/DATA_POOL/Videos) failed: No such file or directory

On any of the datasets on the backup system if you Edit ACL you get the following error:

And the ACL's on backup system dataset's are empty.

At least a reboot fixes the issue for now.

 

[warriorcookie]

This has alot of similarities to my issue: https://www.ixsystems.com/community/threads/zfs-replication-iocage-dataset-spams-console.85426/

I'm not running SMB on my backup NAS though and it's only my iocage dataset that's an issue as my other replications don't have child datasets.

 

[tomasin]

I was wondering what does FreeNAS do on a restart/reboot that fixes this issue and could that be put into a cron job to schedule each day until there is a fix?

FYI: Associated bug report here: https://jira.ixsystems.com/browse/NAS-106435