Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

ZFS Replication iocage dataset spams console

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
I have 4 replication tasks setup to replicate some datasets from one freenas server to another

3 of the tasks complete without issue, but the 4th is for my iocage dataset and it spams my console with a number of errors:

Code:
Jun 8 23:00:29 backupnas collectd[1574]: statfs(/mnt/backupnas/Data1/iocage/jails) failed: No such file or directory
Jun 8 23:00:49 backupnas collectd[1574]: statfs(/mnt/backupnas/Data1/iocage/log) failed: No such file or directory
Jun 8 23:00:49 backupnas collectd[1574]: statfs(/mnt/backupnas/Data1/iocage/releases) failed: No such file or directory
Jun 8 23:00:59 backupnas collectd[1574]: statfs(/mnt/backupnas/Data1/iocage/templates) failed: No such file or directory


-Recursive is enabled
-Enabling or disabling "Replicate from scratch if incremental is not possible" makes no difference
-If I delete the dataset on the backup server then manually run the backup task it completes without error and does not spam the console. But the next scheduled run generates the same 4 errors.
-I can see the snapshots on the backup server gui, but ssh'ing in and navigating to the /mnt/backupnas/Data1/iocage/ directory shows defaults.json, defaults_backup.json and defaults_rc.json, no other files or directories.
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
any ideas?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
2,695
Are the jails active during the replication cycle?
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
2,695
Try shutting the jails down. This is probably something like needing to quiesce VMs before snapshotting them.
 

adrianwi

Neophyte Sage
Joined
Oct 15, 2013
Messages
1,178
I'm seeing a similar issue with iocage replication, as described here

Are you sure they are completing correctly? I'm not seeing the iocage folders when I SSH into my target machine, although they are showing as there in the WUI. Whenever they change replication, it tends to be a mess for a few releases afterwards :(

ps: the jails don't need to be stopped as the replication is moving snapshots already taken
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
Stopping the jails actually generated more errors at the console.

I'm seeing a similar issue with iocage replication, as described here

Are you sure they are completing correctly? I'm not seeing the iocage folders when I SSH into my target machine, although they are showing as there in the WUI. Whenever they change replication, it tends to be a mess for a few releases afterwards :(

ps: the jails don't need to be stopped as the replication is moving snapshots already taken
The sending FreeNAS box generates no errors and I can see the snapshots listed on the receiving "Backup" FreeNAS. The errors are on the console of the receiving box only and the directory is empty except those .json files.

It is strange because the first replication always completes successfully. If I delete the dataset and manually run the replication task it completes without error. It's all my subsequent nightlys that fail.
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48

adrianwi

Neophyte Sage
Joined
Oct 15, 2013
Messages
1,178
Yes, I think the replication is working but it's hiding nested datasets/zvols on the target machine.

Hopefully something they can fix for U4
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
It seems to be working, but yes it's hiden until I reboot the backup server. Then I can browse until the next replication...

Really frustrating as ZFS send is a staple backup tool for FreeNAS. When I have more time I should create a new dataset to test restoring a backup from a child dataset and make sure it works. I do offsite nightlies too using the cloud backup feature so I'm not super worried.
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
FYI, bug is marked resolved and seems the fix will be pushed in the next update.
 

warriorcookie

Member
Joined
Apr 17, 2017
Messages
48
11.3-U4.1 seems to have resolved the issue.
 

adrianwi

Neophyte Sage
Joined
Oct 15, 2013
Messages
1,178
Yes, it resolved this for me.
 
Top