-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum has become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Replication validation
- Thread starter normanu
- Start date
droeders
Contributor
- Joined
- Mar 21, 2016
- Messages
- 179
If you have different pool layouts between the systems involved in the replication, you'll see different sizes.
If you're really paranoid, you could generate checksums for all the files and compare them between the original and the replicated version.
If you're really paranoid, you could generate checksums for all the files and compare them between the original and the replicated version.
See this difference, above original, below backup.
Thats quite a difference
Thats quite a difference
Code:
storage/NFS 420G 1.37T 409G /mnt/storage/NFS storage/backup 176K 1.37T 176K /mnt/storage/backup storage/iSCSI 1.72T 1.37T 176K /mnt/storage/iSCSI storage/iSCSI/vm-100-disk-0 15.5G 1.38T 6.69G - storage/iSCSI/vm-103-disk-0 953G 1.37T 903G - storage/iSCSI/vm-103-disk-1 218G 1.37T 217G - storage/iSCSI/vm-105-disk-0 313G 1.37T 226G - storage/iSCSI/vm-107-disk-1 98.5G 1.37T 83.8G - storage/iSCSI/vm-109-disk-0 17.8G 1.38T 6.68G - storage/iSCSI/vm-113-disk-0 149G 1.37T 149G -
Code:
External/NASBackup 1.08T 5.84T 96K /External/NASBackup External/NASBackup/NFS 211G 5.84T 205G /External/NASBackup External/NASBackup/iSCSI 895G 5.84T 88K /External/NASBackup External/NASBackup/iSCSI/vm-100-disk-0 11.0G 5.85T 3.34G - External/NASBackup/iSCSI/vm-103-disk-0 482G 5.84T 454G - External/NASBackup/iSCSI/vm-103-disk-1 110G 5.84T 109G - External/NASBackup/iSCSI/vm-105-disk-0 154G 5.84T 114G - External/NASBackup/iSCSI/vm-107-disk-1 50.2G 5.84T 42.1G - External/NASBackup/iSCSI/vm-109-disk-0 13.3G 5.85T 3.36G - External/NASBackup/iSCSI/vm-113-disk-0 74.9G 5.84T 74.9G -
droeders
Contributor
- Joined
- Mar 21, 2016
- Messages
- 179
That seems like too big of a difference to be explained by pool layout differences alone.
Can you check the properties (compression especially) for a dataset and compare them between the primary and backup? Something like:
Can you check the properties (compression especially) for a dataset and compare them between the primary and backup? Something like:
zfs get all storage/NFSzfs get all External/NASBackup/NFSCaraquetFlyer
Dabbler
- Joined
- Feb 6, 2019
- Messages
- 13
Did you setup compression in the replication?
- Select a compression algorithm to reduce the size of the data being replicated: lz4, pigz, plzip, off
Please find below the settings,
Code:
root@freenas[~]# zfs get all storage/NFS NAME PROPERTY VALUE SOURCE storage/NFS type filesystem - storage/NFS creation Tue Feb 5 13:42 2019 - storage/NFS used 424G - storage/NFS available 1.42T - storage/NFS referenced 410G - storage/NFS compressratio 1.11x - storage/NFS mounted yes - storage/NFS quota none default storage/NFS reservation none default storage/NFS recordsize 8K local storage/NFS mountpoint /mnt/storage/NFS default storage/NFS sharenfs off default storage/NFS checksum on default storage/NFS compression lz4 inherited from storage storage/NFS atime off local storage/NFS devices on default storage/NFS exec on default storage/NFS setuid on default storage/NFS readonly off default storage/NFS jailed off default storage/NFS snapdir hidden default storage/NFS aclmode discard local storage/NFS aclinherit passthrough inherited from storage storage/NFS canmount on default storage/NFS xattr off temporary storage/NFS copies 1 local storage/NFS version 5 - storage/NFS utf8only off - storage/NFS normalization none - storage/NFS casesensitivity sensitive - storage/NFS vscan off default storage/NFS nbmand off default storage/NFS sharesmb off default storage/NFS refquota none default storage/NFS refreservation none default storage/NFS primarycache all default storage/NFS secondarycache all default storage/NFS usedbysnapshots 14.3G - storage/NFS usedbydataset 410G - storage/NFS usedbychildren 0 - storage/NFS usedbyrefreservation 0 - storage/NFS logbias latency default storage/NFS dedup off default storage/NFS mlslabel - storage/NFS sync standard inherited from storage storage/NFS refcompressratio 1.09x - storage/NFS written 187M - storage/NFS logicalused 235G - storage/NFS logicalreferenced 224G - storage/NFS volmode default default storage/NFS filesystem_limit none default storage/NFS snapshot_limit none default storage/NFS filesystem_count none default storage/NFS snapshot_count none default storage/NFS redundant_metadata all default storage/NFS org.freebsd.ioc:active yes inherited from storage
Code:
root@nod2:~# zfs get all External/NASBackup/NFS NAME PROPERTY VALUE SOURCE External/NASBackup/NFS type filesystem - External/NASBackup/NFS creation Fri Feb 15 22:22 2019 - External/NASBackup/NFS used 213G - External/NASBackup/NFS available 5.87T - External/NASBackup/NFS referenced 206G - External/NASBackup/NFS compressratio 1.11x - External/NASBackup/NFS mounted no - External/NASBackup/NFS quota none default External/NASBackup/NFS reservation none default External/NASBackup/NFS recordsize 8K received External/NASBackup/NFS mountpoint /External/NASBackup/NFS default External/NASBackup/NFS sharenfs off default External/NASBackup/NFS checksum on default External/NASBackup/NFS compression on inherited from External External/NASBackup/NFS atime off received External/NASBackup/NFS devices on default External/NASBackup/NFS exec on default External/NASBackup/NFS setuid on default External/NASBackup/NFS readonly on inherited from External/NASBackup External/NASBackup/NFS zoned off default External/NASBackup/NFS snapdir hidden default External/NASBackup/NFS aclinherit restricted default External/NASBackup/NFS createtxg 401144 - External/NASBackup/NFS canmount on default External/NASBackup/NFS xattr on default External/NASBackup/NFS copies 1 received External/NASBackup/NFS version 5 - External/NASBackup/NFS utf8only off - External/NASBackup/NFS normalization none - External/NASBackup/NFS casesensitivity sensitive - External/NASBackup/NFS vscan off default External/NASBackup/NFS nbmand off default External/NASBackup/NFS sharesmb off default External/NASBackup/NFS refquota none default External/NASBackup/NFS refreservation none default External/NASBackup/NFS guid 5431305586540292837 - External/NASBackup/NFS primarycache all default External/NASBackup/NFS secondarycache all default External/NASBackup/NFS usedbysnapshots 7.47G - External/NASBackup/NFS usedbydataset 206G - External/NASBackup/NFS usedbychildren 0B - External/NASBackup/NFS usedbyrefreservation 0B - External/NASBackup/NFS logbias latency default External/NASBackup/NFS dedup off inherited from External External/NASBackup/NFS mlslabel none default External/NASBackup/NFS sync standard default External/NASBackup/NFS dnodesize legacy default External/NASBackup/NFS refcompressratio 1.09x - External/NASBackup/NFS written 0 - External/NASBackup/NFS logicalused 235G - External/NASBackup/NFS logicalreferenced 223G - External/NASBackup/NFS volmode default default External/NASBackup/NFS filesystem_limit none default External/NASBackup/NFS snapshot_limit none default External/NASBackup/NFS filesystem_count none default External/NASBackup/NFS snapshot_count none default External/NASBackup/NFS snapdev hidden default External/NASBackup/NFS acltype off default External/NASBackup/NFS context none default External/NASBackup/NFS fscontext none default External/NASBackup/NFS defcontext none default External/NASBackup/NFS rootcontext none default External/NASBackup/NFS relatime off default External/NASBackup/NFS redundant_metadata all default External/NASBackup/NFS overlay off default
droeders
Contributor
- Joined
- Mar 21, 2016
- Messages
- 179
I don't see anything obvious to explain the differences, but I'm certainly no expert. The logicalused/logicalreferenced space is essentially identical between the two, but I'm not sure how that differs from used/referenced (even after I read the zfs manpage).
Can you run the following commands and show us the output?
Code:
storage/NFS used 424G - storage/NFS referenced 410G - storage/NFS logicalused 235G - storage/NFS logicalreferenced 224G -
Code:
External/NASBackup/NFS used 213G - External/NASBackup/NFS referenced 206G - External/NASBackup/NFS logicalused 235G - External/NASBackup/NFS logicalreferenced 223G -
Can you run the following commands and show us the output?
Code:
zfs list -t all -o space storage/NFS zfs list -t all -o space External/NASBackup/NFS
Hi it seems the backup is just must smaller.
Also the the space used for the snapshots, while the replication task just says its up to date.
I also don't get any transfer errors.
Also the the space used for the snapshots, while the replication task just says its up to date.
I also don't get any transfer errors.
Code:
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD storage/NFS 1.39T 426G 15.9G 410G 0 0
Code:
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD External/NASBackup/NFS 5.85T 214G 8.46G 206G 0B 0B
I see this is a bug in FreeNAS noted here https://redmine.ixsystems.com/issues/24137
So now the question is, if it wouldn't be better to use https://github.com/jimsalterjrs/sanoid in the mean time untill it is implemented.
For the replication of the datasets and snapshots as with the current solution you are never sure if the replicated backup will work.
So in otherwords its un-usable.
Update: tried syncing with Sanoid, it syncs completely and end up with the same size image as before, huge difference to BSD.
Can it be that it counts the size x2 as it is RaidZ2?
So now the question is, if it wouldn't be better to use https://github.com/jimsalterjrs/sanoid in the mean time untill it is implemented.
For the replication of the datasets and snapshots as with the current solution you are never sure if the replicated backup will work.
So in otherwords its un-usable.
Update: tried syncing with Sanoid, it syncs completely and end up with the same size image as before, huge difference to BSD.
Can it be that it counts the size x2 as it is RaidZ2?
Last edited:
droeders
Contributor
- Joined
- Mar 21, 2016
- Messages
- 179
I don't think the bug you referenced has anything to do with these space differences unless your replications are not completing. That bug is for resuming replications that have been interrupted. Are your replication tasks showing complete without errors?
I've never used Sanoid so can't comment on it. That said, if it were me, I would prefer the FreeNAS solution and get to the bottom of these space differences.
I don't think this is a fair assessment, as there are lots of people using FreeNAS replication without issues.
Feel free to use this Sanoid tool if you're comfortable that it's generating proper backups. I would test the backups before trusting my data with any tool.
If you're still interested in pursuing these differences, can you also post the full output from:
zpool statuszpool listzfs listzfs list -o space
Last edited:
I am definitely interested, but it seems as the size is a 2x difference that it has something todo with the RaidZ type selected.
So my FreeNAS has RaidZ2 and the destination is a single zfs disc.
The space reported by zpool list
Code:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT freenas-boot 55.5G 1.02G 54.5G - - - 1% 1.00x ONLINE - storage 5.44T 3.25T 2.19T - - 12% 59% 1.00x ONLINE /mnt NAME USED AVAIL REFER MOUNTPOINT External 1.11T 5.92T 104K /External External/NASBackup 1.10T 5.92T 104K /External/NASBackup External/NASBackup/NFS 217G 5.92T 206G /External/NASBackup /NFS External/NASBackup/iSCSI 914G 5.92T 88K /External/NASBackup /iSCSI External/NASBackup/iSCSI/vm-100-disk-0 12.3G 5.92T 3.43G - External/NASBackup/iSCSI/vm-103-disk-0 506G 5.92T 454G - External/NASBackup/iSCSI/vm-103-disk-1 111G 5.92T 109G - External/NASBackup/iSCSI/vm-105-disk-0 195G 5.92T 114G - External/NASBackup/iSCSI/vm-109-disk-0 14.9G 5.93T 3.36G - External/NASBackup/iSCSI/vm-113-disk-0 74.9G 5.92T 74.9G -
5,44T is the raw space not accounting the RAIDZ2 reservation.
So my theory is now, replication is going correctly.
And space shows for example 400GB of 5.44T (also showing parity space taken)
Instead of what you would assume 200GB of 3.25T (which is the space available to you)
So above the whole NASBackup shows 1.1T taken on a single disk and 2.2T on the RAIDZ2 disks.
