deepblue1968
Cadet
- Joined
- May 16, 2017
- Messages
- 5
# This it not an issue or bug-report. More a "fundamental research" topic, out of curiosity #
Abstract:
Why is replication via Resilio Sync (Home edition) a factor x20 slower (<5 MByte/s) between 2 TrueNAS 13 servers, when (on the same network) replication between Synology DS923+ and one of these TrueNAS boxes is running at maximum Ethernet speed for a 1 GBis/s LAN? (~114 MByte/s)
Context:
Since a couple of years, I'm running quite happily a handful of FreeNAS servers for non-commercial purposes, mainly to store large amounts of photo and video files related to my semiprofessional home-studio production, video surveillance data from our home protection environment and other private data... including Apple Time Machine backups for the whole family, who except me run on Mac OS.... I use Windows since 3.11 and build my own workstations and server since 35 years, just to keep up with technology.
All servers are now upgraded to TrueNAS-13.0-U5.3, hardware is ASUS Prime Z-370 boards (32 GB RAM for main production server, 8 GB for rest. Sorry, no ECC) and all are interconnected via a single 16 port TP-Link switch 1 GB/s, with Cat 6 patch cables (all <5 meter).
Total storage capacity is >100 TB usable. Server infrastructure includes:
F2 = 9x 4 TB WD Red NAS Raid-Z3
F3 = 9x 8 TB WD Red NAS Raid-Z3
F4 = test system, currently used as a workstation
F5 = 1x 4 TB Sandisk SSD, Mini-ITX board and small chassis + small external power supply (~12W average power consumption)
F6 = 5x 4 TB WD Red NAS Stripe (!)
Additional hardware includes SATA controllers (2x Syba SI-PEX40064 each for F2 and F3) and for F3 a 10Gtek® Intel X550-2 dual port 10GbaseT network card, to attach two workstations with 10G via direct Cat 7 cable, and achieving >1GByte/s transfer speeds - so up to the physical limit (at least when reading from cache ;-)
After a steep initial learning curve, everything now works like a charm, including RSYNCing files from F5 (video surveillance data from only filer running 24x7) -> F3 and periodic Snapshotting and replication of all datasets from F3 -> F2, which is powered up only once per day for 45 minutes and shuts itself down after having received the latest snapshots from F3.
While from a functionality perspective, everything is splendid, the power consumption of the environment is quite high for a private setup. This is why I introduced F5 as a minimal configuration, with a Mini-ITX footprint and only 1 boot and 1 data SSD, consuming <12W and being powered up 24x7, to provide essential FTP and file services. The other machines are only powered up when required, i.e. to upload and process photo and video files.
[I know, that from an enterprise IT standpoint, this is a nightmare, to not leave hardware warmed up and running permanently, but so far I did not have a single server or disk failure in 5 years, so there is a business case to be had, in terms of electricity cost, especially after inflation induced price increases. Please bear in mind, that all data is replicated to F3, snapshotted there, snapshots being then replicated to F2 and all data is backed up to external USB drives at certain intervals. The biggest thread to my data is myself and my sometimes absent minded brain itself, overwriting files, which is why I run constant snapshots on F3, so I can use the versioning function in Windows 11 File Explorer]
Recently I experimented with a Synology 923+, which only consumes ~40W with 4x WD 4TB NAS. The goal was to setup a synchronous bi-directional mirroring scheme of selected datasets from the main production server F3. After a couple of days of fumbling with Syncthing, both on TrueNAS 13 and Synology DSM and solving all issues with access permissions and mounting datasets to the jails / docker containers, I finally got it working - just to find that while it did mirroring correctly, it was dead-slow. Maybe this was due to the vast amount of files or sheer data volume and Syncthing not written with that use case in mind ... but replication speed was very very slow, like ~2-3 MByte/s.
By chance, I came across Resilio Sync and installed the Home Edition on both TrueNAS (F3) and the Synology DS923+.
Installation and configuration was very straightforward - paying the same attention to permissions on the filer Main OS and mounting the required file systems correctly to the jail / docker container running the Resio / rslsync application. To my big surprise and delight, the replication proceeded at 114MByte/s - the practical limit of 1 Gbit/s LAN technology.
Inspired by this, I setup another TrueNAS server from my spare part pool: ASUS Prime Z-170 board, 8GB RAM, 5x 4TB WD Red NAS and TrueNAS-13.0-U5.3.
The configuration was done similar to the (already running and performing server F3), the same LAN cable was used. [I can hear people screaming: "How can you run without ECC RAM and use Stripe (no Raid), you are asking for trouble!" But please kindly keep in mind: The additional server is just supposed to be a "Shadow Copy" of the primary filer, where *all* data is stored in RAID-Z3, snapshoted on a Custom (* * * * *) schedule, snapshots replicated and all data backed up to external drives]
# Here comes the issue:
- When I started syncing datasets with Resilio, the replication speed was limited to ~2-3MByte/s per dataset.
- An aggregated rate of <6 MByte/s was achievable, when running 4 datasets in parallel. That is ~a factor of x20 slower than possible!
- I made sure no bandwidth limitation was set in the configuration, by explicitly setting the Advanced > Bandwidth > Global limit: to 10000000 (9.5GB/s) in Resilio Sync
In summary: Resilio Sync Home works like a charm on TrueNAS 13 from a functionality perspective, provided some fundamental settings (permissions, mount points) are configured correctly. However, the performance between 2 TrueNAS servers on literally the same network infrastructure (same LAN switch, < 3m cable each) leaves a lot to wish for.
My workaround is - now that the complete dataset replication scheme is configured and running properly - to manually *copy* all the content from F3 to the new system F6 via my workstation, through mounted SMB shares. That runs at the expected 114MByte/s - so there is *no hardware problem whatsoever* to my best knowledge.
Once that is done, I expect that the achievable speed of 2-3MByte/s is sufficient to keep a total of 12-15 TB of data in sync with Resilio, since the change rate per day is quite low. I'm not stuck. But I still would like to understand where the limitation might come from... just out of curiosity.
==> Has anybody tried this (Resilio Sync between 2 TrueNAS systems)? What were their findings? Is this due to an (if yes, then undocumented) limitation in the *free* version of Resilio Sync Home? I would be more than happy to purchase a license, but given the fact that it *did* work between TrueNAS and Synology, I seriously doubt that this would solve the issue!
Any feedback or recommendation is greatly appreciated.
Guido
Abstract:
Why is replication via Resilio Sync (Home edition) a factor x20 slower (<5 MByte/s) between 2 TrueNAS 13 servers, when (on the same network) replication between Synology DS923+ and one of these TrueNAS boxes is running at maximum Ethernet speed for a 1 GBis/s LAN? (~114 MByte/s)
Context:
Since a couple of years, I'm running quite happily a handful of FreeNAS servers for non-commercial purposes, mainly to store large amounts of photo and video files related to my semiprofessional home-studio production, video surveillance data from our home protection environment and other private data... including Apple Time Machine backups for the whole family, who except me run on Mac OS.... I use Windows since 3.11 and build my own workstations and server since 35 years, just to keep up with technology.
All servers are now upgraded to TrueNAS-13.0-U5.3, hardware is ASUS Prime Z-370 boards (32 GB RAM for main production server, 8 GB for rest. Sorry, no ECC) and all are interconnected via a single 16 port TP-Link switch 1 GB/s, with Cat 6 patch cables (all <5 meter).
Total storage capacity is >100 TB usable. Server infrastructure includes:
F2 = 9x 4 TB WD Red NAS Raid-Z3
F3 = 9x 8 TB WD Red NAS Raid-Z3
F4 = test system, currently used as a workstation
F5 = 1x 4 TB Sandisk SSD, Mini-ITX board and small chassis + small external power supply (~12W average power consumption)
F6 = 5x 4 TB WD Red NAS Stripe (!)
Additional hardware includes SATA controllers (2x Syba SI-PEX40064 each for F2 and F3) and for F3 a 10Gtek® Intel X550-2 dual port 10GbaseT network card, to attach two workstations with 10G via direct Cat 7 cable, and achieving >1GByte/s transfer speeds - so up to the physical limit (at least when reading from cache ;-)
After a steep initial learning curve, everything now works like a charm, including RSYNCing files from F5 (video surveillance data from only filer running 24x7) -> F3 and periodic Snapshotting and replication of all datasets from F3 -> F2, which is powered up only once per day for 45 minutes and shuts itself down after having received the latest snapshots from F3.
While from a functionality perspective, everything is splendid, the power consumption of the environment is quite high for a private setup. This is why I introduced F5 as a minimal configuration, with a Mini-ITX footprint and only 1 boot and 1 data SSD, consuming <12W and being powered up 24x7, to provide essential FTP and file services. The other machines are only powered up when required, i.e. to upload and process photo and video files.
[I know, that from an enterprise IT standpoint, this is a nightmare, to not leave hardware warmed up and running permanently, but so far I did not have a single server or disk failure in 5 years, so there is a business case to be had, in terms of electricity cost, especially after inflation induced price increases. Please bear in mind, that all data is replicated to F3, snapshotted there, snapshots being then replicated to F2 and all data is backed up to external USB drives at certain intervals. The biggest thread to my data is myself and my sometimes absent minded brain itself, overwriting files, which is why I run constant snapshots on F3, so I can use the versioning function in Windows 11 File Explorer]
Recently I experimented with a Synology 923+, which only consumes ~40W with 4x WD 4TB NAS. The goal was to setup a synchronous bi-directional mirroring scheme of selected datasets from the main production server F3. After a couple of days of fumbling with Syncthing, both on TrueNAS 13 and Synology DSM and solving all issues with access permissions and mounting datasets to the jails / docker containers, I finally got it working - just to find that while it did mirroring correctly, it was dead-slow. Maybe this was due to the vast amount of files or sheer data volume and Syncthing not written with that use case in mind ... but replication speed was very very slow, like ~2-3 MByte/s.
By chance, I came across Resilio Sync and installed the Home Edition on both TrueNAS (F3) and the Synology DS923+.
Installation and configuration was very straightforward - paying the same attention to permissions on the filer Main OS and mounting the required file systems correctly to the jail / docker container running the Resio / rslsync application. To my big surprise and delight, the replication proceeded at 114MByte/s - the practical limit of 1 Gbit/s LAN technology.
Inspired by this, I setup another TrueNAS server from my spare part pool: ASUS Prime Z-170 board, 8GB RAM, 5x 4TB WD Red NAS and TrueNAS-13.0-U5.3.
The configuration was done similar to the (already running and performing server F3), the same LAN cable was used. [I can hear people screaming: "How can you run without ECC RAM and use Stripe (no Raid), you are asking for trouble!" But please kindly keep in mind: The additional server is just supposed to be a "Shadow Copy" of the primary filer, where *all* data is stored in RAID-Z3, snapshoted on a Custom (* * * * *) schedule, snapshots replicated and all data backed up to external drives]
# Here comes the issue:
- When I started syncing datasets with Resilio, the replication speed was limited to ~2-3MByte/s per dataset.
- An aggregated rate of <6 MByte/s was achievable, when running 4 datasets in parallel. That is ~a factor of x20 slower than possible!
- I made sure no bandwidth limitation was set in the configuration, by explicitly setting the Advanced > Bandwidth > Global limit: to 10000000 (9.5GB/s) in Resilio Sync
In summary: Resilio Sync Home works like a charm on TrueNAS 13 from a functionality perspective, provided some fundamental settings (permissions, mount points) are configured correctly. However, the performance between 2 TrueNAS servers on literally the same network infrastructure (same LAN switch, < 3m cable each) leaves a lot to wish for.
My workaround is - now that the complete dataset replication scheme is configured and running properly - to manually *copy* all the content from F3 to the new system F6 via my workstation, through mounted SMB shares. That runs at the expected 114MByte/s - so there is *no hardware problem whatsoever* to my best knowledge.
Once that is done, I expect that the achievable speed of 2-3MByte/s is sufficient to keep a total of 12-15 TB of data in sync with Resilio, since the change rate per day is quite low. I'm not stuck. But I still would like to understand where the limitation might come from... just out of curiosity.
==> Has anybody tried this (Resilio Sync between 2 TrueNAS systems)? What were their findings? Is this due to an (if yes, then undocumented) limitation in the *free* version of Resilio Sync Home? I would be more than happy to purchase a license, but given the fact that it *did* work between TrueNAS and Synology, I seriously doubt that this would solve the issue!
Any feedback or recommendation is greatly appreciated.
Guido
Last edited:
