Curious - Fluctuations in Throughput During Replication?

[Phase]

Hi just curious. I'm replicating a 35 TiB pool from one server to another and I see the pattern below. What are these gradual dips with restorative pickups?

Each 6-hour block corresponds roughly to 11.3 TiB transferred.

Is it GC? Is it from the OS, ZFS, Replication Engine? Thoughts?

These are the disk on the target, source and network on the source

 

[Samuel Tai]

This looks like classic TCP sawtooth behavior. Try changing the TCP congestion control algorithm on both sides of the replication.

TCP congestion control algorithm selection in TrueNAS Core/FreeNAS

Networking-minded TrueNAS Core/FreeNAS admins may need to change the TCP congestion control algorithm for their specific installations from the default TCP new Reno to achieve the best performance. (For reference, see http://intronetworks.cs.luc.edu/current/html/newtcps.html.) There are several...

www.truenas.com

 

[Phase]

That is an interesting thought. However, the network bandwidth is around 27 Gib/s as tested with multiple threads of iperf3 between these 2 machines, probably being limited by 4x PCIe 3.0 sockets. The replication is consuming only 6 Gib/s and that seems to max out the target disks -- all target disks as on the same controller.

Would the TCP sawtooth behavior still be present if the network is being underutilized? (this is a card-to-card connection, no routers or switches)

 

[Samuel Tai]

TCP is notorious for going into sawtooth far short of the full bandwidth of the network.
If this is a direct link, you should also enable jumbo frames to help mitigate the onset of sawtooth.

 

[Phase]

Cool. Thanks!