Replication (zfs send?) more than double speed of OS transfers

[SAK]

I was testing out replication speeds between 2 TrueNAS servers. One is running Core, the other Scale. From one 8-disk z2 pool to another 8-disk z2 pool on the other box. Over a 10Gb network I was getting speeds of about 1GiB/s. While in linux virtual machines on the same physical boxes (the Core box is actually a virtualized TrueNAS in proxmox), I was getting speeds closer to 400MiB/s and even slightly under at times as well. I was really surprised at what a difference this is. The data was mostly huge media files, so should be nice sequential reads in theory.

Does that seem normal? I'm thankful for the awesome speeds with replication, but wondering if this is just normal, or I need to tweak configs to be able to bring it up.

 

[morganL]

Might be impacted by the amount of RAM allocated to ARC.... since your machines are very RAM-heavy.

Increasing SCALE ARC size

We are pleased to see that upstream OpenZFS has merged changes which will increase the default ARC sizes on Linux to match what we had on FreeBSD. This change is expected to land in the next major release of TrueNAS SCALE 24.04 (Dragonfish). In the meantime, it is possible for you to manually...

www.truenas.com

 

[SAK]

Thank you, it has been some time since I've looked into that. The SCALE machine is:

566.8GiB​

total available (ECC)

Free: 530.5 GiB

ZFS Cache: 18.5 GiB

Services: 17.9 GiB

I have seen this CORE server to use up to half of this memory for ZFS cache (that was also when I had 768 installed, but a DIMM just went bad so I had to remove a set of them). At the moment it appears to be using very little.

The Core machine (virtualized) is:

78.1GiB​

total available (ECC)

Free: 3.4 GiB

ZFS Cache: 67.2 GiB

Services: 7.4 GiB

So ARC could have something to do with slow speeds via SMB share/OS usage versus the ZFS send/receive replication?