Replication (zfs send?) more than double speed of OS transfers

SAK

Dabbler
Joined
Dec 9, 2022
Messages
20
I was testing out replication speeds between 2 TrueNAS servers. One is running Core, the other Scale. From one 8-disk z2 pool to another 8-disk z2 pool on the other box. Over a 10Gb network I was getting speeds of about 1GiB/s. While in linux virtual machines on the same physical boxes (the Core box is actually a virtualized TrueNAS in proxmox), I was getting speeds closer to 400MiB/s and even slightly under at times as well. I was really surprised at what a difference this is. The data was mostly huge media files, so should be nice sequential reads in theory.

Does that seem normal? I'm thankful for the awesome speeds with replication, but wondering if this is just normal, or I need to tweak configs to be able to bring it up.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Might be impacted by the amount of RAM allocated to ARC.... since your machines are very RAM-heavy.

 

SAK

Dabbler
Joined
Dec 9, 2022
Messages
20
Thank you, it has been some time since I've looked into that. The SCALE machine is:

566.8GiB​

total available (ECC)

Free: 530.5 GiB

ZFS Cache: 18.5 GiB

Services: 17.9 GiB

I have seen this CORE server to use up to half of this memory for ZFS cache (that was also when I had 768 installed, but a DIMM just went bad so I had to remove a set of them). At the moment it appears to be using very little.

The Core machine (virtualized) is:

78.1GiB​

total available (ECC)

Free: 3.4 GiB

ZFS Cache: 67.2 GiB

Services: 7.4 GiB

So ARC could have something to do with slow speeds via SMB share/OS usage versus the ZFS send/receive replication?
 
Top