Limit zfs send/receive speed

Sasquatch

Explorer
Joined
Nov 11, 2017
Messages
86
Is there a way to limit local zfs send/receive speeds?
Massive dataset was "accidentally" nuked by moron co-"admin" in attempt to remove SMB share...
Now I have 16 hours replication to do, but when replication runs i cannot use receiving pool for anything, it takes 2-3 minutes to list folder contents.
it's HDD I/O limited, CPU runs at 10-15% utilization.
With 10 hours non business hours it's impossible to run replication without huge work impact, If i could limit replication speed to 80-100mB/s that would leave me some headroom for other tasks.

zfs set writelimit=100mB pool/dataset

throws invalid property error.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

Sasquatch

Explorer
Joined
Nov 11, 2017
Messages
86

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I haven't used it for that, so didn't think about that part.

You may be able to fudge it by using the ssh transport even with a local target... ungainly, but maybe it will work?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Too bad you cannot run the task over a weekend, I reckon.

One dramatic improvement re: responsiveness I have found is adding a special VDEV for metadata. Setting up a sVDEV takes care, however, since losing the sVDEV will destroy the pool. So, redundancy, planning (i.e. how big), etc. is required. Plus, the pool has to be set up with the sVDEV in place before copying the data to get the full benefit. But the impact is dramatic vs. using L2ARC, even if that can be made persistent in TrueNAS 12 (in FreeNAS 11 and earlier, L2ARC has to be "reheated" after every reboot).

L2ARC can be added to a pool after the fact, is not finicky re: what SSD is used to store the data, and can be lost without any ill effect. It might help you make the receiving system more responsive during the replication task and afterwards too for directory traversals and other like browsing. The NAS has to have enough RAM for ARC, enough L2ARC capacity, and L2ARC likely should be set up for metadata only.

L2ARC made a huge time savings for my rsync backups. See here for L2ARC impacts on largely dormant data with rsync. The new TrueNAS pool with a sVDEV for metadata and small files storage appears to be even faster (it also helps that I reduced the number of small files to back up by packing most of them into Apple disk images). ZFS-send does this whole packing business on the fly and perhaps that's what is causing your slowdowns.
 
Last edited:

Sasquatch

Explorer
Joined
Nov 11, 2017
Messages
86
I would say that i3-4160 isn't best choice of CPU for encrypted pools. Simple copy to Nas over smb eats 50% CPU.

I didn't think through migration to encrypted pool.
Not sure if sVdev or l2arc will help with that.
E3-12xx v3 Seems obvious easy solution

P.s.
Slow replication over ssh worked, needed it restored asap, waiting for weekend wasn't an option.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Surprised it’s that cpu-intensive. I’ll have to monitor my CPUs one day during a big SMB transfer to see what happens, my recollection was that only one thread is active for SMB and none of them exceeded 50% consistently. That’s one reason I think I would have been better served with the D-1508 version of my motherboard rather than the D-1537 I bought since the former may have fewer cores / threads but also has a significantly higher clock speed.

The C2758 I use here as a backup server doesn't seem to choke on being on the receiving end of a ZFS send and its single-core performance is 1/3 of your CPU. Even with all cores active, the C2758 lags significantly behind the i3-4160. The sender (a D-1537) also has the single-core performance of ~1/2 of yours and only if you could include all cores does it best the i3-4160 per passmark. Using 10GbE with optical connections, the two machines transfer at about 250-300MB/s when using ZFS-send or when loaded up with multiple parallel SSH sessions. Single Z3 encrypted pool on either end.
 
Last edited:

Pandry

Cadet
Joined
Aug 5, 2021
Messages
2
Hi @Sasquatch
As far as I saw, the utility that TrueNAS (and thus zettarepl) uses is called "throttle".
I THINK you could use it via CLI by using a command like this:
zfs send tank/a@snap | throttle -M 100 | zfs receive tank2/b
 
Top