Compression performance impact and measurement

[CJRoss]

I have a dataset for my ebooks. They don't change and they're mostly text so I currently have compression set to zstd-7. I've looked at using zstd-19 but I'm concerned about the cpu usage when compressing.

I've noticed that when I move a bunch of ebooks from a different pool(or even a different dataset on the same pool) to my ebook dataset that the mv command will complete and yet the cpu on the Truenas will continue to be pegged for some time afterward.

Do I need to turn on sync in order for the compression to be taken into account when moving data into the dataset? Is there an easy way to do it just for the one command or session?

Additionally, is there a way to limit the cpu impact of the compression? I'm okay with the moves taking longer if I can use my cpu for other things in the meantime.

Thanks.

 

[artlessknave]

1. forum rules.
2. sync does not apply
3. a pegged CPU doesn't mean much when we dont know WHAT CPU. see #1. if you have an atom d525, its going to barely be able to run the webUI at all, while an epyc CPU being pegged merely from compression would be extremely concerning.
4. zstd uses resources when compression but is designed to use much less when decompressing. you usually read ebooks more than write them, so its generally not relevent. if you are truly concerned, use the (still recomeneded) lz4 compression, which is fast both ways

 

[CJRoss]

1. forum rules.
2. sync does not apply
3. a pegged CPU doesn't mean much when we dont know WHAT CPU. see #1. if you have an atom d525, its going to barely be able to run the webUI at all, while an epyc CPU being pegged merely from compression would be extremely concerning.
4. zstd uses resources when compression but is designed to use much less when decompressing. you usually read ebooks more than write them, so its generally not relevent. if you are truly concerned, use the (still recomeneded) lz4 compression, which is fast both ways

Yes, I am aware of how zstd works and that is why I want to use it instead of lz4. My concern is not what happens when I access one ebook. It's what happens when I dump a thousand into the dataset at once. Additionally, I use zstd for other data. ebooks is just the current example of what I'm working on.

My issue is not how much of the CPU is being used. My issue is twofold.

1. How do I get the mv/cp command to block until the compression is finished?
2. How can I lower the priority of the compression so that it doesn't impact other tasks?

You stated that sync will not work to fix the first issue. What will?

 

[winnielinnie]

1. How do I get the mv/cp command to block until the compression is finished?

As far as I'm aware, inline compression is integral with writes using ZFS. If you are concerned with performance issues, and hence want a user-application (such as mv or cp to "wait for ZFS to finish"), then you might as well use a faster compression method, such as LZ4 stated above. That same principle would apply to ZFS native encryption. The user applications are agnostic to what ZFS does under-the-hood.

2. How can I lower the priority of the compression so that it doesn't impact other tasks?

I think there's a tuneable to change the number of CPUs / cores that can be used for checksums and compression, but this may in fact decrease overall performance if you lower it below the default of 75%. (It appears to limit the concurrent records being processed per CPU thread; one record per thread. So if your CPU supports 8 total threads, and you limit this to 50%, then it means it will concurrently process no more than 4 records at a time, rather than the default of 6 records at a time, since 75% is the default.)

EDIT: I might have done the math wrong or mis-read the purpose of this variable. Goodness when developers and engineers write out documentation, it makes you cross your eyes.


zio_taskq_batch_pct

Module Parameters — OpenZFS documentation

openzfs.github.io

I'm concerned about the cpu usage when compressing.

The ZFS developers considered this concern ahead of time, hence it's limited to 75% of total CPU threads, rather than 100%. Before you try to tweak anything, why not just test it out? Once records are written, you're good to go, even if the process was slow. You can always change a dataset's compression property at any time, and resort to a tuenable later on if it's still needed.

Rather than speculate, what matters in the end is your personal experience with your system and data.


EDIT 2: Apparently the default was changed to 80%, yet it will still round down to the next integer.

 

[Volts]

Easy enough to test - temporarily toggle sync on.

But I believe compression happens during transaction group creation, so I don’t think sync will affect compression or how quickly mv and cp return. And obviously sync will have other significant performance implications.

If the files are epub, they may already be zip compressed. I’ve unzipped, used better compression (such as imageOptim) on the images, and re-zipped them with higher compression.

I think changing ZFS compression is generally a waste of time. It’s fast enough to be invisible. Leave it enabled with default settings. Pretend it isn’t there at all. The difference between 5% additional compression on already-compressed files, vs. 10%, isn’t worth a change.

 

[Volts]

this may in fact decrease overall performance

Especially other tasks that also need filesystem IO!

… forcing trucks to drive slowly doesn’t help other cars go faster.

I’ve never explored that setting, however. Off to learn more about it!

 

[winnielinnie]

I think changing ZFS compression is generally a waste of time. It’s fast enough to be invisible. Leave it enabled with default settings. Pretend it isn’t there at all. The difference between 5% additional compression on already-compressed files, vs. 10%, isn’t worth a change.

LZ4 has "early abort", which addresses the issue of wasting CPU cycles to "compress" records already highly compressed and/or incompressible.

Last I read, ZSTD will also use "early abort", if it hasn't been implemented already.

From April 2020:

What’s Next?
The integration of Zstd into ZFS has just begun and the future undoubtedly holds many improvements. Already, we have thoughts along these lines. For example, we expect using the advanced Zstd API to provide more hints about the maximum size of the input data could reduce memory usage and improve Zstd’s ability to take advantage of “early abort,” which we spoke of early in the article. There are likely a number of opportunities to optimize the way ZFS sets up and tears down Zstd compression contexts and to increase the reuse of these contexts with the Zstd reset API, which one would expect to significantly improve compression performance with small blocks.

 

[winnielinnie]

I’ve never explored that setting, however. Off to learn more about it!

I tend to play it safe and leave everything at the defaults, unless I have an explicit need to address a dire problem or concern.

 

[WI_Hedgehog]

Additionally, is there a way to limit the cpu impact of the compression? I'm okay with the moves taking longer if I can use my cpu for other things in the meantime.

rsync --bwlimit
(or cwrsync, depending)

Tune the bandwidth to your desired outcome.

 

[CJRoss]

Interesting. I wasn't aware of the 75-80% default or tuneable. Unfortunately, I don't want to tweak a global setting just for something I do occasionally. Not sure if it would take affect immediately but it's probably too much hassle to change it every time I load files.

I normally use lz4 for most of my things, but I have several WORM datasets where I prefer to save every last byte I can as I won't generally be writing to it. The ebooks is just one example. Additionally, I generally like to preserve original files as I've been bitten several times by modifying the files themselves to save space.

I had actually considered using the bandwidth limit of rsync but never tried it out. I think I was concerned about the scan times and io that happen before it can start moving. May have to give it another go.

Thanks all.

 

[Volts]

“I wish I had spent more time micro-optimizing my ebook hoarding storage compression settings.”

I don’t think that tunable would be a great idea anyway. It might leave more system CPU available for non-IO tasks, but it would slow down ALL IO operations. You probably don’t want that.

Rsync is probably easier to use in most cases, but check this out too:

https://www.truenas.com/community/threads/limit-zfs-send-receive-speed.93165/post-654121

Could also do tar | throttle | tar. (But at that point rsync has advantages.)