Confusing "Auto TRIM" description

[RaphM123]

I was considering enabling TRIM for my SSD Mirror pool, but the description text for "Auto TRIM" somewhat confuses me:


Enable for TrueNAS to periodically review data blocks and identify empty blocks of obsolete blocks that can be deleted. Unset to incorporate day block overwrites when a device write is started (default).

How does this translate into the technical ZFS world?

It's clear that checking the setting will enable `autotrim=on` on the respective ZFS pool. But there are arguments for preferring regular manual trims over autotrim (small performance overhead, small amounts not trimmed, see https://serverfault.com/a/1047815).

Now my actual question:
The 2nd bold part of the sentence leaves me quite puzzled - when _unset_, day block overwrites (?) happen when a device write is started (?).
What is this even supposed to mean?
The "day block overwrite" hints at something occuring daily, but as far as I can tell from the output of `zpool stats -t <NAME` shows that no manual trims have happened, so it cannot mean that.

In general, is creating a custom cron (e.g. weekly) running `zpool trim <NAME>` a reasonable solution over checking "Auto TRIM"?

 

[HoneyBadger]

My guess as to the "day block overwrite" is that it's a typo or misunderstanding, and should say "dirty block overwrite" instead - basically, dirty blocks have to be handled inline with writes by the SSD's firmware-based garbage collection.

Regarding the second part, I'll give you a two-part answer in return.

Firstly, the behaviour of certain SSDs under TRIM - especially queued TRIM, supported in OpenZFS 2.x - has been, shall we say, "less than optimal." There was a large amount of back-and-forth and finger-pointing between Samsung and the Linux kernel developers as to who was at fault for a situation that resulted in data loss under queued TRIM for certain drives - I think the 840 EVO was one of the most commonly noted. Other drives will aggressively TRIM very small sections, which results in write amplification as the drive says "oh, I can clean that 128K segment" and has to shuffle the remainder of its erase block size over to new NAND. This causes premature wear on the SSD itself.

The second piece is the performance argument. Some drives that don't support queued TRIM have to treat TRIM as a non-queued operation - so they have to flush their entire I/O queue, pause, complete the TRIM, and then resume. Erase times are usually measured in milliseconds which seems fast, but program/write times are measured in microseconds - this causes a "stall" in the drive.

So summarizing - if your drives can handle TRIM without losing data, then you can and arguably should TRIM them in some manner. If they can handle TRIM without losing performance (or at least not impacting your running workload) then you can enable pool autotrim. If it significantly impacts performance, or you want to err on the side of caution, then you can schedule the cron job for your off-hours.

 

[RaphM123]

Thanks, just as I imagined - the wording of the tooltip is just a bit poor.

So summarized, if your SSDs can _safely_ (see kernel discussion) support trim you should either:

* Enable autotrim (_can_ have a _small, continuous_ performance impact depending on model)
_or_
* Schedule a recurring manual `zfs trim` (which _will_ have some performance impact, but you can schedule it to off-hours)

 

[HoneyBadger]

Correct on the summary. Most modern SSDs will be honest about their TRIM/queued TRIM support status, so it's just a matter of determining if it causes those minor performance hiccups when it erases a page. You'll likely have better luck with "professional" oriented drives as opposed to "consumer" or "gamer" gear.

 

[winnielinnie]

I concur! Weekly trims for SSD-exclusive pools seem to be the best middle ground, while leaving "Auto TRIM" disabled.

Auto Trim for NVMe Pool

I currently have a single NVMe Pool (nvme1) for my apps / docker data. Based on my loose knowledge I know that trim feature is good for SSDs / NVMe storage. The SSD is a 512GB Samsung NVMe SSD 950 PRO partitioned as 32 GB boot / 480 GB storage. The 32GB boot pool partition is also mirrored to...

www.truenas.com

Mine runs at 3am every Sunday. (Once a week.)

When the pool receives the "zpool trim" command, you can view if it's currently in progress with zpool status -v ssdpool, or by going to Storage -> Pools -> cogwheel -> Status. You'll see the SSD drives with the "trimming" status next to them:

Code:

NAME                       STATE      READ         WRITE         CKSUM
ssdpool                    ONLINE      0            0               0
mirror-0                   ONLINE      0            0               0
gptid/UUID-XXXX-1234-5678  ONLINE      0            0               0    (trimming)
gptid/UUID-XXXX-8888-ZZZZ  ONLINE      0            0               0    (trimming)

More than likely you won't "catch it in time", since it lasts only seconds in many cases.

 

[winnielinnie]

Thanks, just as I imagined - the wording of the tooltip is just a bit poor.

Blasphemy! The tooltips in TrueNAS have always been very clear, and not once were there any glaring mistakes in them!

 

[morganL]

I concur! Weekly trims for SSD-exclusive pools seem to be the best middle ground, while leaving "Auto TRIM" disabled.

Daily, Weekly, Monthly.. would all work and depends on workload and whether there is a period where performance is a non-issue.

It should be noted that the nature of ZFS writes makes TRIM less important than in other file systems which have more smaller random writes and overwrites.

That and the SSDs have improved over the years.... there used to be some real dogs.

 

[EsTaF]

Very interesting.
If you type truenas ssd trim into google, for some reason the first topics are not links to documentation, but to "Confusing..." etc ))) Maybe that's the way it's supposed to be...
From the topic I understood that you don't need to do anything at all :)

 

[morganL]

FYI, we have found when doing performance benchmarks that autotrim is very bad on many SATA SSDs, but much better on performance NVMe SSDs. We assume that is that NVMe SSDs can have many more commands operating in parallel and have less Head-of-queue blocking.

 

[winnielinnie]

FYI, we have found when doing performance benchmarks that autotrim is very bad on many SATA SSDs

Maybe a nice feature to have is a GUI option to use either "None", "Auto" or "Weekly".

From the user's perspective, it would be a drop-down selection when navigating to the pool's options:

TRIM

[X] None​

[ ‎‎‏‏‎ ‎] Auto​

[ ‏‏‎ ‎] Weekly​

​

The default for a pool would be "None".

Choosing "Auto" would pop-up a disclaimer (such as what was found with SATA SSDs) before the user can continue.

There can be a general tooltip / description, which informs the user that selecting either "Auto" or "Weekly" should only be used on pools that are comprised of solid-state drives (SSD) that support TRIM.



On the backend, the selections would do this...

"None" sets autotrim=off for this pool and removes the generated timer for this pool.​

"Auto" sets autotrim=on for this pool and removes the generated timer for this pool.​

"Weekly" sets autotrim=off for this pool and creates a weekly timer that issues zpool trim for this pool.​

​

​

​

This provides a user-friendly GUI for making the selection, without the user having to resort to creating their own makeshift task or timer. (The reason for "Weekly" is that it strikes a nice balance, and it's also what most Linux distros use for their systemd timer if the fstrim timer is enable.)

Auto, weekly, and none seem like three good options that can cover most users without being too cluttered or confusing.

 

[morganL]

Perhaps.... the problem is that we do find the problems can be SSD vendor specific. Weekly is not a bad default solution.
If there is concensus here... lets put a suggestion ticket in.

 

[EsTaF]

It doesn't matter...
The ssd will still get killed if you use it under l2arc)))
There is nothing to stop the disk from being 100% clogged))
And it's not funny to use trim/not use...

 

[HoneyBadger]

It doesn't matter...
The ssd will still get killed if you use it under l2arc)))
There is nothing to stop the disk from being 100% clogged))
And it's not funny to use trim/not use...

The nature of L2ARC is to fill itself and "eat its own tail" so to speak, so it will always fill itself to 100% of the logical space. SSD overprovisioning can help from an endurance and steady-state performance point; if you have one, have you checked that you're getting a good hit rate from your cache device?

 

[EsTaF]

The nature of L2ARC is to fill itself and "eat its own tail" so to speak, so it will always fill itself to 100% of the logical space. SSD overprovisioning can help from an endurance and steady-state performance point; if you have one, have you checked that you're getting a good hit rate from your cache device?

The result, little to say, is simply satisfying. It is very much to my satisfaction.
There are a ton of small files in storage. Often they took a long time to recalculate when accessing them. All the brakes are a thing of the past.
It's just that the problem is that you can't give up 75% of the disk under l2arc, not all of it.