Auto Trim for NVMe Pool

[pixelwave]

I currently have a single NVMe Pool (nvme1) for my apps / docker data. Based on my loose knowledge I know that trim feature is good for SSDs / NVMe storage.

The SSD is a 512GB Samsung NVMe SSD 950 PRO partitioned as 32 GB boot / 480 GB storage.

The 32GB boot pool partition is also mirrored to an additional 60GB SSD.

So should I enable "Auto-Trim" for that NVMe (nvme1) storage pool?

 

[morganL]

Autotrim isn't enabled by default because we find that for many SSDs it actually makes ZFS performance worse and we haven't found many cases where it significantly improves anything.

ZFS is not like most file systems... data is aggregated before it is written to the drives. The SSDs don't wear out as fast as would be expected. The SSD performance is better because there are fewer random writes.

Autotrim ends up with more operations being issued to each SSD. The extra TRIM operations are not free... they are like writes of all zeros. The SSDs do "housekeeping" to free up the space and that housekeeping involves its own flash write operations.

 

[pixelwave]

… so I better leave it off then?

 

[morganL]

… so I better leave it off then?

Yes, Its one of those things that would need to be tested with your specific SSDs and with your specific workload.
It's unlikely to help, but we don't mind anyone testing.
We just don't recommend turning it on for important pools, without testing. (CYA is a reasonable accusation)
Unfortunately, testing these things can take weeks.

 

[winnielinnie]

I use an alternative method. With a weekly Cron Task, the "zpool trim" command is issued only to my pool comprised of two SSDs:

zpool trim ssdpool

It only runs once a week.

EDIT: To be clear, I have "Auto Trim" disabled on all of my pools, while I have a weekly Cron Task that issues "zpool trim" on only a very specific pool (comprised solely of SSDs.)

 

[morganL]

I use an alternative method. With a weekly Cron Task, the "zpool trim" command is issued only to my pool comprised of two SSDs:

zpool trim ssdpool

It only runs once a week.

EDIT: To be clear, I have "Auto Trim" disabled on all of my pools, while I have a weekly Cron Task that issues "zpool trim" on only a very specific pool (comprised solely of SSDs.)

If your workload has a weekly "quiet" period, this makes sense. It reduces the extra TRIM workload, but takes advantage of any large deletions of data.

 

[pixelwave]

I use an alternative method. With a weekly Cron Task, the "zpool trim" command is issued only to my pool comprised of two SSDs:

zpool trim ssdpool

It only runs once a week.

EDIT: To be clear, I have "Auto Trim" disabled on all of my pools, while I have a weekly Cron Task that issues "zpool trim" on only a very specific pool (comprised solely of SSDs.)

thanks ... I might try that instead!

 

[winnielinnie]

thanks ... I might try that instead!

Mine runs at 3am every Sunday. (Once a week.)

When the pool receives the "zpool trim" command, you can view if it's currently in progress with zpool status -v ssdpool, or by going to Storage -> Pools -> cogwheel -> Status. You'll see the SSD drives with the "trimming" status next to them:

Code:

NAME                       STATE      READ         WRITE         CKSUM
ssdpool                    ONLINE      0            0               0
mirror-0                   ONLINE      0            0               0
gptid/UUID-XXXX-1234-5678  ONLINE      0            0               0    (trimming)
gptid/UUID-XXXX-8888-ZZZZ  ONLINE      0            0               0    (trimming)

I believe when a pool receives the "zpool trim" command, only the drives that support trims will be targeted, while any non-trimmable drives (such as HDDs) will ignore it. I cannot test this for sure, since my pools are either "only SSDs" or "only HDDs."

The trim process usually lasts less than a minute; sometimes completing within seconds.