TRIM takes ages to complete.

[IvoT]

Hello,

I have a pool of 8 SAS SSDs in RAIDZ2. When I issue zpool trim, it takes considerable time to finish the TRIM process. I've issued that a few months back and it took 30-40 minutes. Now it takes DAYS!
Where can I look for insight on what may be the issue?

 

[jgreco]

Would it be because you now have more stuff on your pool and the drives have to do more reshuffling to complete the TRIM?

 

[NugentS]

Hardware list as per forum rules would help. Otherwise we are just guessing

 

[IvoT]

Would it be because you now have more stuff on your pool and the drives have to do more reshuffling to complete the TRIM?

Even with more data it should not take days.

Hardware list as per forum rules would help. Otherwise we are just guessing

Sorry:
ESXi VM with 8 vCPU and 64GB of ram.
Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 flashed with IT firmware.
8xMZILT3T8HALSV3 in RAIDZ2

 

[jgreco]

Hardware list as per forum rules would help.

Also basic stuff like pool occupancy and fragmentation rates.

 

[HoneyBadger]

Even with more data it should not take days.


Sorry:
ESXi VM with 8 vCPU and 64GB of ram.
Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 flashed with IT firmware.
8x [Samsung PM1643] in RAIDZ2

Added the SSD "friendly name" model for those curious. Is the HBA passed through via IOMMU/PCIe?

In addition to capacity, how much data has been changed in the days between the last TRIM pass? TRIM is about cleaning dirty pages, so if you've done a lot of adds/deletes/overwrites, the drives might have more "laundry to wash" so to speak.

 

[joeschmuck]

So you have an ESXi VM, but what is the system hardware? Which CPU Make/Model, How much RAM do you really have? Any other details?

I'm asking because you can take a four core CPU and make an ESXi VM with 8 cores x 2 threads. I'm trying to not assume that you have a 16 core machine for example.

 

[Davvo]

Are you running the trim manually / with a cronjob or using the auto trim function?

 

[jgreco]

Also raises the question of why you are trying to TRIM. This is only advisable if it's needed. You can cause extra wear on the flash, reducing its endurance.

 

[Davvo]

You can cause extra wear on the flash, reducing its endurance.

Shouldn't trim be using wear leveling?
BTW, does anyone know how frequently the "auto trim" runs?

 

[jgreco]

Shouldn't trim be using wear leveling?

Wear leveling is like when you use a larger piece of sandpaper to sand down your block of wood. The sandpaper wears down more evenly, but is still causing degradation. It's just more evenly spread out. Think: the opposite of a USB thumb drive that lacks a flash controller.

 

[joeschmuck]

It's been a while but I know many SSDs have automatic garbage collection (TRIM) that run internal to the SSD and does not need to be commanded from an external command. I'm now curious if TrueNAS sends a command to disable automatic garbage collection. I need to research it a bit more now that I have an all NVMe NAS.

Wear leveling is like when you use a larger piece of sandpaper to sand down your block of wood. The sandpaper wears down more evenly, but is still causing degradation. It's just more evenly spread out. Think: the opposite of a USB thumb drive that lacks a flash controller.

Good analogy.

 

[Davvo]

It's been a while but I know many SSDs have automatic garbage collection (TRIM) that run internal to the SSD and does not need to be commanded from an external command. I'm now curious if TrueNAS sends a command to disable automatic garbage collection. I need to research it a bit more now that I have an all NVMe NAS.

Let us know.
Edit: this post might be a starting point.

 

[joeschmuck]

And a little research (8 minutes) after watching a few hours of TV...

Looks like all SSDs have Automatic Garbage Collection which will move some data around as needed and also delete the data that TRIM has identified as obsolete, when the system is idle. So two different things but working towards the same goal. The benefit of using the TRIM command is it makes all discarded cells ready to be written to. If TRIM is not used, when the drive needs to write data to a cell, the computer must first erase it and then it can write the data, which slows drive operations down. But does it really need to be used, especially in a Home Use environment? I could see that it may be beneficial in a high speed corporate environment. For my home system, I am leaving the default of OFF.

Here is a reference and there are many more out there, and within our own forums.

I've issued that a few months back and it took 30-40 minutes. Now it takes DAYS!

This does not sound correct. You have not provided us enough detail on your system and how it's configured. When dealing with ESXi you need to be very clear how you are passing the drives through, and that is an assumption that you are passing the drives through. For all we know, you are creating Virtual Disks which is a terrible thing to do except for the boot drive, a virtual disk here works fine. If this is what you did, that may explain your issues.

 

[IvoT]

Also basic stuff like pool occupancy and fragmentation rates.

How to get fragmentation rates? Pool size is 19.73 TiB of which 10.58 TiB is free space. 2x 2TB iSCSI LUNs + 5TB NFS/CIFS.
HBA is hardware passthrough with DirectPath IO to the VM no virtual disks - the VM sees the actual hardware (this has not changed since the last TRIM). ESXi has 10CPU/20HT and 128GB RAM. TRIM is issued manually for no real reason, just decided to do some housekeeping.

The thing that bothers me is that when I monitor via iostat this is what I get:
https://codeshare.io/PdvjXw
So it is trimming, but very slow, like it is rate limited or something. I don't see any slowdown or whatever on the frontend. I deceded to post only because it was strange to me.
This is what it has trimmed for 12 hours:

pool: SAS
state: ONLINE
scan: scrub repaired 0B in 01:15:01 with 0 errors on Sun Nov 19 01:15:02 2023
config:

NAME STATE READ WRITE CKSUM
SAS ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
547ba0f9-ba44-4f01-ad33-4bda61310bdd ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
ae6f3e47-7d8e-45ff-b56c-0907c47b83bf ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
555672f7-1f29-451d-8fad-820612d19d05 ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
fba71670-e645-4906-818b-20238c8f97fd ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
5e41b921-b752-490d-bf6d-b79ecb9b1c22 ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
d1b7284c-ab83-498c-bad3-72ea74a7f6c8 ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
03efaaf3-eb40-492c-8f16-87c2b88f2aed ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)
b26c8881-71ed-40b7-95c4-e45df9b4d9e1 ONLINE 0 0 0 (15% trimmed, started at Tue Dec 19 12:50:07 2023)

errors: No known data errors

 

[IvoT]

Another thing that bothers me is that I have 1 more of theese SSDs which in linux (fstrim) trims with 400-500mb/s.

 

[joeschmuck]

Your Scrub looks to be fast, 1 hour 15 minutes for approx 9TB. My 7TB takes 8h 33m but it is spinning rust.


What else do you have going on in TrueNAS? Look at the disk activity, take a screen capture and post it. I'm curious if there is a lot of drive activity causing the delay. You may need to turn off some of your services in order to narrow down the issue. I'm currently out of ideas.

 

[winnielinnie]

How to get fragmentation rates?

Code:

zpool list -o name,size,allocated,free,capacity,fragmentation <name-of-pool>

 

[IvoT]

Your Scrub looks to be fast, 1 hour 15 minutes for approx 9TB. My 7TB takes 8h 33m but it is spinning rust.


What else do you have going on in TrueNAS? Look at the disk activity, take a screen capture and post it. I'm curious if there is a lot of drive activity causing the delay. You may need to turn off some of your services in order to narrow down the issue. I'm currently out of ideas.

Where to get disk activity? I've already sent a link with output "iostat -mxz 1" and there is no serious disk activity. I've also tried with everything running on truenas stopped - iSCSI/NFS/SMB and it is the same.

Code:

zpool list -o name,size,allocated,free,capacity,fragmentation <name-of-pool>

Here it is:
NAME SIZE ALLOC FREE CAP FRAG
SAS 27.9T 12.9T 15.0T 46% 31%

 

[Davvo]

Where to get disk activity?

Reporting tab in the WebUI.

NAME SIZE ALLOC FREE CAP FRAG
SAS 27.9T 12.9T 15.0T 46% 31%

Looks good.

Just FIY, the codeshare link doesn't load for me. Might be a temporary issue, but it's usually suggested to either post the output in [CODE][/CODE] brackets or to upload a .txt using the forum.