Device is causing slow io on SSD drive?

[da_da]

Every once in a while my jail (a mirror drive SSDs) sends out the slow IO drive message. What should I do in this instance?

Code:

zpool iostat -v
                                                  capacity     operations     bandwidth
pool                                            alloc   free   read  write   read  write
----------------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                                    15.3G   201G      0      0  25.4K    350
  nvd0p2                                        15.3G   201G      0      0  25.4K    350
----------------------------------------------  -----  -----  -----  -----  -----  -----
jails                                           46.0G   882G      0     51  8.86K  1.07M
  mirror-0                                      46.0G   882G      0     51  8.86K  1.07M
    gptid/4b2b1e1a-73e7-11eb-b9c3-3cecef0d9cde      -      -      0     25  4.22K   549K
    gptid/4be677aa-73e7-11eb-b9c3-3cecef0d9cde      -      -      0     25  4.63K   549K
----------------------------------------------  -----  -----  -----  -----  -----  -----

 

[da_da]

Are there ways to diagnose the SSD pool?

 

[Samuel Tai]

More details about your setup, please, per the Forum Rules.

 

[da_da]

This is run on IX Systems TrueNAS Mini XL+ 64Gig RAM.

 

[Patrick M. Hausen]

zpool get autotrim jails, please

 

[da_da]

Code:

NAME      PROPERTY  VALUE     SOURCE
jails     autotrim  off       default

I enabled AutoTRIM

Code:

NAME      PROPERTY  VALUE     SOURCE
jails     autotrim  on        local

Would this fix it?

 

[Samuel Tai]

That will enable autotrim going forward. You may also want to run zpool trim jails to kick off a TRIM now.

 

[Patrick M. Hausen]

TRIM can be a cause of seemingly random "freezes" if the SSD firmware is buggy/non-optimal. The absence of TRIM commands on a pool that sparely filled should not have any effect like this. I suspected autotrim to be enabled. I doubt enabling it will fix your problem. But with a reliable SSD it's a good idea to have it enabled, anyway.

You could leave a terminal open with zpool iostat running and try to spot something.

Did you check if there are pending firmware updates by your SSD manufacturer? In my experience at least Intel, Samsung and Crucial/Micron publish these regularly and they do fix problems.

 

[da_da]

These are two Samsung SSD 870 QVO 1TB. how do I check what firmware they are on in TrueNAS? Also, not sure if there's a way to update these on the TrueNAS systems...

Tool & Software Download | Samsung Semiconductor Global

Download various software related to your product, including Samsung Magician, designed for use with Samsung's memory products, and find related information.

semiconductor.samsung.com

 

[Patrick M. Hausen]

smartctl -a /dev/<device> will tell you the firmware version, among a ton of other things.

On that very page you linked there's a download link for a bootable ISO image:
https://semiconductor.samsung.com/resources/software-resources/Samsung_SSD_870_QVO_SVQ02B6Q_Win.iso

Don't get confused by the "Win" in the filename. It contains a bootable Linux - I checked.

 

[da_da]

so, put it on a flash drive and boot to the flash on the NAS?

 

[Patrick M. Hausen]

Of course

Also QVO drives might not be the best choice for your use case. The QLC flash has got a sustained write rate of about 80 MB/s (anandtech benchmark via search engine). Therefore there's an SLC cache on the device. Possibly you are experiencing stalls whenever that fills up. Firmware update might make the drive handle writes a bit better. Then again it might not. Can't argue with physics.

I am quite satisfied with 970 EVO Plus (TLC flash) for home use.

 

[da_da]

I ran the Multi-Report v1.6f dtd:2022-12-27 (TrueNAS Core 13.0-U3.1)
and it does not contain all the smartctl info for the drives...

Code:

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 QVO 1TB
Serial Number:    S5XXXXXXXXXXXXXXXXXX
LU WWN Device Id: 5 002538 f31190496
Firmware Version: SVQ01B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)

 

[da_da]

Of course

Also QVO drives might not be the best choice for your use case. The QLC flash has got a sustained write rate of about 80 MB/s (anandtech benchmark via search engine). Therefore there's an SLC cache on the device. Possibly you are experiencing stalls whenever that fills up. Firmware update might make the drive handle writes a bit better. Then again it might not. Can't argue with physics.

I am quite satisfied with 970 EVO Plus (TLC flash) for home use.

aargh!!! Thanks for that. :(

 

[da_da]

OK, so one more question... I use the jails drive as that. The jails are containing the apps but the data is on the large drives that are mounted on the jails... So you are saying if I do a sustained 700mbps+ it'll hit the limit.

 

[Patrick M. Hausen]

I do not have any of these drives and it's hard to come up with a definite statement concerning all the cacheing tricks vendors play nowadays. One would need to benchmark that in your setup.

OTOH as I posted here the data of the jail itself should be served almost completely from ARC.

So let's take a step back ... are you experiencing any problems at all? Or just an occasional log message that can probably be attributed to the SLC cache filling up? If there are no observable "hangs" of your application, I'd probably ignore the messages. After firmware update and some rough checks, i.e. zpool iostat as I already suggested.

 

[da_da]

OK. It's only happened a handful of times that might have affected the plex server app. But that might have been coincidental. Since I am not having these happen on a regular basis, I'll just keep looking out for them to see if they occur on a particular situation.

One thing, I know it's easy to pop in and out the drives on the big pool. How would I handle the mirror drive replacements without killing the drives.