SOLVED "zpool trim" degrades multiple disks at once

[blanchet]

Hi,

I have a TrueNAS 12.0u6.1 All-Flash-Array (12 disks + 6 spares)
I have just run the command zpool trim on the pool, and I have suddenly lost 5 disks.

My hardware

• SuperMicro server
• Motherboard: Supermicro X11DPi-NT
• CPU: 2x Xeon Gold 6226R
• RAM: 512 GB DDR4 ECC
• HBA: LSI 9300-8i IT
• SSD: Intel SATA D3-S4610 3.84 TB
• Pool Layout: 2xRaidz2 with 6 disks + 6 hot-spares
• TrueNAS 12.0u6.1

Output for zpool status

Code:

# zpool status  -t flash1                                                                                                        
  pool: flash1                                                                                                                           
 state: DEGRADED                                                                                                                         
status: One or more devices is currently being resilvered.  The pool will                                                                
    continue to function, possibly in a degraded state.                                                                                  
action: Wait for the resilver to complete.                                                                                               
  scan: resilver in progress since Sun Nov 28 15:53:59 2021                                                                              
    3.62T scanned at 1.61G/s, 2.34T issued at 1.04G/s, 3.63T total                                                                       
    3.56G resilvered, 64.40% done, 00:21:15 to go                                                                                        
config:                                                                                                                                  
                                                                                                                                         
    NAME                                              STATE     READ WRITE CKSUM                                                         
    flash1                                            DEGRADED     0     0     0                                                         
      raidz2-0                                        DEGRADED     0     0     0                                                         
        gptid/8c2744aa-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0 2.14M  (untrimmed)                                            
        spare-1                                       DEGRADED     0     0 2.17M                                                         
          gptid/8c5c56a4-22b7-11ec-9ec2-3cecef1ec130  FAULTED      3     0     0  too many errors  (61% trimmed, started at Sun Nov 28 15:2\
9:14 2021)                                                                                                                               
          gptid/fcd7c7d7-280f-11ec-8929-3cecef1ec130  ONLINE       0     0     0  (resilvering)  (untrimmed)                             
        spare-2                                       DEGRADED     0     0 2.16M                                                         
          gptid/8c8a5da6-22b7-11ec-9ec2-3cecef1ec130  FAULTED      0     0     0  too many errors  (63% trimmed, started at Sun Nov 28 15:2\
9:14 2021)                                                                                                                               
          gptid/fcdb7244-280f-11ec-8929-3cecef1ec130  ONLINE       0     0     0  (resilvering)  (untrimmed)                             
        spare-3                                       DEGRADED     0     0 1.91M                                                         
          gptid/8c91ba8f-22b7-11ec-9ec2-3cecef1ec130  FAULTED      0     0     0  too many errors  (61% trimmed, started at Sun Nov 28 15:2\
9:14 2021)                                                                                                                               
          gptid/316217e0-505e-11ec-9fd9-3cecef1ec130  ONLINE       0     0     0  (resilvering)  (untrimmed)                             
        gptid/8cbd1f08-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0 2.14M  (untrimmed)                                            
        spare-5                                       DEGRADED     0     0 1.81M                                                         
          gptid/8ce5963f-22b7-11ec-9ec2-3cecef1ec130  FAULTED      0     0     0  too many errors  (61% trimmed, started at Sun Nov 28 15:2\
9:14 2021)                                                                                                                               
          gptid/317abb89-505e-11ec-9fd9-3cecef1ec130  ONLINE       0     0     0  (resilvering)  (untrimmed)                             
      raidz2-1                                        DEGRADED     0     0     0                                                         
        gptid/8d01c6a3-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0     0  (untrimmed)                                            
        spare-1                                       DEGRADED     0     0     0                                                         
          gptid/8d1376ac-22b7-11ec-9ec2-3cecef1ec130  FAULTED      3     0     0  too many errors  (58% trimmed, started at Sun Nov 28 15:2\
9:14 2021)                                                                                                                               
          gptid/311080be-505e-11ec-9fd9-3cecef1ec130  ONLINE       0     0     0  (resilvering)  (untrimmed)                             
        gptid/8d1a8746-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0     0  (untrimmed)                                            
        gptid/8d294785-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0     0  (untrimmed)                                            
        gptid/8d34b615-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0     0  (untrimmed)                                            
        gptid/8d3c6e89-22b7-11ec-9ec2-3cecef1ec130    ONLINE       0     0     0  (untrimmed)                                            
    spares                                                                                                                               
      gptid/fcd7c7d7-280f-11ec-8929-3cecef1ec130      INUSE     currently in use  (untrimmed)                                            
      gptid/fcdb7244-280f-11ec-8929-3cecef1ec130      INUSE     currently in use  (untrimmed)                                            
      gptid/311080be-505e-11ec-9fd9-3cecef1ec130      INUSE     currently in use  (untrimmed)                                            
      gptid/316217e0-505e-11ec-9fd9-3cecef1ec130      INUSE     currently in use  (untrimmed)                                            
      gptid/317abb89-505e-11ec-9fd9-3cecef1ec130      INUSE     currently in use  (untrimmed)                                            
      gptid/3190b5cc-505e-11ec-9fd9-3cecef1ec130      AVAIL     (untrimmed)    

Question

• Is zpool trim a really dangerous command or do I have an hardware issue ?

 

[blanchet]

I have waited for the end of resilvering but finally my pool is corrupted.
Because my hardware worked well until now, I guess that I have encountered of software bug following the burst of I/O.
I have recreated a new pool as a stripe of mirrors (to have faster resilvering) and I have restored the data from the backup.

Of course, I will not use anymore zpool trim.

 

[awasb]

What disks/SSDs did you (try to) trim?

 

[blanchet]

I have tried to trim 24 x Intel SSD D3-S4610 SATA 3.84 TB
The exact reference is ATA INTEL SSDSC2KG03 3840

 

[awasb]

Strange, since those SSDs "officially" support Read Zero After TRIM (which is needed for the SAS3008-cards).

 

[blanchet]

I have switched to a stripe for mirrors for several months.
I do not have any issue: now it works very well. even zfs trim works normally.

 

[sretalla]

Are you on this version of firmware?

LSI 9300-xx Firmware Update

Hey Community, If you are using an LSI 9300 HBA with FreeNAS or the soon-to-be TrueNAS CORE, you may experience some performance issues causing the controller to reset when using SATA HDDs. After working with Broadcom, we’ve come up with a...

www.truenas.com

 

[blanchet]

@sretalla You are right, I am running with an older firmware 16.00.01.00 instead of 16.00.12.00

I should upgrade.