Performance Issues on SSD Pool

[ksdmg]

Hello all,
I'm trying to find the problem in my SSD pool for a long time now. I read a lot in the forum and tried recommended optimizations. I also tried to disable sync, compression etc. but nothing changed the massive performance drops I have. I hope that someone can help me.
My system details are in my signature.
Here is my test scenario:
I just copy the game Red Dead Redemption 2 from my workstation over SMB to my pool. On the Ironwolf pool with only HDDs I get consistent write speeds:

Speed is somewhere between 400MB/s up to 600MB/s

Now my SSD pool for comparison:
Again same set of data, Red Dead Redeption 2

Speed drops down to 10MB/s, stays there for a while and peaks up to 200MB/s just to drop again.

Here are my pool stats:

Code:

root@truenas[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
SSD        2.72T   389G  2.34T        -         -     3%    13%  1.00x    ONLINE  /mnt
boot-pool   220G  11.8G   208G        -         -     2%     5%  1.00x    ONLINE  -
ironwolf   54.5T  26.5T  28.1T        -         -     0%    48%  1.00x    ONLINE  /mnt

root@truenas[~]# zpool iostat -v
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
SSD                                        403G  2.33T      1     97  56.7K  2.79M
  raidz1-0                                 403G  2.33T      1     97  56.7K  2.79M
    b43d33d3-ed83-4225-981b-87fc54005ab1      -      -      0     31  19.2K   951K
    bb441c9d-1620-479b-bf02-ea49e01ba1a2      -      -      0     35  19.0K   951K
    ce6b35db-f921-4f27-ad87-b1869cf6db52      -      -      0     31  18.5K   951K
----------------------------------------  -----  -----  -----  -----  -----  -----
ironwolf                                  26.5T  28.1T     35     30  1.47M  1.45M
  raidz1-0                                26.5T  28.1T     35     30  1.47M  1.45M
    b4e23e80-118c-4443-9a0e-ba0ff94ec993      -      -      7      6   304K   298K
    673bf22b-4598-4a6d-bd73-a319fd6cfc69      -      -      6      5   297K   298K
    b3737655-bb3a-48e0-b53a-157d6f0b0f00      -      -      6      5   298K   298K
    03ff53e2-7639-41da-aaf0-50ae020a1ea0      -      -      7      6   300K   298K
    5cde391e-b06c-4f1a-ba04-14b6484ff741      -      -      7      5   302K   298K
----------------------------------------  -----  -----  -----  -----  -----  -----

I tried plugging the SSDs into the Raid controller and directly onto the mainboard, but the performance is exactly the same.

Here is the CPU utilization during the copy process

What else can I try? I would really like to use the ssd pool as vm storage for Proxmox, but the performance drops cause a very high IO delay, so I had to move all the volumes back to the proxmox host for now.

 

[ThisTruenasUser]

I quickly looked at the specs of the SSDs used.
Are they QLC, TLC, MLC, or SLC?
So Quad, Triple, Multiple or single bits per cell in the SSD?

My guess is it is a QLC with an SLC cache.
Large writes mean when the SLC cache runs low, write speeds just go down badly.

Also if using truenas to store windows games , I suggest using ISCSI instead of smb shares.

My games are on truneas scale, using hard drives for storage.
Games are accessed using ISCSI and 2.5Gb network.
On the windows machine, I also use a 1/2 TB NVME as a cache and primocache software to speed up access.

 

[ksdmg]

Thanks for taking time to look into my issue.
The SSDs are Toshiba TLC. I connected a single SSD directly to my workstation to do the same copy job. It copied the whole game with a constant 500MB/s, so the SSDs should be able to handle the load. SSD Details

Sorry for the confusion about storing games, I just used this as test data. I want to use the SSD pool to store my Proxmox VMs and CTs, but because it has such a high IO delay I could not use it. It slowed Proxmox down as hell. In proxmox I used a NFS share. I did speed tests too, and they show exactly the same characteristics.

And btw. read speeds are totally fine, like 900MB/s constant over more than 100GB transfer.