Pool not responsive at regular intervals

[lmch]

We are trying to diagnose an issue.
Every Wednesday at 10am for a period of ~15 minutes the pool tank_hdd stops responding
We see the use time and write bandwidth saturate the SLOG devices, but there is no actual data being writen to the pool from the clients.
We looked into the SMART and scrub schedule but doesn't seem to be related.

Any ideas on what could be the cause?


Usecase
ISCSi storage for proxmox virtualization (200+ VMs)

We have the following setup:
TrueNAS-13.0-U5.3 (3dd62f69ac)
Xeon Silver 4216 CPU - 128GB ECC RAM
Dual port 40GBit ethernet

Code:

Chip Name        Board Name
   SAS3416     HBA 9400-16i
   SAS3416     HBA 9400-16i
LSISAS3224      SAS9305-24i

Code:

dev         size    model                      pool          vdev
/dev/da14    480GB  Micron5400MTFDDAK480TGA    tank_hdd      logs mirror-2
/dev/da15    480GB  Micron5400MTFDDAK480TGA    tank_hdd      logs mirror-2
/dev/da0      20TB  ST20000NM007D              tank_hdd      raidz2-0
/dev/da1      20TB  ST20000NM007D              tank_hdd      raidz2-0
/dev/da4      20TB  ST20000NM007D              tank_hdd      raidz2-0
/dev/da5      20TB  ST20000NM007D              tank_hdd      raidz2-0
/dev/da6    7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-0
/dev/da7    7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-0
/dev/da8    7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-1
/dev/da9    7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-1
/dev/da10   7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-2
/dev/da11   7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      mirror-2
/dev/da12   7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      logs mirror-3
/dev/da13   7.68TB  Micron5400MTFDDAK7T6TGA    tank_ssd      logs mirror-3
/dev/da16    480GB  Micron5400MTFDDAK480TGA    tank_hdd_prd  logs mirror-4
/dev/da17    480GB  Micron5400MTFDDAK480TGA    tank_hdd_prd  logs mirror-4
/dev/da2      20TB  ST20000NM007D              tank_hdd_prd  mirror-0
/dev/da3      20TB  ST20000NM007D              tank_hdd_prd  mirror-0
/dev/da18     20TB  ST20000NM007D              tank_hdd_prd  mirror-1
/dev/da19     20TB  ST20000NM007D              tank_hdd_prd  mirror-1
/dev/da20     20TB  ST20000NM007D              tank_hdd_prd  mirror-2
/dev/da21     20TB  ST20000NM007D              tank_hdd_prd  mirror-2
/dev/da22     20TB  ST20000NM007D              tank_hdd_prd  mirror-3
/dev/da23     20TB  ST20000NM007D              tank_hdd_prd  mirror-3