SOLVED Suddenly extreme slow write speed

[ECC]

Hi,
Today I noticed that my truenas server has extremly low write speeds since yesterdays (380kB/s vs 200-500MB/s normally). I don't know why, maybe it has to do something about a test ssd pool, which I configured today. I destroyed this ssd pool, but the problem is still existing.

The read speed on the other hand is normal (>200MB/s)

Code:

root@truenas[~]# fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=10s --readwrite=write --name=testfile

testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][3.6%][eta 40m:26s]
testfile: (groupid=0, jobs=1): err= 0: pid=2754: Thu Mar 7 15:13:30 2024
write: IOPS=40, BW=40.7MiB/s (42.7MB/s)(3351MiB/82276msec); 0 zone resets
bw ( KiB/s): min= 8062, max=3846144, per=100.00%, avg=620361.27, stdev=1086590.45, samples=11
iops : min= 7, max= 3756, avg=605.18, stdev=1061.35, samples=11
cpu : usr=0.06%, sys=1.21%, ctx=9666, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3351,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=3351MiB (3514MB), run=82276-82276msec

I tried to
- test a different 10G cable on my 10G sfp+ nic
- switched to my 2.5NIC
- switched back to my 10g nic
- switched between standard and disabled sync mode on my pool

I don't know where the problem is coming from. Maybe you can help me here, thanks

 

[ECC]

I figured it out myself: The problem was my other server (proxmox). After restarting my proxmox server, the problem went away