TrueNas in proxmox, really fast read, but slow write

[Maca757]

Hi,

I have TrueNAS in proxmox with 8 cores, 32GB of ram and pci passtrough of the SATA controller.
My issue is I have really fast read speed, but horrible write. I am running RAIDZ1.

TrueNAS version: TrueNAS-SCALE-22.02.4
Proxmox version: 7.3-3

Proxmox config:

My configuration:
Intel Xeon Silver 4210 CPU
96GB Samsung DDR4 2933 Mhz ECC Ram
4x SEAGATE HDD Mobile Barracuda25 Guardian (2.5/ 2TB/ SATA 6Gb/s/ rmp 5400) - (its not a real nas drive, but it will get upgraded soon), these are in RAIDZ1


Here is the result of write and read speed in TrueNAS shell:

Code:

pool1: (groupid=0, jobs=1): err= 0: pid=30860: Sat Dec  3 10:15:41 2022
  write: IOPS=27, BW=27.7MiB/s (29.0MB/s)(5120MiB/184910msec); 0 zone resets
    slat (usec): min=12, max=7850, avg=38.36, stdev=110.32
    clat (usec): min=187, max=37853k, avg=36074.89, stdev=1035687.52
     lat (usec): min=211, max=37853k, avg=36113.25, stdev=1035687.27
    clat percentiles (usec):
     |  1.00th=[     212],  5.00th=[     223], 10.00th=[     231],
     | 20.00th=[     255], 30.00th=[     289], 40.00th=[     334],
     | 50.00th=[     347], 60.00th=[     359], 70.00th=[     371],
     | 80.00th=[     392], 90.00th=[     453], 95.00th=[     515],
     | 99.00th=[     938], 99.50th=[    3589], 99.90th=[12012487],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=133120, max=1681408, per=100.00%, avg=895958.00, stdev=543132.74, samples=11
   iops        : min=  130, max= 1642, avg=874.91, stdev=530.46, samples=11
  lat (usec)   : 250=18.28%, 500=76.15%, 750=3.98%, 1000=0.59%
  lat (msec)   : 2=0.29%, 4=0.33%, 10=0.16%, 20=0.02%, 2000=0.04%
  lat (msec)   : >=2000=0.16%
  cpu          : usr=0.12%, sys=0.03%, ctx=5924, majf=7, minf=49
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=27.7MiB/s (29.0MB/s), 27.7MiB/s-27.7MiB/s (29.0MB/s-29.0MB/s), io=5120MiB (5369MB), run=184910-184910msec



fio --name=pool1 --size=5g --rw=read --ioengine=posixaio --direct=1 --bs=1m
pool1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1)
pool1: (groupid=0, jobs=1): err= 0: pid=56990: Sat Dec  3 10:26:53 2022
  read: IOPS=2717, BW=2718MiB/s (2850MB/s)(5120MiB/1884msec)
    slat (nsec): min=1379, max=458656, avg=6651.71, stdev=6769.14
    clat (usec): min=291, max=1858, avg=359.85, stdev=40.09
     lat (usec): min=297, max=1864, avg=366.50, stdev=41.51
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  334], 10.00th=[  338], 20.00th=[  343],
     | 30.00th=[  347], 40.00th=[  351], 50.00th=[  351], 60.00th=[  359],
     | 70.00th=[  363], 80.00th=[  371], 90.00th=[  388], 95.00th=[  404],
     | 99.00th=[  461], 99.50th=[  515], 99.90th=[  873], 99.95th=[  996],
     | 99.99th=[ 1860]
   bw (  MiB/s): min= 2718, max= 2732, per=100.00%, avg=2724.19, stdev= 6.98, samples=3
   iops        : min= 2718, max= 2732, avg=2724.00, stdev= 7.21, samples=3
  lat (usec)   : 500=99.43%, 750=0.43%, 1000=0.10%
  lat (msec)   : 2=0.04%
  cpu          : usr=3.51%, sys=3.13%, ctx=5352, majf=0, minf=47
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2718MiB/s (2850MB/s), 2718MiB/s-2718MiB/s (2850MB/s-2850MB/s), io=5120MiB (5369MB), run=1884-1884msec

Any advice or help is appreciated!

 

[ChrisRJ]

Are these SMR drives?

 

[Maca757]

According to the datasheet they are SMR. Model: ST2000LM015
Just did some research and found out they can impact write speeds. But is it really that much?
I mean I bought some, and I had some found laying around. So didn't really bother to look into other options.

If that is the reason, could you recommend me some good 2 or 4TB 2.5" drives. And what is the best way to move data to new drives.

Thanks!

 

[jgreco]

But is it really that much?

Oh dear.

Western Digital sued over SMR debacle in HDDs

Back in April, there was quite a bit of discussion about WD mot mention that a part of their NAS line of HDDs was not RAID compatible due to storage methodology. They're now getting sued about it....

www.guru3d.com

They really are that bad for ZFS, or even just in general.

If that is the reason, could you recommend me some good 2 or 4TB 2.5" drives.

There are no good 2.5" CMR drives beyond 2.4TB.