Transfer Speeds on a New Setup

kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
Hello,

I have a new TrueNAS Scale system I just set up.

It has the following hardware:

Motherboard: MSI X79-GD49 PLUS
CPU: I7-4930K
RAM: 24GB
SSD: 256GB
HDD's: 3 x HGST Ultrastar HUH728080ALE601 8TB

The system was connected to an ethernet switch that was connected to a TP-Link Archer AX11000 router. My PC was connected to the same ethernet switch.

When I tried transferring files I was only getting about 10.4 MBS.

Then I read thru some thread on the forum and decided to connect the system directly to the router. The ethernet cable is plugged in directly in to the motherboard. I am not using a nic. Now I am getting about 36 MBS.

From what I understand after reading the forum threads, it could be the cable, motherboard, the PC i am using, router, etc. I just don't even know where to begin and was hoping someone could assist me in the right direction.

I ran the code below and got the following message.s

Code:
admin@truenas[~]# sudo fio --name=seq-read --rw=read --ioengine=posixaio \
    --blocksize=128k --iodepth=64 --runtime=300 --size=4g \
    --direct=1 --invalidate=1 --time_based --numjobs="$(nproc)"
[sudo] password for admin: 
seq-read: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=posixaio, iodepth=64
...
fio-3.25
Starting 16 processes
Jobs: 16 (f=16): [R(16)][100.0%][r=550MiB/s][r=4403 IOPS][eta 00m:00s]
seq-read: (groupid=0, jobs=1): err= 0: pid=15159: Sun Aug  7 09:33:49 2022
  read: IOPS=254, BW=31.9MiB/s (33.4MB/s)(9560MiB/300139msec)
    slat (nsec): min=559, max=171251, avg=1878.01, stdev=1610.78
    clat (msec): min=23, max=6039, avg=250.98, stdev=272.62
     lat (msec): min=23, max=6039, avg=250.99, stdev=272.62
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  167], 10.00th=[  171], 20.00th=[  178],
     | 30.00th=[  182], 40.00th=[  188], 50.00th=[  192], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  236], 90.00th=[  418], 95.00th=[  558],
     | 99.00th=[  743], 99.50th=[ 1670], 99.90th=[ 5336], 99.95th=[ 6007],
     | 99.99th=[ 6007]
   bw (  KiB/s): min=  512, max=49763, per=6.59%, avg=35344.95, stdev=12518.87, samples=554
   iops        : min=    4, max=  388, avg=276.08, stdev=97.77, samples=554
  lat (msec)   : 50=0.01%, 250=82.02%, 500=11.32%, 750=5.81%, 1000=0.10%
  lat (msec)   : 2000=0.50%, >=2000=0.25%
  cpu          : usr=0.10%, sys=0.13%, ctx=9272, majf=3, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.0%, 16=25.0%, 32=51.5%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.1%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=76482,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15160: Sun Aug  7 09:33:49 2022
  read: IOPS=254, BW=31.8MiB/s (33.3MB/s)(9528MiB/300019msec)
    slat (nsec): min=558, max=170203, avg=1853.23, stdev=1550.34
    clat (msec): min=152, max=3390, avg=251.73, stdev=221.26
     lat (msec): min=152, max=3390, avg=251.73, stdev=221.26
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  167], 10.00th=[  169], 20.00th=[  176],
     | 30.00th=[  184], 40.00th=[  190], 50.00th=[  201], 60.00th=[  213],
     | 70.00th=[  249], 80.00th=[  275], 90.00th=[  305], 95.00th=[  477],
     | 99.00th=[ 1720], 99.50th=[ 2022], 99.90th=[ 2601], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min= 1792, max=49250, per=6.66%, avg=35720.92, stdev=10551.39, samples=545
   iops        : min=   14, max=  384, avg=279.02, stdev=82.41, samples=545
  lat (msec)   : 250=70.24%, 500=25.45%, 750=2.72%, 1000=0.50%, 2000=0.59%
  lat (msec)   : >=2000=0.50%
  cpu          : usr=0.09%, sys=0.13%, ctx=9270, majf=4, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=76224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15161: Sun Aug  7 09:33:49 2022
  read: IOPS=269, BW=33.7MiB/s (35.4MB/s)(9.88GiB/300107msec)
    slat (nsec): min=558, max=222095, avg=1873.28, stdev=1659.20
    clat (msec): min=13, max=2976, avg=237.06, stdev=114.67
     lat (msec): min=13, max=2976, avg=237.06, stdev=114.67
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  169], 10.00th=[  176], 20.00th=[  186],
     | 30.00th=[  199], 40.00th=[  209], 50.00th=[  224], 60.00th=[  236],
     | 70.00th=[  253], 80.00th=[  275], 90.00th=[  300], 95.00th=[  313],
     | 99.00th=[  430], 99.50th=[  523], 99.90th=[ 1989], 99.95th=[ 2970],
     | 99.99th=[ 2970]
   bw (  KiB/s): min=  512, max=49250, per=6.55%, avg=35138.20, stdev=9537.74, samples=589
   iops        : min=    4, max=  384, avg=274.46, stdev=74.49, samples=589
  lat (msec)   : 20=0.01%, 50=0.01%, 250=68.26%, 500=31.11%, 750=0.40%
  lat (msec)   : 2000=0.16%, >=2000=0.08%
  cpu          : usr=0.13%, sys=0.12%, ctx=9848, majf=5, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.0%, 16=25.0%, 32=50.5%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=80961,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15162: Sun Aug  7 09:33:49 2022
  read: IOPS=263, BW=32.9MiB/s (34.5MB/s)(9872MiB/300134msec)
    slat (nsec): min=559, max=176978, avg=1884.72, stdev=1571.51
    clat (msec): min=26, max=3356, avg=243.04, stdev=160.88
     lat (msec): min=26, max=3356, avg=243.04, stdev=160.88
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  169], 10.00th=[  176], 20.00th=[  184],
     | 30.00th=[  194], 40.00th=[  205], 50.00th=[  222], 60.00th=[  239],
     | 70.00th=[  255], 80.00th=[  271], 90.00th=[  288], 95.00th=[  309],
     | 99.00th=[  642], 99.50th=[ 1770], 99.90th=[ 2299], 99.95th=[ 3306],
     | 99.99th=[ 3306]
   bw (  KiB/s): min=  512, max=49250, per=6.52%, avg=34990.20, stdev=9787.95, samples=577
   iops        : min=    4, max=  384, avg=273.31, stdev=76.45, samples=577
  lat (msec)   : 50=0.01%, 250=67.37%, 500=30.68%, 750=1.30%, 1000=0.08%
  lat (msec)   : 2000=0.33%, >=2000=0.24%
  cpu          : usr=0.11%, sys=0.13%, ctx=9599, majf=4, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.5%, 16=25.0%, 32=51.1%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=78977,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15163: Sun Aug  7 09:33:49 2022
  read: IOPS=263, BW=32.0MiB/s (34.6MB/s)(9896MiB/300050msec)
    slat (nsec): min=558, max=238299, avg=1874.96, stdev=1712.14
    clat (msec): min=25, max=3658, avg=242.38, stdev=163.77
     lat (msec): min=25, max=3658, avg=242.38, stdev=163.77
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  169], 10.00th=[  174], 20.00th=[  182],
     | 30.00th=[  188], 40.00th=[  201], 50.00th=[  213], 60.00th=[  232],
     | 70.00th=[  255], 80.00th=[  271], 90.00th=[  292], 95.00th=[  330],
     | 99.00th=[  676], 99.50th=[  776], 99.90th=[ 1989], 99.95th=[ 3641],
     | 99.99th=[ 3641]
   bw (  KiB/s): min=  512, max=49408, per=6.56%, avg=35169.24, stdev=10163.72, samples=575
   iops        : min=    4, max=  386, avg=274.71, stdev=79.37, samples=575
  lat (msec)   : 50=0.01%, 250=67.28%, 500=30.12%, 750=1.94%, 1000=0.16%
  lat (msec)   : 2000=0.41%, >=2000=0.08%
  cpu          : usr=0.11%, sys=0.12%, ctx=9630, majf=7, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.0%, 16=25.0%, 32=51.5%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.1%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=79170,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15164: Sun Aug  7 09:33:49 2022
  read: IOPS=256, BW=32.0MiB/s (33.6MB/s)(9608MiB/300152msec)
    slat (nsec): min=628, max=289701, avg=1887.67, stdev=1812.02
    clat (msec): min=156, max=4436, avg=249.74, stdev=203.27
     lat (msec): min=156, max=4436, avg=249.74, stdev=203.27
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  167], 10.00th=[  171], 20.00th=[  178],
     | 30.00th=[  184], 40.00th=[  190], 50.00th=[  199], 60.00th=[  209],
     | 70.00th=[  241], 80.00th=[  288], 90.00th=[  309], 95.00th=[  460],
     | 99.00th=[ 1116], 99.50th=[ 1888], 99.90th=[ 1989], 99.95th=[ 3708],
     | 99.99th=[ 3708]
   bw (  KiB/s): min=  512, max=49250, per=6.55%, avg=35116.37, stdev=11339.85, samples=560
   iops        : min=    4, max=  384, avg=274.29, stdev=88.56, samples=560
  lat (msec)   : 250=71.98%, 500=23.94%, 750=2.83%, 1000=0.17%, 2000=1.00%
  lat (msec)   : >=2000=0.08%
  cpu          : usr=0.12%, sys=0.11%, ctx=9359, majf=8, minf=45
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=76864,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15165: Sun Aug  7 09:33:49 2022
  read: IOPS=259, BW=32.4MiB/s (33.0MB/s)(9720MiB/300159msec)
    slat (nsec): min=628, max=172298, avg=1846.76, stdev=1638.97
    clat (msec): min=22, max=4427, avg=246.88, stdev=209.22
     lat (msec): min=22, max=4427, avg=246.88, stdev=209.22
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  167], 10.00th=[  171], 20.00th=[  178],
     | 30.00th=[  184], 40.00th=[  190], 50.00th=[  197], 60.00th=[  203],
     | 70.00th=[  215], 80.00th=[  241], 90.00th=[  384], 95.00th=[  531],
     | 99.00th=[  667], 99.50th=[ 1552], 99.90th=[ 3004], 99.95th=[ 4396],
     | 99.99th=[ 4396]
   bw (  KiB/s): min=  256, max=49408, per=6.52%, avg=34964.02, stdev=12130.99, samples=569
   iops        : min=    2, max=  386, avg=273.10, stdev=94.76, samples=569
  lat (msec)   : 50=0.01%, 250=81.06%, 500=12.68%, 750=5.51%, 1000=0.01%
  lat (msec)   : 2000=0.49%, >=2000=0.25%
  cpu          : usr=0.11%, sys=0.12%, ctx=9449, majf=3, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.7%, 16=25.0%, 32=50.8%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=77759,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15166: Sun Aug  7 09:33:49 2022
  read: IOPS=266, BW=33.3MiB/s (34.9MB/s)(9984MiB/300119msec)
    slat (nsec): min=559, max=180051, avg=1859.75, stdev=1588.79
    clat (msec): min=22, max=3138, avg=240.30, stdev=135.00
     lat (msec): min=22, max=3138, avg=240.30, stdev=135.00
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  171], 10.00th=[  178], 20.00th=[  186],
     | 30.00th=[  197], 40.00th=[  211], 50.00th=[  226], 60.00th=[  241],
     | 70.00th=[  257], 80.00th=[  271], 90.00th=[  292], 95.00th=[  309],
     | 99.00th=[  600], 99.50th=[  667], 99.90th=[ 2056], 99.95th=[ 3138],
     | 99.99th=[ 3138]
   bw (  KiB/s): min= 1024, max=49250, per=6.54%, avg=35050.26, stdev=9098.72, samples=583
   iops        : min=    8, max=  384, avg=273.77, stdev=71.06, samples=583
  lat (msec)   : 50=0.01%, 250=66.02%, 500=32.68%, 750=0.81%, 1000=0.08%
  lat (msec)   : 2000=0.24%, >=2000=0.16%
  cpu          : usr=0.11%, sys=0.13%, ctx=9695, majf=4, minf=55
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=79872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15167: Sun Aug  7 09:33:49 2022
  read: IOPS=262, BW=32.8MiB/s (34.4MB/s)(9856MiB/300128msec)
    slat (nsec): min=558, max=238158, avg=1883.56, stdev=1763.24
    clat (msec): min=23, max=3768, avg=243.43, stdev=172.55
     lat (msec): min=23, max=3768, avg=243.43, stdev=172.55
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  169], 10.00th=[  174], 20.00th=[  186],
     | 30.00th=[  192], 40.00th=[  201], 50.00th=[  211], 60.00th=[  224],
     | 70.00th=[  239], 80.00th=[  264], 90.00th=[  309], 95.00th=[  393],
     | 99.00th=[  701], 99.50th=[  995], 99.90th=[ 2400], 99.95th=[ 3775],
     | 99.99th=[ 3775]
   bw (  KiB/s): min=  512, max=49250, per=6.60%, avg=35395.31, stdev=9645.17, samples=569
   iops        : min=    4, max=  384, avg=276.47, stdev=75.32, samples=569
  lat (msec)   : 50=0.01%, 250=74.99%, 500=23.31%, 750=0.97%, 1000=0.24%
  lat (msec)   : 2000=0.24%, >=2000=0.24%
  cpu          : usr=0.11%, sys=0.12%, ctx=9559, majf=2, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=78848,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15168: Sun Aug  7 09:33:49 2022
  read: IOPS=259, BW=32.4MiB/s (33.0MB/s)(9720MiB/300109msec)
    slat (nsec): min=559, max=195206, avg=1873.82, stdev=1560.58
    clat (msec): min=31, max=2531, avg=246.82, stdev=178.39
     lat (msec): min=31, max=2531, avg=246.82, stdev=178.39
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  167], 10.00th=[  171], 20.00th=[  180],
     | 30.00th=[  188], 40.00th=[  197], 50.00th=[  209], 60.00th=[  226],
     | 70.00th=[  245], 80.00th=[  279], 90.00th=[  309], 95.00th=[  384],
     | 99.00th=[  693], 99.50th=[ 1888], 99.90th=[ 2333], 99.95th=[ 2500],
     | 99.99th=[ 2500]
   bw (  KiB/s): min= 1280, max=49250, per=6.58%, avg=35277.04, stdev=10264.06, samples=563
   iops        : min=   10, max=  384, avg=275.55, stdev=80.18, samples=563
  lat (msec)   : 50=0.01%, 250=72.28%, 500=24.60%, 750=2.22%, 1000=0.08%
  lat (msec)   : 2000=0.58%, >=2000=0.25%
  cpu          : usr=0.10%, sys=0.13%, ctx=9442, majf=4, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=77760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15169: Sun Aug  7 09:33:49 2022
  read: IOPS=256, BW=32.0MiB/s (33.6MB/s)(9616MiB/300145msec)
    slat (nsec): min=558, max=329651, avg=1841.20, stdev=1918.44
    clat (msec): min=153, max=5285, avg=249.53, stdev=230.30
     lat (msec): min=153, max=5285, avg=249.53, stdev=230.30
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  167], 10.00th=[  171], 20.00th=[  178],
     | 30.00th=[  182], 40.00th=[  188], 50.00th=[  197], 60.00th=[  203],
     | 70.00th=[  213], 80.00th=[  234], 90.00th=[  380], 95.00th=[  584],
     | 99.00th=[  768], 99.50th=[ 1804], 99.90th=[ 2299], 99.95th=[ 5269],
     | 99.99th=[ 5269]
   bw (  KiB/s): min=  256, max=49250, per=6.59%, avg=35360.29, stdev=12158.39, samples=557
   iops        : min=    2, max=  384, avg=276.20, stdev=94.97, samples=557
  lat (msec)   : 250=83.99%, 500=8.70%, 750=5.98%, 1000=0.50%, 2000=0.50%
  lat (msec)   : >=2000=0.33%
  cpu          : usr=0.10%, sys=0.12%, ctx=9369, majf=4, minf=49
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=76928,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15170: Sun Aug  7 09:33:49 2022
  read: IOPS=259, BW=32.4MiB/s (34.0MB/s)(9736MiB/300156msec)
    slat (nsec): min=558, max=288165, avg=1897.43, stdev=1927.94
    clat (msec): min=20, max=3059, avg=246.46, stdev=182.39
     lat (msec): min=20, max=3059, avg=246.46, stdev=182.39
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  169], 10.00th=[  174], 20.00th=[  180],
     | 30.00th=[  188], 40.00th=[  199], 50.00th=[  209], 60.00th=[  232],
     | 70.00th=[  257], 80.00th=[  275], 90.00th=[  292], 95.00th=[  326],
     | 99.00th=[  760], 99.50th=[ 1888], 99.90th=[ 2265], 99.95th=[ 3037],
     | 99.99th=[ 3037]
   bw (  KiB/s): min=  256, max=49408, per=6.51%, avg=34898.95, stdev=10384.84, samples=571
   iops        : min=    2, max=  386, avg=272.59, stdev=81.09, samples=571
  lat (msec)   : 50=0.01%, 250=66.90%, 500=30.04%, 750=1.99%, 1000=0.25%
  lat (msec)   : 2000=0.41%, >=2000=0.41%
  cpu          : usr=0.11%, sys=0.12%, ctx=9481, majf=6, minf=52
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.1%, 16=25.0%, 32=51.4%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=77889,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15171: Sun Aug  7 09:33:49 2022
  read: IOPS=266, BW=33.3MiB/s (34.9MB/s)(9.77GiB/300099msec)
    slat (nsec): min=558, max=167270, avg=1889.08, stdev=1642.16
    clat (msec): min=20, max=2945, avg=239.90, stdev=139.93
     lat (msec): min=20, max=2945, avg=239.90, stdev=139.93
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  169], 10.00th=[  174], 20.00th=[  180],
     | 30.00th=[  188], 40.00th=[  197], 50.00th=[  211], 60.00th=[  228],
     | 70.00th=[  253], 80.00th=[  275], 90.00th=[  300], 95.00th=[  317],
     | 99.00th=[  625], 99.50th=[  701], 99.90th=[ 1921], 99.95th=[ 2937],
     | 99.99th=[ 2937]
   bw (  KiB/s): min=  512, max=49250, per=6.54%, avg=35080.34, stdev=10704.25, samples=583
   iops        : min=    4, max=  384, avg=274.01, stdev=83.59, samples=583
  lat (msec)   : 50=0.01%, 250=68.88%, 500=28.21%, 750=2.51%, 2000=0.32%
  lat (msec)   : >=2000=0.08%
  cpu          : usr=0.11%, sys=0.13%, ctx=9734, majf=3, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.4%, 16=25.0%, 32=51.1%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=80001,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15172: Sun Aug  7 09:33:49 2022
  read: IOPS=270, BW=33.9MiB/s (35.5MB/s)(9.92GiB/300080msec)
    slat (nsec): min=559, max=187244, avg=1883.96, stdev=1572.28
    clat (msec): min=22, max=3868, avg=236.10, stdev=122.49
     lat (msec): min=22, max=3868, avg=236.10, stdev=122.49
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  174], 10.00th=[  180], 20.00th=[  190],
     | 30.00th=[  203], 40.00th=[  213], 50.00th=[  228], 60.00th=[  243],
     | 70.00th=[  255], 80.00th=[  271], 90.00th=[  288], 95.00th=[  296],
     | 99.00th=[  338], 99.50th=[  523], 99.90th=[ 2039], 99.95th=[ 3842],
     | 99.99th=[ 3842]
   bw (  KiB/s): min=  512, max=49408, per=6.55%, avg=35131.79, stdev=8883.00, samples=591
   iops        : min=    4, max=  386, avg=274.41, stdev=69.38, samples=591
  lat (msec)   : 50=0.01%, 250=66.28%, 500=33.00%, 750=0.55%, 2000=0.01%
  lat (msec)   : >=2000=0.16%
  cpu          : usr=0.11%, sys=0.13%, ctx=9870, majf=5, minf=50
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.6%, 16=25.0%, 32=51.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=81281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15173: Sun Aug  7 09:33:49 2022
  read: IOPS=266, BW=33.3MiB/s (34.0MB/s)(9.77GiB/300101msec)
    slat (nsec): min=628, max=164406, avg=1863.71, stdev=1483.51
    clat (msec): min=17, max=3732, avg=239.71, stdev=149.39
     lat (msec): min=17, max=3732, avg=239.71, stdev=149.39
    clat percentiles (msec):
     |  1.00th=[  165],  5.00th=[  169], 10.00th=[  174], 20.00th=[  180],
     | 30.00th=[  184], 40.00th=[  190], 50.00th=[  199], 60.00th=[  209],
     | 70.00th=[  224], 80.00th=[  245], 90.00th=[  338], 95.00th=[  527],
     | 99.00th=[  676], 99.50th=[  693], 99.90th=[ 1938], 99.95th=[ 3138],
     | 99.99th=[ 3138]
   bw (  KiB/s): min=  256, max=49408, per=6.57%, avg=35228.43, stdev=11976.08, samples=581
   iops        : min=    2, max=  386, avg=275.17, stdev=93.55, samples=581
  lat (msec)   : 20=0.01%, 50=0.01%, 250=81.11%, 500=13.29%, 750=5.20%
  lat (msec)   : 1000=0.15%, 2000=0.16%, >=2000=0.08%
  cpu          : usr=0.12%, sys=0.12%, ctx=9751, majf=5, minf=49
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.6%, 16=25.0%, 32=50.9%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=80065,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq-read: (groupid=0, jobs=1): err= 0: pid=15174: Sun Aug  7 09:33:49 2022
  read: IOPS=261, BW=32.7MiB/s (34.3MB/s)(9816MiB/300150msec)
    slat (nsec): min=628, max=192692, avg=1889.95, stdev=1858.62
    clat (msec): min=152, max=2758, avg=244.45, stdev=164.53
     lat (msec): min=152, max=2758, avg=244.45, stdev=164.53
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  169], 10.00th=[  171], 20.00th=[  180],
     | 30.00th=[  186], 40.00th=[  194], 50.00th=[  205], 60.00th=[  213],
     | 70.00th=[  228], 80.00th=[  268], 90.00th=[  368], 95.00th=[  426],
     | 99.00th=[  718], 99.50th=[ 1586], 99.90th=[ 2433], 99.95th=[ 2735],
     | 99.99th=[ 2735]
   bw (  KiB/s): min= 1280, max=49250, per=6.54%, avg=35063.18, stdev=10766.05, samples=573
   iops        : min=   10, max=  384, avg=273.87, stdev=84.07, samples=573
  lat (msec)   : 250=77.11%, 500=20.63%, 750=1.36%, 1000=0.24%, 2000=0.50%
  lat (msec)   : >=2000=0.16%
  cpu          : usr=0.11%, sys=0.12%, ctx=9539, majf=3, minf=52
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
     issued rwts: total=78527,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=524MiB/s (549MB/s), 31.8MiB/s-33.9MiB/s (33.3MB/s-35.5MB/s), io=154GiB (165GB), run=300019-300159msec
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Suggest you test the network bandwidth first....iperf.
 
Whattteva

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
Whattteva said:
That's because on SCALE, only iperf3 is installed.
Click to expand...
sorry about that.

Here is the outcome from a win10 machine:

Code:
c:\tools>iperf3 -c 192.168.1.249 -p 5201 -f m
Connecting to host 192.168.1.249, port 5201
[  4] local 192.168.1.160 port 49512 connected to 192.168.1.249 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  40.2 MBytes   336 Mbits/sec
[  4]   1.00-2.00   sec  40.9 MBytes   344 Mbits/sec
[  4]   2.00-3.00   sec  43.1 MBytes   362 Mbits/sec
[  4]   3.00-4.00   sec  44.9 MBytes   376 Mbits/sec
[  4]   4.00-5.00   sec  43.8 MBytes   368 Mbits/sec
[  4]   5.00-6.00   sec  43.9 MBytes   367 Mbits/sec
[  4]   6.00-7.00   sec  42.9 MBytes   360 Mbits/sec
[  4]   7.00-8.00   sec  42.8 MBytes   359 Mbits/sec
[  4]   8.00-9.00   sec  42.0 MBytes   351 Mbits/sec
[  4]   9.00-10.01  sec  44.1 MBytes   369 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   428 MBytes   359 Mbits/sec                  sender
[  4]   0.00-10.01  sec   428 MBytes   359 Mbits/sec                  receiver

iperf Done.
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
32 MByte/s storage is pretty high with only 42 MByte/s of network.
 
Whattteva

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
kamalov said:
how do I increase my network speed/bandwidth?
Click to expand...
Those speeds seem like you're on WiFi. Connect with wire and you should get better iperf. If you have a WiFi card on your PC, disable it.

Also, make sure you're on the same subnet as the PC. You mention that you're getting less speed on the switch, but you should not see any difference in speed. If you do, it's probably cause you're double NAT-ing somehow and that will be subject to both your router's WAN port (which can be limited to 100Mbit on older routers and would make your 10 MB/s transfer rate make perfect sense) and also your router's CPU not being beefy enough to push faster speeds.
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
1st of all document what you have..... NICs and switch ports.
Are you expecting mult-Gbit speeds?
 
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
morganL said:
1st of all document what you have..... NICs and switch ports.
Are you expecting mult-Gbit speeds?
Click to expand...
No NIC. Just the ethernet port on the Motherboard.
I have plugged in both the NAS and the PC directly into the router with ethernet cables. No switch.
Not expecting mult-Gbit speeds. Just asking if I can increase the speed I have.
 
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
Whattteva said:
Those speeds seem like you're on WiFi. Connect with wire and you should get better iperf. If you have a WiFi card on your PC, disable it.

Also, make sure you're on the same subnet as the PC. You mention that you're getting less speed on the switch, but you should not see any difference in speed. If you do, it's probably cause you're double NAT-ing somehow and that will be subject to both your router's WAN port (which can be limited to 100Mbit on older routers and would make your 10 MB/s transfer rate make perfect sense) and also your router's CPU not being beefy enough to push faster speeds.
Click to expand...
Just plugged both the NAS and the PC into the same router with ethernet cords. Went up to about 37MBs.
 
Whattteva

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
That's still really slow for a 1GbE connection (less than 50%).
Here's what you should be seeing.

Code:
$ ./iperf3.exe -c nas1.lan
Connecting to host nas1.lan, port 5201
[  4] local 172.27.140.164 port 57743 connected to 172.16.4.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   111 MBytes   932 Mbits/sec
[  4]   1.00-2.00   sec   108 MBytes   908 Mbits/sec
[  4]   2.00-3.00   sec   112 MBytes   938 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   946 Mbits/sec
[  4]   4.00-5.00   sec   108 MBytes   908 Mbits/sec
[  4]   5.00-6.01   sec   108 MBytes   903 Mbits/sec
[  4]   6.01-7.00   sec   108 MBytes   908 Mbits/sec
[  4]   7.00-8.00   sec   109 MBytes   913 Mbits/sec
[  4]   8.00-9.00   sec   111 MBytes   929 Mbits/sec
[  4]   9.00-10.00  sec   108 MBytes   903 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  receiver

iperf Done.


Since both your NAS and PC are connected to the same router directly, you should be getting full speed. Maybe bad cable or ethernet controller? Have you tried using different cables?
 
Last edited:
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
Whattteva said:
That's still really slow for a 1GbE connection (less than 50%).
Here's what you should be seeing.

Code:
$ ./iperf3.exe -c nas1.lan
Connecting to host nas1.lan, port 5201
[  4] local 172.27.140.164 port 57743 connected to 172.16.4.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   111 MBytes   932 Mbits/sec
[  4]   1.00-2.00   sec   108 MBytes   908 Mbits/sec
[  4]   2.00-3.00   sec   112 MBytes   938 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   946 Mbits/sec
[  4]   4.00-5.00   sec   108 MBytes   908 Mbits/sec
[  4]   5.00-6.01   sec   108 MBytes   903 Mbits/sec
[  4]   6.01-7.00   sec   108 MBytes   908 Mbits/sec
[  4]   7.00-8.00   sec   109 MBytes   913 Mbits/sec
[  4]   8.00-9.00   sec   111 MBytes   929 Mbits/sec
[  4]   9.00-10.00  sec   108 MBytes   903 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  receiver

iperf Done.


Since both your NAS and PC are connected to the same router directly, you should be getting full speed. What are the lengths of your cables?
Click to expand...
thx.

it looks like the iperf speed is higher mbs. The actual transfer rate thru win10 about 37mbs. see below:

Code:
c:\tools>iperf3 -c 192.168.1.249 -p 5201 -f m
Connecting to host 192.168.1.249, port 5201
[  4] local 192.168.1.194 port 51592 connected to 192.168.1.249 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  88.5 MBytes   742 Mbits/sec
[  4]   1.00-2.00   sec  78.6 MBytes   660 Mbits/sec
[  4]   2.00-3.00   sec  81.0 MBytes   679 Mbits/sec
[  4]   3.00-4.00   sec  82.9 MBytes   695 Mbits/sec
[  4]   4.00-5.00   sec  81.2 MBytes   681 Mbits/sec
[  4]   5.00-6.00   sec  83.6 MBytes   701 Mbits/sec
[  4]   6.00-7.00   sec  80.1 MBytes   671 Mbits/sec
[  4]   7.00-8.00   sec  77.4 MBytes   650 Mbits/sec
[  4]   8.00-9.00   sec  80.8 MBytes   677 Mbits/sec
[  4]   9.00-10.01  sec  84.6 MBytes   705 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   819 MBytes   686 Mbits/sec                  sender
[  4]   0.00-10.01  sec   819 MBytes   686 Mbits/sec                  receiver

iperf Done.
 
Whattteva

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
kamalov said:
thx.

it looks like the iperf speed is higher mbs. The actual transfer rate thru win10 about 37mbs. see below:

Code:
c:\tools>iperf3 -c 192.168.1.249 -p 5201 -f m
Connecting to host 192.168.1.249, port 5201
[  4] local 192.168.1.194 port 51592 connected to 192.168.1.249 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  88.5 MBytes   742 Mbits/sec
[  4]   1.00-2.00   sec  78.6 MBytes   660 Mbits/sec
[  4]   2.00-3.00   sec  81.0 MBytes   679 Mbits/sec
[  4]   3.00-4.00   sec  82.9 MBytes   695 Mbits/sec
[  4]   4.00-5.00   sec  81.2 MBytes   681 Mbits/sec
[  4]   5.00-6.00   sec  83.6 MBytes   701 Mbits/sec
[  4]   6.00-7.00   sec  80.1 MBytes   671 Mbits/sec
[  4]   7.00-8.00   sec  77.4 MBytes   650 Mbits/sec
[  4]   8.00-9.00   sec  80.8 MBytes   677 Mbits/sec
[  4]   9.00-10.01  sec  84.6 MBytes   705 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   819 MBytes   686 Mbits/sec                  sender
[  4]   0.00-10.01  sec   819 MBytes   686 Mbits/sec                  receiver

iperf Done.
Click to expand...
Aaahh now that's more like it. Maybe the issue is CPU. Check CPU utilization on the NAS and see if you're getting bottlenecked there. Also, how full is your pool?
 
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
Whattteva said:
Aaahh now that's more like it. Maybe the issue is CPU. Check CPU utilization on the NAS and see if you're getting bottlenecked there. Also, how full is your pool?
Click to expand...
thx. is that the cpu usage? the max has been 8%.
 
ChrisRJ

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
kamalov said:
I have plugged in both the NAS and the PC directly into the router with ethernet cables. No switch.
Click to expand...
Very likely your router is the culprit here. I have had the exact same situation around 12 years ago. The cheapest 15 Euros/USD switch will be faster and probably solve this.
 
kamalov

kamalov

Dabbler
Joined
Dec 14, 2022
Messages
43
ChrisRJ said:
Very likely your router is the culprit here. I have had the exact same situation around 12 years ago. The cheapest 15 Euros/USD switch will be faster and probably solve this.
Click to expand...
what kind of switch?
 
ChrisRJ

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
The cheapest model you can find from Netgear etc. Literally anything for 20 to 40 USD should give you full 1 Gbps speed
 
Whattteva

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
It could be the switch though that would be new to me. I don't think I've ever had a switch severely under-perform that much below the specs. They're typically the last thing you think of is the problem.
 
ChrisRJ

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
Currently there is not dedicated switch in the setup. Instead the built-in one from the router is being used. And my personal experience suggests that this may be the bottleneck. Since getting a simple 5/8 port switch is really cheap, I suggested to do so.
 
You must log in or register to reply here.

Similar threads

andrwhmmr
RAID-Z2 vs Striped Mirror (RAID10) Performance comparison
Replies
14
Views
18K
pschatz100
P
M
TrueNas in proxmox, really fast read, but slow write
Replies
3
Views
2K
jgreco
jgreco
fisherwei
sync write is very poor
Replies
0
Views
916
fisherwei
fisherwei
Donny Davis
Will it do a million IOPS
Replies
12
Views
6K
hescominsoon
hescominsoon
A
NVME speeds get slower with each additional SSD added to a striped pool
Replies
1
Views
266
sretalla
sretalla
Top