speeds about right for the setup i have?

[cheabred]

10x4TB in ZFS2 array with no SSD cache only 72GB of ram.
24 CPU cores, and 10G direct DAC connection. disk IO wait times are all ns so thats not an issue, is the spikes up to full 10G really just it trying to fill the RAM write cache?

Boot drive is 2x 120g SSDs in zfs1

test file was a 30G .rar file

just want to see if anyone has some suggestions to get the speeds to be more steady than this as i know it should be more steady since there is more than 30G of FREE ram in the system when i move files to it it should write it straight to RAM and then to disk correct?

 

[jgreco]

ZFS2 array

Do you mean a RAIDZ2 pool?

You didn't mention whether you're using CORE or SCALE. Due to the way memory is allocated on these systems, they behave somewhat differently. Also there's no mention of what sort of disk controller you're using.

It's not really out of the realm of possibility for this to be completely okay. At 829MBytes/sec, you are seeing very good sequential performance; a RAIDZ array is typically only capable of sustaining speeds that are some multiple of the underlying drives. If your 10x4TB HDD's are only capable of sustaining 150-200MBytes/sec each, for example, I wouldn't expect more than 4x or 5x that speed for writes. RAIDZ is not optimized for parallelism.

just want to see if anyone has some suggestions to get the speeds to be more steady than this as i know it should be more steady

Well, the dips are a little bit low so I'm wondering what you're using to attach the disks to your system, and if maybe you're using SCALE, which uses Linux and its crappy memory management. But the peaks and valleys are certainly not that unusual, as ZFS builds transaction groups in memory and then flushes them to disk, which can result in the sort of varying throughput you're seeing.

 

[cheabred]

Do you mean a RAIDZ2 pool?

You didn't mention whether you're using CORE or SCALE. Due to the way memory is allocated on these systems, they behave somewhat differently. Also there's no mention of what sort of disk controller you're using.

It's not really out of the realm of possibility for this to be completely okay. At 829MBytes/sec, you are seeing very good sequential performance; a RAIDZ array is typically only capable of sustaining speeds that are some multiple of the underlying drives. If your 10x4TB HDD's are only capable of sustaining 150-200MBytes/sec each, for example, I wouldn't expect more than 4x or 5x that speed for writes. RAIDZ is not optimized for parallelism.



Well, the dips are a little bit low so I'm wondering what you're using to attach the disks to your system, and if maybe you're using SCALE, which uses Linux and its crappy memory management. But the peaks and valleys are certainly not that unusual, as ZFS builds transaction groups in memory and then flushes them to disk, which can result in the sort of varying throughput you're seeing.

i forgot scale even exsisted.. haha its truenas CORE


and yes RAIDz2


the SAS card im using is a

Adaptec ASR 71605 16 Ports 2274400-R Raid Card AFM-700 1GB Flash CacheVault BBU

its in HBA mode for the 12 disks its probably not the best but its what i had laying around, im not as concerned about the total throughput as much as i am with the drops. as it should write to RAM and then to the disks correct? and if so the 70G of free ram should be able to handle the 30G file without too much issue.when its moving at almost 1G its maxing out the CPU cores, which is expected but then it drops and the CPU's drop as well.

i checked thermal (it was running hot) so i re-thermaled the CPU's and its a LOT better, that made it slightly better but still the same dips.

there was a drive with 100ms write waits so i replaced that and got them alll into ns and that got the low speeds into the 20MB/s instead of Kb/s which it was doing before.

im just trying to get it to be a steady 100/200 ish MB/s area and smooth it out some

 

[jgreco]

Adaptec ASR 71605 16 Ports 2274400-R Raid Card AFM-700 1GB Flash CacheVault BBU

Well there's your problem.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[cheabred]

so even though the card has a HBA mode its still not a true HBA?

 

[jgreco]

so even though the card has a HBA mode its still not a true HBA?

Please read the linked article. It has the comprehensive answer to this question.

 

[cheabred]

so would somthing like this work better than?

Supermicro 8-Port 12Gb/s SAS-3 2x SFF-8643 Internal HBA JBOD IT Chipset REV 2.00 913792939840 | eBay

Find many great new & used options and get the best deals for Supermicro 8-Port 12Gb/s SAS-3 2x SFF-8643 Internal HBA JBOD IT Chipset REV 2.00 at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[jgreco]

That's a decent card especially if you have a Supermicro mainboard (but should work fine even if not). Please make sure the right firmware is loaded on it.

 

[cheabred]

That's a decent card especially if you have a Supermicro mainboard (but should work fine even if not). Please make sure the right firmware is loaded on it.

i do have a supermicro main board, am i just making sure it has a IT firmware loaded? or is there a specific version i would need to load?

i read that post and saw reference to LSI firmware, but i don't see any mention to LSI on that card.. so just wanted to check.

i thought that link was an ad at first.. so i read it now haha....

 

[jgreco]

Specific firmware versions are listed in point 3) of the linked article. It is important to use the correct firmware.

 

[cheabred]

ok, but that whole section only list LSI cards. there is no reference to LSI on that Supermicro card? so how does one figure that out?

 

[jgreco]

The card in your eBay listing is an LSI SAS 9300-8i HBA. It just has "Supermicro" graffiti'd on it; Supermicro may "make" the card but it is LSI silicon and an LSI design.

 

[cheabred]

ok thank you, can i ask how one would go about to identify that? i tried to look at the card and all the chips, and i dont see anything listed

 

[jgreco]

It's sort of the same way you can find out whether your PC is powered by Intel or AMD. It may not say, but when you do a little research on the card, or carefully observe the boot process, or just ask the card, it will tell you or at least give you more clues to help identify it.

Code:

# sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------    ------  ------
   0     SAS3008       1000h   97h    00h:0bh:00h:00h      15d9h   0808h
SAS3IRCU: Utility Completed Successfully.

 

[cheabred]

OK new cards installed. and flashed to correct firmware.
rebooted server after installing as well just as a good measure.

Same setup, same variables, just new cards, same test file and same result. so far.. will do some more testing. but speeds still droping down to Kb/s from almost 1G/s

and then it spikes up for a seccond

anyone have somthing more to try? same result between a RAID card in IT mode and an actual HBA card.

Edit:

Here is a CPU and RAM usage picture as well WHILE its transferring the data

Another while its spiking upto the 10G area

Just a plan ISO file, same result, just looks better as its smaller file.

 

[sretalla]

OK, time to shift the focus to network and actual pool performance.

you'll need to learn a little about iperf3 and fio to get to the next steps.

Since I can see this as images, can you elaborate how many files we're talking about inside that 4GB?

It also appears your tests are reading on the TrueNAS side, so the fio command which would be of interest is:
fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=50g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=1 --runtime=120 --group_reporting

cd to a location on the pool in question first.

check out iperf3 and run it on the client and on TrueNAS to get a clear report on how your network runs.

 

[cheabred]

OK, time to shift the focus to network and actual pool performance.

you'll need to learn a little about iperf3 and fio to get to the next steps.

Since I can see this as images, can you elaborate how many files we're talking about inside that 4GB?

It also appears your tests are reading on the TrueNAS side, so the fio command which would be of interest is:
fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=50g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=1 --runtime=120 --group_reporting

cd to a location on the pool in question first.

check out iperf3 and run it on the client and on TrueNAS to get a clear report on how your network runs.

getting it turned back on to do the fio test, the network currently is direct DAC cable from my PC to the server no switch or anything, and Intel SPF+ cards. so i dont see how that can be lower than as expected, but will post the results of the iperf test as well.

The 4G ISO file is just a random ISO i had laying around, with looks like some bootable image i had

thats what the inside of the ISO looks like


Tests that you said to run...

Show : iperf3 test

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 497 MBytes 417 Mbits/sec sender
[ 4] 0.00-10.00 sec 497 MBytes 417 Mbits/sec receiver
[ 6] 0.00-10.00 sec 517 MBytes 434 Mbits/sec sender
[ 6] 0.00-10.00 sec 517 MBytes 433 Mbits/sec receiver
[ 8] 0.00-10.00 sec 496 MBytes 416 Mbits/sec sender
[ 8] 0.00-10.00 sec 496 MBytes 416 Mbits/sec receiver
[ 10] 0.00-10.00 sec 497 MBytes 417 Mbits/sec sender
[ 10] 0.00-10.00 sec 497 MBytes 417 Mbits/sec receiver
[ 12] 0.00-10.00 sec 556 MBytes 466 Mbits/sec sender
[ 12] 0.00-10.00 sec 556 MBytes 466 Mbits/sec receiver
[ 14] 0.00-10.00 sec 850 MBytes 713 Mbits/sec sender
[ 14] 0.00-10.00 sec 850 MBytes 713 Mbits/sec receiver
[ 16] 0.00-10.00 sec 626 MBytes 525 Mbits/sec sender
[ 16] 0.00-10.00 sec 626 MBytes 525 Mbits/sec receiver
[ 18] 0.00-10.00 sec 825 MBytes 692 Mbits/sec sender
[ 18] 0.00-10.00 sec 825 MBytes 692 Mbits/sec receiver
[ 20] 0.00-10.00 sec 520 MBytes 437 Mbits/sec sender
[ 20] 0.00-10.00 sec 520 MBytes 436 Mbits/sec receiver
[ 22] 0.00-10.00 sec 500 MBytes 419 Mbits/sec sender
[ 22] 0.00-10.00 sec 500 MBytes 419 Mbits/sec receiver
[ 24] 0.00-10.00 sec 489 MBytes 410 Mbits/sec sender
[ 24] 0.00-10.00 sec 489 MBytes 410 Mbits/sec receiver
[ 26] 0.00-10.00 sec 492 MBytes 413 Mbits/sec sender
[ 26] 0.00-10.00 sec 492 MBytes 413 Mbits/sec receiver
[ 28] 0.00-10.00 sec 488 MBytes 410 Mbits/sec sender
[ 28] 0.00-10.00 sec 488 MBytes 409 Mbits/sec receiver
[ 30] 0.00-10.00 sec 482 MBytes 404 Mbits/sec sender
[ 30] 0.00-10.00 sec 482 MBytes 404 Mbits/sec receiver
[ 32] 0.00-10.00 sec 514 MBytes 431 Mbits/sec sender
[ 32] 0.00-10.00 sec 514 MBytes 431 Mbits/sec receiver
[ 34] 0.00-10.00 sec 522 MBytes 438 Mbits/sec sender
[ 34] 0.00-10.00 sec 522 MBytes 438 Mbits/sec receiver
[ 36] 0.00-10.00 sec 629 MBytes 528 Mbits/sec sender
[ 36] 0.00-10.00 sec 629 MBytes 528 Mbits/sec receiver
[ 38] 0.00-10.00 sec 631 MBytes 529 Mbits/sec sender
[ 38] 0.00-10.00 sec 630 MBytes 529 Mbits/sec receiver
[ 40] 0.00-10.00 sec 569 MBytes 477 Mbits/sec sender
[ 40] 0.00-10.00 sec 569 MBytes 477 Mbits/sec receiver
[ 42] 0.00-10.00 sec 561 MBytes 471 Mbits/sec sender
[ 42] 0.00-10.00 sec 561 MBytes 470 Mbits/sec receiver
[SUM] 0.00-10.00 sec 11.0 GBytes 9.45 Gbits/sec sender
[SUM] 0.00-10.00 sec 11.0 GBytes 9.44 Gbits/sec receiver

Show : FIO test

stats on RAM and CPU while running the test

A while into the test

While it was printing results

Actual test output:

Code:

root@tns-backups[/mnt/Backups/data]# fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=50g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=1 --runtime=120 --group_reporting
TEST: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16
fio-3.28
Starting 1 process
TEST: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][5.8%][r=363MiB/s][r=2906 IOPS][eta 01m:53s]
Jobs: 1 (f=1): [R(1)][10.0%][r=301MiB/s][r=2410 IOPS][eta 01m:48s]
Jobs: 1 (f=1): [R(1)][14.2%][r=485MiB/s][r=3882 IOPS][eta 01m:43s]
Jobs: 1 (f=1): [R(1)][18.3%][r=286MiB/s][r=2291 IOPS][eta 01m:38s]
Jobs: 1 (f=1): [R(1)][23.3%][r=161MiB/s][r=1286 IOPS][eta 01m:32s]
Jobs: 1 (f=1): [R(1)][27.5%][r=155MiB/s][r=1243 IOPS][eta 01m:27s]
Jobs: 1 (f=1): [R(1)][31.7%][r=152MiB/s][r=1216 IOPS][eta 01m:22s]
Jobs: 1 (f=1): [R(1)][35.8%][r=153MiB/s][r=1225 IOPS][eta 01m:17s]
Jobs: 1 (f=1): [R(1)][40.0%][r=153MiB/s][r=1221 IOPS][eta 01m:12s]
Jobs: 1 (f=1): [R(1)][44.2%][r=156MiB/s][r=1244 IOPS][eta 01m:07s]
Jobs: 1 (f=1): [R(1)][48.3%][r=156MiB/s][r=1246 IOPS][eta 01m:02s]
Jobs: 1 (f=1): [R(1)][52.5%][r=151MiB/s][r=1210 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [R(1)][56.7%][r=154MiB/s][r=1229 IOPS][eta 00m:52s]
Jobs: 1 (f=1): [R(1)][60.8%][r=153MiB/s][r=1224 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [R(1)][65.0%][r=152MiB/s][r=1218 IOPS][eta 00m:42s]
Jobs: 1 (f=1): [R(1)][69.2%][r=158MiB/s][r=1267 IOPS][eta 00m:37s]
Jobs: 1 (f=1): [R(1)][73.3%][r=156MiB/s][r=1247 IOPS][eta 00m:32s]
Jobs: 1 (f=1): [R(1)][77.5%][r=157MiB/s][r=1257 IOPS][eta 00m:27s]
Jobs: 1 (f=1): [R(1)][82.5%][r=150MiB/s][r=1198 IOPS][eta 00m:21s]
Jobs: 1 (f=1): [R(1)][86.7%][r=157MiB/s][r=1257 IOPS][eta 00m:16s]
Jobs: 1 (f=1): [R(1)][90.8%][r=158MiB/s][r=1262 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [R(1)][95.8%][r=157MiB/s][r=1257 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [R(1)][100.0%][r=152MiB/s][r=1218 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2507: Wed Jan 18 15:03:29 2023
  read: IOPS=1577, BW=197MiB/s (207MB/s)(23.1GiB/120001msec)
    clat (usec): min=30, max=387501, avg=631.48, stdev=2088.42
     lat (usec): min=30, max=387501, avg=631.67, stdev=2088.42
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   72], 10.00th=[   77], 20.00th=[   85],
     | 30.00th=[   96], 40.00th=[  766], 50.00th=[  775], 60.00th=[  791],
     | 70.00th=[  799], 80.00th=[  807], 90.00th=[  816], 95.00th=[  840],
     | 99.00th=[ 1004], 99.50th=[ 1598], 99.90th=[27132], 99.95th=[41157],
     | 99.99th=[78119]
   bw (  KiB/s): min=68047, max=554581, per=100.00%, avg=202088.29, stdev=95531.28, samples=239
   iops        : min=  531, max= 4332, avg=1578.48, stdev=746.30, samples=239
  lat (usec)   : 50=0.56%, 100=30.96%, 250=3.12%, 500=1.86%, 750=0.04%
  lat (usec)   : 1000=62.47%
  lat (msec)   : 2=0.52%, 4=0.07%, 10=0.14%, 20=0.12%, 50=0.12%
  lat (msec)   : 100=0.02%, 250=0.01%, 500=0.01%
  cpu          : usr=0.39%, sys=85.42%, ctx=2767, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=189243,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=23.1GiB (24.8GB), run=120001-120001msec

Pool looks like it can do about 150MB/s ish, which is what i would expect out of straight platter drives in a RAIDZ2

so i would think the transfer speeds could be around 100 ish steadily right?

 

[Davvo]

just want to see if anyone has some suggestions to get the speeds to be more steady than this as i know it should be more steady since there is more than 30G of FREE ram in the system when i move files to it it should write it straight to RAM and then to disk correct?

Wrong. ARC works as read cache, not as write cache: you only benefit from it when you read things from the NAS.

In your tests I would also include a run of jgreco's solnet array; you also didn't test large files, but that should be covered by the fio command sretalla wrote you.

Generally, in order to get better write speed you need more vdevs.
Generally, in order to get better performance with small files you need mirrors.

If you want better write speeds with small files, your best shot is likely to go with 5 vdevs of 2-way mirrors. The second best alternative is to go with 2 vdevs of 5 disks in RAIDZ2.

Please read the following resources:

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

Also, please list your complete hardware specs since it might help.

 

[cheabred]

Wrong. ARC works as read cache, not as write cache: you only benefit from it when you read things from the NAS.

In your tests I would also include a run of jgreco's solnet array; you also didn't test large files, but that should be covered by the fio command sretalla wrote you.

Generally, in order to get better write speed you need more vdevs.
Generally, in order to get better performance with small files you need mirrors.

If you want better write speeds with small files, your best shot is likely to go with 5 vdevs of 2-way mirrors. The second best alternative is to go with 2 vdevs of 5 disks in RAIDZ2.

Please read the following resources:

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

Also, please list your complete hardware specs since it might help.

im not actually worried about better performance, I'll be 100% happy with 100MB/s in fact that's more than enough, this is just for backups, i just want the drops down to 0Bytes/s to go away, as its causing backups to crash as it thinks its froze or thread locked.


Specs:

2x Intel(R) Xeon(R) CPU X5675 @ 3.07GHz - total 24 Cores
72GB RAM
10x4TB HDDs (Mixed of WD Drives and some "Enterprise" drives all the same speed and capacity)
1x 4TB hotspare
512GB SSD as boot drive (overkill i know, but i had it laying around.)

2x sas cards are the ones listed above with the truenas firmwhere update on them, 3 sas ports used

Supermicro board in a 2U case with a 2u backplane that has 3 sas ports on it.