Slow Performance of TrueNAS R20 Enterprise All-Flash

[Scampicfx]

Dear Community,

after having dealt with many FreeNAS / TrueNAS Core systems, I wanted to get in touch with my first all-flash server. Therefore, I recently bought an TrueNAS R20 all-flash server by iXSystems. I had high expectations, but after connecting it via Fibre Channel to my ESXi host, I noticed that there is something wrong.

I would like to hear your opinion regarding the performance of this machine. In my opinion, these performance values are way too low! When I compare some other threads here in the forums, they show performance well above 1000 MB/s and ten thousands of IOPS. I'm miles away from these values. Here are some benchmarks:

Command:

Code:

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
erf

Result:

Code:

random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.27
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [F(1)][100.0%][w=86.3MiB/s][w=22.1k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=80346: Thu Oct 14 21:11:41 2021
  write: IOPS=24.2k, BW=94.5MiB/s (99.1MB/s)(5729MiB/60625msec); 0 zone resets
    slat (nsec): min=1083, max=8310.5k, avg=5143.45, stdev=13051.82
    clat (nsec): min=649, max=40429k, avg=34843.03, stdev=106826.69
     lat (usec): min=8, max=40430, avg=39.99, stdev=106.96
    clat percentiles (nsec):
     |  1.00th=[    724],  5.00th=[    852], 10.00th=[   7392],
     | 20.00th=[  11840], 30.00th=[  13120], 40.00th=[  14016],
     | 50.00th=[  15424], 60.00th=[  18816], 70.00th=[  36096],
     | 80.00th=[  45824], 90.00th=[  64256], 95.00th=[  74240],
     | 99.00th=[ 105984], 99.50th=[1122304], 99.90th=[1122304],
     | 99.95th=[1138688], 99.99th=[1171456]
   bw (  KiB/s): min=57968, max=223809, per=100.00%, avg=97774.81, stdev=26052.37, samples=119
   iops        : min=14492, max=55952, avg=24443.35, stdev=6513.04, samples=119
  lat (nsec)   : 750=1.97%, 1000=4.89%
  lat (usec)   : 2=2.02%, 4=0.28%, 10=2.23%, 20=50.45%, 50=19.46%
  lat (usec)   : 100=17.63%, 250=0.29%, 500=0.04%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.74%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=6.12%, sys=11.03%, ctx=1676099, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1466606,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=94.5MiB/s (99.1MB/s), 94.5MiB/s-94.5MiB/s (99.1MB/s-99.1MB/s), io=5729MiB (6007MB), run=60625-60625msec

Command:

Code:

NRFILES=2; NUMJOBS=10; ARCMAX=$((`sysctl -n vfs.zfs.arc_max` * 2 / 1024 / 1024 / $NRFILES / $NUMJOBS)); fio --name=seq128k --ioengine=posixaio --direct=1 --buffer_compress_percentage=20 --rw=rw --rwmixread=30 --directory=/mnt/tank/perf --filename_format=seq.\$jobnum.\$filenum --nrfiles=$NRFILES --filesize=${ARCMAX}m --blocksize=128k --time_based=1  --runtime=300 --group_reporting --numjobs=$NUMJOBS --iodepth=4

Result:

Code:

fio-3.27
Starting 10 processes
Jobs: 10 (f=20):
seq128k: (groupid=0, jobs=10): err= 0: pid=80687: Thu Oct 14 21:32:22 2021
  read: IOPS=2351, BW=294MiB/s (308MB/s)(86.1GiB/300010msec)
    slat (nsec): min=946, max=1471.4M, avg=249756.50, stdev=5392261.38
    clat (nsec): min=1142, max=1723.6M, avg=598333.38, stdev=10144015.29
     lat (usec): min=16, max=1723.8k, avg=848.09, stdev=11487.95
    clat percentiles (nsec):
     |  1.00th=[     1336],  5.00th=[     1448], 10.00th=[     1528],
     | 20.00th=[     1672], 30.00th=[     1944], 40.00th=[     2416],
     | 50.00th=[     8768], 60.00th=[    43264], 70.00th=[    61184],
     | 80.00th=[    84480], 90.00th=[   242688], 95.00th=[   675840],
     | 99.00th=[  5537792], 99.50th=[ 10682368], 99.90th=[149946368],
     | 99.95th=[185597952], 99.99th=[408944640]
   bw (  KiB/s): min= 3810, max=2014864, per=100.00%, avg=307335.48, stdev=14589.87, samples=5787
   iops        : min=   24, max=15737, avg=2397.04, stdev=114.00, samples=5787
  write: IOPS=5487, BW=686MiB/s (719MB/s)(201GiB/300010msec); 0 zone resets
    slat (nsec): min=988, max=1333.7M, avg=121656.50, stdev=3189799.70
    clat (nsec): min=1317, max=2157.0M, avg=6576278.93, stdev=25930458.84
     lat (usec): min=25, max=2157.0k, avg=6697.94, stdev=26119.99
    clat percentiles (usec):
     |  1.00th=[     3],  5.00th=[   169], 10.00th=[   273], 20.00th=[   498],
     | 30.00th=[  1123], 40.00th=[  3490], 50.00th=[  6849], 60.00th=[  6980],
     | 70.00th=[  7111], 80.00th=[  7242], 90.00th=[  7439], 95.00th=[  7570],
     | 99.00th=[ 55837], 99.50th=[130548], 99.90th=[354419], 99.95th=[526386],
     | 99.99th=[960496]
   bw (  KiB/s): min= 8128, max=4730752, per=100.00%, avg=715079.87, stdev=33556.76, samples=5805
   iops        : min=   57, max=36954, avg=5581.13, stdev=262.17, samples=5805
  lat (usec)   : 2=9.77%, 4=6.53%, 10=0.74%, 20=1.21%, 50=2.59%
  lat (usec)   : 100=6.13%, 250=6.32%, 500=8.98%, 750=4.18%, 1000=2.50%
  lat (msec)   : 2=4.85%, 4=4.48%, 10=39.70%, 20=0.64%, 50=0.54%
  lat (msec)   : 100=0.32%, 250=0.37%, 500=0.09%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.01%
  cpu          : usr=2.56%, sys=0.44%, ctx=3818899, majf=0, minf=10
  IO depths    : 1=0.9%, 2=16.0%, 4=83.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=705519,1646428,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=86.1GiB(92.5GB), run=300010-300010msec
  WRITE: bw=686MiB/s (719MB/s), 686MiB/s-686MiB/s (719MB/s-719MB/s), io=201GiB (216 GB), run=300010-300010msec

EDIT: I had to trim these values a bit because otherwise the posting would exceed the allowed character limit.


Pool Layout:

Code:

  pool: tank
state: ONLINE
  scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Oct 10 00:00:00 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0 0
          mirror-0                                      ONLINE       0     0 0
            gptid/43ce125d-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
            gptid/43d723d8-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
          mirror-1                                      ONLINE       0     0 0
            gptid/43b92e81-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
            gptid/43dc9dd2-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
          mirror-2                                      ONLINE       0     0 0
            gptid/4390162d-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
            gptid/43e6a086-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
          mirror-3                                      ONLINE       0     0 0
            gptid/43aa6c09-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
            gptid/43dc5d25-eff2-11eb-9c5d-3cecef5ee76e  ONLINE       0     0 0
        spares
          gptid/43f04a14-eff2-11eb-9c5d-3cecef5ee76e    AVAIL

The pool consists of 9 SSDs in total. Each SSD is a SEAGATE XS1920SE70094 with 1920 GByte capacity.


Shouldn't there be a lot more speeds with that machine?

A poor Xeon Bronze is built into this system, with a clock rate of 1,9 GHz without Turbo. I can see 100% CPU usage during some tests. Could this be the bottleneck?

Thank you for all your help!

 

[Patrick M. Hausen]

The odds of finding someone with that high end enterprise gear on the community forum are probably lower than you guessed. Why don't you open a support ticket with iXsystems if you have a problem with one of their commercial offerings?

 

[Scampicfx]

Hey Patrick,
thanks for your quick reply. My ticket is already being processed at iXSystems, but when looking back at all the previous years, the community forums here have helped me really a lot! There is much expertise in here and a lot of talented persons who know what they're doing. I have read many postings in here which were really detailed a lot. So thanks again for all your help!

But back to topic:
After I spotted this problem at the R20, I started to compare some performance values of other home-built systems presented here in the forums and was a bit shocked. I myself are also running a couple of other FreeNAS (TrueNAS Core) systems with hybrid pool layout (HDDs and SSD SLOG + L2ARC). Some of them are virtualized. Some of them are running on nearly 20-year old desktop hardware. But they all have one in common: They outperform the R20!

The R20 is my first all-flash system. When looking into the datasheet of a SEAGATE XS1920SE70094, following specs are mentioned:

Read: 2200MB/s
Write: 1650MB/s
IOPS 4k read/write: 230k/85k

This R20 server is equipped with 4 mirrored vdevs consisting of this disk. But when looking at my pool performance, I can see these values:

READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=86.1GiB(92.5GB), run=300010-300010msec
WRITE: bw=686MiB/s (719MB/s), 686MiB/s-686MiB/s (719MB/s-719MB/s), io=201GiB (216 GB), run=300010-300010msec


This was my CPU usage during the benchmark:

It reached 100% easily!

Here are some other benchmark results via Fibre Channel 8 Gbit/s:

8 Gbit Fibre Channel, All-Flash, Sync=Always


8 Gbit Fibre Channel, All-Flash, Sync=Disabled


In contrast, this is the performance of one of my 5-years old FreeNAS servers with hybrid pool:

10 Gbit/s Ethernet, Hybrid Pool, Sync=Always


Please ignore the first line since the limiting factor is the Ethernet / Fibre Channel link.


Thanks; Every piece of information is appreciated!

 

[Patrick M. Hausen]

I wasn't intending to be snarky - I just have developed an attitude towards commercial systems to make any issue the supplier's problem as early in the process as possible. Not intending to abuse suppliers, just expecting that they know best.

Junior operator: "That firewall doesn't behave like expected and I spent hours reading and re-reading the docs and I can't make it work ..."
Me: "I'd have called the supplier after 1 hour of fiddling. Period. 'I wan't to do X, your product claims to be able to do X - make it work. Please ;-)'"

Kind regards,
Patrick

 

[NickF]

I'm actually working on doing some similar fact-finding with an OLD system that's all flash. I am very interested in what you find with IX. I wish I had some meaningful advice, to give, but all-flash ZFS systems are new to me as well.

 

[sretalla]

A couple of points I notice in your testing:

The first test was way too short and with a single thread and therefore not sufficient to see anything that matters.

The second test was better, using 10 threads, but...

Using random as the source will likely bottleneck at the CPU as you saw.

You need to use an input file as the source that you prepared earlier to have random zeros and ones to eliminate that bottleneck (or at leat not test it at the same time as the other components).

I have seen several other examples of good performance in the forums when using fio and pre-prepared random file inputs.

 

[ChrisRJ]

Adding to what @sretalla wrote, the critical part of performance testing is to find a scenario that is representative of the actual workload. That is far(!) from trivial. Spending a couple of hours on this is time is well spent. In addition, creating that simulated workload may prove to be equally challenging. I know this does not solve your problem directly. But there is no easy solution for this kind of situation.

Good luck!

 

[NugentS]

err - how have you got the pool configured? Mirrors, RAIDZn?