High End NVME CPU requirements?

[firesyde424]

Initial testing with FIO yielded just over 11GB/sec and 100K IO with randrw, which meets our goal but seems a little low.

 

[Davvo]

Initial testing with FIO yielded just over 11GB/sec and 100K IO with randrw, which meets our goal but seems a little low.

Knowing the parameters would be useful to understand those numbers.

 

[firesyde424]

The original command was fio --bs=64k --direct=1 --directory=/mnt/test01 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=8 --ramp_time=10 --runtime=60 --rw=randrw --size=128M

That returned a result of
READ: bw=11.4GiB/s (12.2GB/s), 11.4GiB/s-11.4GiB/s (12.2GB/s-12.2GB/s), io=502MiB (526MB), run=43-43msec
WRITE: bw=11.9GiB/s (12.7GB/s), 11.9GiB/s-11.9GiB/s (12.7GB/s-12.7GB/s), io=522MiB (548MB), run=43-43msec

Then I tried this:

fio --bs=128k --direct=1 --directory=/mnt/test01 --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_base

This was a command a community member asked me to run on our older server in a different forum thread. I thought it would be a good starting point when comparing this system to that older server. The result of this newer server was:

READ: bw=15.5GiB/s (16.6GB/s), 15.5GiB/s-15.5GiB/s (16.6GB/s-16.6GB/s), io=928GiB (997GB), run=60005-60005msec
WRITE: bw=15.5GiB/s (16.6GB/s), 15.5GiB/s-15.5GiB/s (16.6GB/s-16.6GB/s), io=928GiB (997GB), run=60005-60005msec

When I get servers like this, I try to push them as far as I can so the pool config here is 24 vdevs of 1 disk each. Yes, I know that's dangerous, this is just for testing. CPU usage during this test is 30-40%.

 

[firesyde424]

Same command, but change the --rw= to "read" and this happens.
READ: bw=70.6GiB/s (75.8GB/s), 70.6GiB/s-70.6GiB/s (75.8GB/s-75.8GB/s), io=4238GiB (4550GB), run=60002-60002msec

Holy Sh....

 

[Etorix]

And a pure write test?
SSDs are typically very good at 100% reads, good at 100% writes but not-so-good at 50/50 or 70/30 mixed workloads—Optane was the lone exception to maintain performance across any mix.

 

[Davvo]

That randrw is really a good result. As etorix said, if you need more performance you have to go optane (rip).

 

[firesyde424]

There's more. Same command as above but --numjobs=16.
READ: bw=85.8GiB/s (92.1GB/s), 85.8GiB/s-85.8GiB/s (92.1GB/s-92.1GB/s), io=5146GiB (5526GB), run=60002-60002msec
92.1GB/sec or 736.8Gb/sec.

Same command but rw=write
WRITE: bw=16.3GiB/s (17.5GB/s), 16.3GiB/s-16.3GiB/s (17.5GB/s-17.5GB/s), io=978GiB (1050GB), run=60004-60004msec

 

[Davvo]

Are you sure your are not leveraging ARC?

 

[firesyde424]

Are you sure your are not leveraging ARC?

How could I tell that?

 

[Davvo]

How could I tell that?

reboot each time you start a new fio (unless you have edited the standard behaviour this will clear the arc every time) or monitor the arc (ie with arc_summary or by the graphs in the UI).

 

[firesyde424]

reboot each time you start a new fio (unless you have edited the standard behaviour this will clear the arc every time) or monitor the arc (ie with arc_summary or by the graphs in the UI).

Good call. Immediately after a reboot, same command:
READ: bw=41.1GiB/s (44.1GB/s), 41.1GiB/s-41.1GiB/s (44.1GB/s-44.1GB/s), io=2463GiB (2645GB), run=60005-60005msec

 

[Davvo]

Congrats, your ARC works fine!

 

[firesyde424]

Why are you assuming this? CORE should work just as fine with EPYC—and not waste 50% of the available RAM for ARC.

In addition to my prior response, I've run into some more problems. The two biggest are that TrueNAS Core will only detect 16 of the 24 NVME drives, and the drives that are detected are not hot-swappable. Either of those issues are deal breakers. My assumption is that it must be an Intel vs AMD support issue as we do not see either problem on our Intel based NVME servers.

TrueNAS Scale worked out of the box, detecting all 24 drives and allowing for hot-add and hot-removal.

 

[Etorix]

Thanks for this feedback about EPYC with Core.

 

[Davvo]

Honestly that's strange behaviour, afaik FreeBSD doesn't have any issues with EPYC CPUs and should work out of the box.

 

[firesyde424]

Honestly that's strange behaviour, afaik FreeBSD doesn't have any issues with EPYC CPUs and should work out of the box.

Not sure what the deal was, but it was repeatable. I tried a few different versions of TrueNAS core and the behavior was the same each time.

 

[Davvo]

Not sure what the deal was, but it was repeatable. I tried a few different versions of TrueNAS core and the behavior was the same each time.

Might be worth opening a bug report then if you have the time.