Dell R730xd slow IOPS/transfer rate with any drive

[Ericloewe]

I've been skeptical of the HBA330 for awhile. People keep telling me that it is just fine, except that it uses the mrsas driver.

I make no bold claims, but the HBA330 does use the proper mpr driver. I checked that myself.

 

[Ericloewe]

This thing has 384 GB of memory, arranged in 16GB sticks.

I haven't checked the specifics in a while, but the IMC is throttled way down when using three DIMMs per channel (additional load on the memory bus means that the signals can't slew quickly enough). Can you try this out with just 128GB installed to see if there's any improvement at all?

At this point I'm out of ideas on this R730 server, and I've got a couple of R630s,

Honestly not too surprising, since the motherboard is the same. Since you're not too likely to be hitting bad hardware...

Other semi-crazy things to try:

1. Connect only one of the SFF-8643 connectors to the backplane (if you're using expander backplanes, as would be the case on an R730xd and on 10-bay R630s)
2. Remove one CPU (to try and exclude NUMA issues)

 

[JoeAtWork]

Hello all,

I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k 4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128

I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.

Any idea what is going on here?

I have a Dell R720xd with sata ssd's from Micron and Samsung dual proc with 384gb of ram and retail lsi sas hba's. I can test your same command to see how ours fare. Have you disabled hyper-threading? Are the bios settings the default? Are you booting from BIOS or UEFI mode?

How large of file does your fio command generate and how long does it take, do you have the output for me to compare?

Thanks,
Joe

 

[JoeAtWork]

Hello all,

I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k 4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128

I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.

Any idea what is going on here?

This is on my 720xd with 192gb RAM and mechanical drives running 13.0-U3.1, 3tb 7.2k - 24 mirrors.

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k

results :
4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128 ... fio-3.28 Starting 8 processes 4ktest: Laying out IO file (1 file / 4096MiB) Jobs: 8 (f=8): [r(8)][100.0%][r=2700MiB/s][r=691k IOPS][eta 00m:00s] 4ktest: (groupid=0, jobs=8): err= 0: pid=68876: Mon Nov 21 16:22:02 2022 read: IOPS=653k, BW=2549MiB/s (2673MB/s)(32.0GiB/12856msec) clat (usec): min=3, max=82712, avg=11.32, stdev=90.57 lat (usec): min=3, max=82712, avg=11.39, stdev=90.57 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 16], | 99.00th=[ 77], 99.50th=[ 153], 99.90th=[ 347], 99.95th=[ 482], | 99.99th=[ 1074] bw ( MiB/s): min= 453, max= 2858, per=100.00%, avg=2552.64, stdev=57.47, samples=200 iops : min=116058, max=731810, avg=653474.08, stdev=14712.96, samples=200 lat (usec) : 4=0.04%, 10=82.10%, 20=14.16%, 50=2.13%, 100=0.61% lat (usec) : 250=0.76%, 500=0.16%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 100=0.01% cpu : usr=10.29%, sys=88.62%, ctx=10955, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8388608,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=2549MiB/s (2673MB/s), 2549MiB/s-2549MiB/s (2673MB/s-2673MB/s), io=32.0GiB (34.4GB), run=12856-12856msec

 

[JoeAtWork]

Hello all,

I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k 4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128

I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.

Any idea what is going on here?

Here is my 720xd dual E5-2660v2, 384gb DDR3 ram, TrueNAS 12.0-U8 with 12-4tb samsung sata drives in a 6 vdev mirror, compression off, atime off and sync set to standard.

4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128 ... fio-3.27 Starting 8 processes 4ktest: Laying out IO file (1 file / 4096MiB) Jobs: 8 (f=8): [r(8)][100.0%][r=2530MiB/s][r=648k IOPS][eta 00m:00s] 4ktest: (groupid=0, jobs=8): err= 0: pid=23762: Mon Nov 21 16:38:31 2022 read: IOPS=650k, BW=2541MiB/s (2664MB/s)(32.0GiB/12896msec) clat (usec): min=3, max=1294, avg=11.30, stdev=16.90 lat (usec): min=3, max=1294, avg=11.38, stdev=16.90 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 10], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 57], 99.50th=[ 106], 99.90th=[ 273], 99.95th=[ 343], | 99.99th=[ 486] bw ( MiB/s): min= 2422, max= 2797, per=100.00%, avg=2565.07, stdev=12.20, samples=198 iops : min=620049, max=716061, avg=656654.96, stdev=3123.71, samples=198 lat (usec) : 4=0.01%, 10=70.80%, 20=25.85%, 50=2.12%, 100=0.51% lat (usec) : 250=0.59%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=9.37%, sys=90.61%, ctx=1958, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8388608,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=2541MiB/s (2664MB/s), 2541MiB/s-2541MiB/s (2664MB/s-2664MB/s), io=32.0GiB (34.4GB), run=12896-12896msec

 

[JoeAtWork]

As for the reset question, I've fully reset this server including NVRAM_CLR jumper. None of the settings or hardware combinations I've attempted has shown any improvement. I even purchased a regular off-the-shelf Broadcom 9300-8i HBA and ran it in one of the slots, and the same speed issue was there. Even bypassing the backplane and connecting the drives directly to the controller was the same.

At this point I'm out of ideas on this R730 server, and I've got a couple of R630s, so I used one of them for testing. I put the HBA330 in one, and placed 8 of these drives in it and installed TrueNAS 13. It produced the same performance numbers as the R730.

How is it possible that multiple 13th gen Dell servers have this problem?

I would go buy a Retail Samsung 980 nvme Pro M.2 and see how that preforms, then you know your server, slots motherboard and CPU are all OK. I have had issues with Dell storage slots and cards for many years, I hate them and just consider that PCIe slot given up to the Dell Gods and never to be useful for anything other than Microsoft Windows.

 

[jgreco]

I make no bold claims, but the HBA330 does use the proper mpr driver. I checked that myself.

My whole world is thrown into disarray. Do I really need to buy one of every card to test stuff? Auugghh.

 

[Ericloewe]

My whole world is thrown into disarray.

Dell not aggressively pushing hardware RAID with juicy margins? Yeah, it's a shocker.

Don't worry, some bits of your old world remain: on Gen 14.5 (R6515 and R7515), which use the Gen 13/14 storage mezzanine, you used to have to trick the online configurator:

1. Start with the SATA/SAS/NVMe option (On 10-bay R6515s, it's always a simple backplane that takes 8 SATA/SAS over a pair of SFF-8643s and 10 SFF-8643s for NVMe). Configurator only gives you the H740P as an option.
2. Switch to NVMe only. Price goes up because reasons.
3. Switch back to SATA/SAS/NVMe. Price went down and the H740P has been replaced with an HBA330.

They "fixed" this a few months back, so that you wouldn't know that an HBA is an option without digging through endless paperwork.

 

[JoeAtWork]

My whole world is thrown into disarray. Do I really need to buy one of every card to test stuff? Auugghh.

for those that are paranoid they can sacrafice one PCIe slot and put a NVME disk in.

I also use this NVME disk to copy files to volumes I am testing, I fill the volume and watch all vdevs fill up.

In this case here it would rule out any system issues and point to the crappy perc.

 

[Ericloewe]

Four of the disk bays on the R730xd should also be wired for PCIe x4, right? They can be easily lit up by adding a redriver/retimer board on a x16 slot and suitable SFF-8643 cables.

 

[HoneyBadger]

This is on my 720xd with 192gb RAM and mechanical drives running 13.0-U3.1, 3tb 7.2k - 24 mirrors.

Code:

4ktest: (groupid=0, jobs=8): err= 0: pid=68876: Mon Nov 21 16:22:02 2022
  read: IOPS=653k, BW=2549MiB/s (2673MB/s)(32.0GiB/12856msec)

Here is my 720xd dual E5-2660v2, 384gb DDR3 ram, TrueNAS 12.0-U8 with 12-4tb samsung sata drives in a 6 vdev mirror, compression off, atime off and sync set to standard.

Code:

4ktest: (groupid=0, jobs=8): err= 0: pid=23762: Mon Nov 21 16:38:31 2022
  read: IOPS=650k, BW=2541MiB/s (2664MB/s)(32.0GiB/12896msec)

This is the kind of performance I would expect to see, as the read workload will be coming exclusively from ARC.

@eclipse5302 The fact that you're seeing this on multiple 13G Dell systems is really puzzling me, unless there's something in the stock Dell/EMC BIOS settings that does Very Bad Things to CPU/MEM throughput. You shouldn't even be hitting your storage controller or disk subsystem at all.

Trying to think of a comparative CPU/MEM benchmark that we can run to narrow this down; we don't have the PHP command-line that I'd use for something like the Phoronix test suite. I'll also see if I can commandeer a dual v4 Xeon system (unfortunately not your exact hardware though) to see if i get similar results under TrueNAS.

 

[JoeAtWork]

This is the kind of performance I would expect to see, as the read workload will be coming exclusively from ARC.

@eclipse5302 The fact that you're seeing this on multiple 13G Dell systems is really puzzling me, unless there's something in the stock Dell/EMC BIOS settings that does Very Bad Things to CPU/MEM throughput. You shouldn't even be hitting your storage controller or disk subsystem at all.

Trying to think of a comparative CPU/MEM benchmark that we can run to narrow this down; we don't have the PHP command-line that I'd use for something like the Phoronix test suite. I'll also see if I can commandeer a dual v4 Xeon system (unfortunately not your exact hardware though) to see if i get similar results under TrueNAS.

memtest x86 would tell us the speed of the ram and cpu info, it could be miss matched CPU's, I have had vendors try to send two different step levels, and that vendor was Dell on a new box for VMware....

 

[JoeAtWork]

sync is not set to always on your pools????

 

[HoneyBadger]

sync is not set to always on your pools????

The fio command being used here and the performance bottleneck being tested is 100% reads - sync writes shouldn't have any impact other than the time needed to build the initial test file.

 

[StorageCurious]

I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.

Any idea what is going on here?

No idea, but I'm running a R730XD with an HBA330 (upgraded to latest Dell firmware I could find, not an LSI one), 2 x E5-2680 v4 @ 2.40GHz, 768GB of RAM.

I ran the test on my boot drive (a pair of mirrored cheap-ish Kingston's SSD) and while I don't quite know how to read those results, I believe it doesn't look like I'm hitting that limitation you experienced. Hoping this helps you.

4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128 4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128 ... fio-3.28 Starting 8 processes 4ktest: Laying out IO file (1 file / 4096MiB) Jobs: 3 (f=3): [_(1),r(1),_(3),r(1),_(1),r(1)][93.3%][r=2519MiB/s][r=645k IOPS][eta 00m:01s] 4ktest: (groupid=0, jobs=8): err= 0: pid=14051: Thu Nov 24 14:50:35 2022 read: IOPS=615k, BW=2403MiB/s (2520MB/s)(32.0GiB/13635msec) clat (nsec): min=1974, max=4290.0k, avg=12224.29, stdev=37554.71 lat (nsec): min=2000, max=4290.1k, avg=12259.91, stdev=37555.56 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 20], | 99.00th=[ 93], 99.50th=[ 223], 99.90th=[ 562], 99.95th=[ 734], | 99.99th=[ 1172] bw ( MiB/s): min= 1390, max= 2845, per=100.00%, avg=2438.07, stdev=49.36, samples=203 iops : min=355996, max=728479, avg=624144.00, stdev=12635.88, samples=203 lat (usec) : 2=0.01%, 4=0.17%, 10=77.69%, 20=17.28%, 50=3.14% lat (usec) : 100=0.83%, 250=0.43%, 500=0.32%, 750=0.10%, 1000=0.03% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01% cpu : usr=5.31%, sys=94.67%, ctx=1836, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8388608,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=2403MiB/s (2520MB/s), 2403MiB/s-2403MiB/s (2520MB/s-2520MB/s), io=32.0GiB (34.4GB), run=13635-13635msec zsh: no matches found: (g=0):

 

[HoneyBadger]

while I don't quite know how to read those results, I believe it doesn't look like I'm hitting that limitation you experienced

Code:

4ktest: (groupid=0, jobs=8): err= 0: pid=14051: Thu Nov 24 14:50:35 2022
  read: IOPS=615k, BW=2403MiB/s (2520MB/s)(32.0GiB/13635msec)

You're flying along just fine there at 615K IOPS.

@eclipse5302 I'm more and more puzzled. Haven't gotten a chance to hijack a v4 system of my own yet, I'll have to try to prioritize that.

 

[JoeAtWork]

Hello all,

I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k 4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128

I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.

Any idea what is going on here?

sync=standard or sync=always?

 

[fredbourdelier]

I know I'm parachuting in on an old thread...

Just ran the same fio test on a Gen3 R730xd dual 14core 3rd gen E5-2690 @ 2.6Ghz. Like the other tester, lots of RAM (maxed out, Crucial low latency 3200MT/S ECC registered RAM, 768GB). NOT using the recommended HBA330, using the 730 Mini in HBA mode (I know, the replacement HBA330's on order). Ran the test on a pair of Seagate Nytro SAS SSD, which Seagate has proven will saturate any controller up to 24GB/s I/O:

Same. Exact. Numbers. Average 150K IOPS. OK. So, it's not even the HBA330. It's not the drives. It's not those particular Xeons (though it could still be Xeons in general, or NUMA, or the Dell memory arbitration functions). It must be the bus architecture, or some kind of throttle/sync to keep everything on the same clock? Whatever it is, Gen13 DELL are not working in this environment as they should be.

Does anyone have throughput numbers for a Windows system on similar test?

 

[eclipse5302]

What is your record size set for? With 128k being the recommended size, performance seemed to be capped as you found. I ended up going to 16k record size to get decent performance.

With 16k record size:
fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k

4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=2822MiB/s][r=722k IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=8): err= 0: pid=10750: Mon Apr 10 07:23:27 2023
read: IOPS=719k, BW=2810MiB/s (2947MB/s)(32.0GiB/11660msec)
clat (usec): min=2, max=2670, avg=10.65, stdev=21.41
lat (usec): min=2, max=2670, avg=10.68, stdev=21.41
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 8],
| 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 18],
| 99.00th=[ 118], 99.50th=[ 157], 99.90th=[ 306], 99.95th=[ 375],
| 99.99th=[ 494]
bw ( MiB/s): min= 2749, max= 2902, per=100.00%, avg=2814.20, stdev= 5.04, samples=176
iops : min=703811, max=743039, avg=720430.77, stdev=1291.25, samples=176
lat (usec) : 4=0.13%, 10=84.69%, 20=11.20%, 50=2.38%, 100=0.59%
lat (usec) : 250=0.83%, 500=0.17%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=5.81%, sys=94.14%, ctx=4195, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=8388608,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
READ: bw=2810MiB/s (2947MB/s), 2810MiB/s-2810MiB/s (2947MB/s-2947MB/s), io=32.0GiB (34.4GB), run=11660-11660msec


After that, NFS performance was still terrible (compared to identical settings on the R720xd). iSCSI performance is excellent. So now we're running iSCSI.

 

[Ericloewe]

With 128k being the recommended size

Not recommended, a default that doesn't suck too much for most cases. This is something that should absolutely be tuned according to the workload.