fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=rw --size=50g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=16 --runtime=120 --group_reporting --output=fiotest-AppPool.txt #### 3 x RZ1 TEST: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16 ... fio-3.33 Starting 16 processes TEST: Laying out IO file (1 file / 51200MiB) TEST: (groupid=0, jobs=16): err= 0: pid=116720: Tue Jan 23 16:02:42 2024 read: IOPS=53.3k, BW=6658MiB/s (6982MB/s)(780GiB/120018msec) clat (usec): min=9, max=492051, avg=85.01, stdev=1071.45 lat (usec): min=9, max=492051, avg=85.19, stdev=1072.31 clat percentiles (usec): | 1.00th=[ 19], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 43], 50.00th=[ 51], 60.00th=[ 60], | 70.00th=[ 70], 80.00th=[ 84], 90.00th=[ 108], 95.00th=[ 137], | 99.00th=[ 545], 99.50th=[ 922], 99.90th=[ 2999], 99.95th=[ 5735], | 99.99th=[31065] bw ( MiB/s): min= 952, max=16929, per=100.00%, avg=6671.67, stdev=152.06, samples=3723 iops : min= 7615, max=135435, avg=53368.28, stdev=1216.34, samples=3723 write: IOPS=53.3k, BW=6661MiB/s (6984MB/s)(781GiB/120018msec); 0 zone resets clat (usec): min=11, max=564552, avg=208.15, stdev=2038.12 lat (usec): min=12, max=564555, avg=211.58, stdev=2049.48 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 47], 20.00th=[ 60], | 30.00th=[ 73], 40.00th=[ 89], 50.00th=[ 113], 60.00th=[ 153], | 70.00th=[ 184], 80.00th=[ 231], 90.00th=[ 322], 95.00th=[ 453], | 99.00th=[ 1106], 99.50th=[ 1680], 99.90th=[ 7832], 99.95th=[18220], | 99.99th=[64750] bw ( MiB/s): min= 861, max=16913, per=100.00%, avg=6672.12, stdev=152.28, samples=3724 iops : min= 6889, max=135306, avg=53377.28, stdev=1218.36, samples=3724 lat (usec) : 10=0.01%, 20=0.70%, 50=30.27%, 100=35.61%, 250=23.81% lat (usec) : 500=6.99%, 750=1.28%, 1000=0.51% lat (msec) : 2=0.55%, 4=0.15%, 10=0.07%, 20=0.03%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=2.36%, sys=43.19%, ctx=3412979, majf=4, minf=5706 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6392874,6395330,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6658MiB/s (6982MB/s), 6658MiB/s-6658MiB/s (6982MB/s-6982MB/s), io=780GiB (838GB), run=120018-120018msec WRITE: bw=6661MiB/s (6984MB/s), 6661MiB/s-6661MiB/s (6984MB/s-6984MB/s), io=781GiB (838GB), run=120018-120018msec #### 4 x Mirrors TEST: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16 fio-3.33 Starting 16 processes TEST: Laying out IO file (1 file / 51200MiB) TEST: (groupid=0, jobs=16): err= 0: pid=120846: Tue Jan 23 16:13:06 2024 read: IOPS=42.7k, BW=5337MiB/s (5596MB/s)(625GiB/120002msec) clat (usec): min=9, max=581372, avg=64.31, stdev=624.14 lat (usec): min=9, max=581372, avg=64.46, stdev=624.24 clat percentiles (usec): | 1.00th=[ 19], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 55], | 70.00th=[ 66], 80.00th=[ 79], 90.00th=[ 100], 95.00th=[ 124], | 99.00th=[ 223], 99.50th=[ 359], 99.90th=[ 922], 99.95th=[ 1418], | 99.99th=[11338] bw ( MiB/s): min= 603, max=18148, per=100.00%, avg=5338.23, stdev=167.10, samples=3807 iops : min= 4827, max=145184, avg=42704.31, stdev=1336.78, samples=3807 write: IOPS=42.7k, BW=5340MiB/s (5599MB/s)(626GiB/120002msec); 0 zone resets clat (usec): min=12, max=592154, avg=304.45, stdev=1920.86 lat (usec): min=12, max=592155, avg=307.19, stdev=1921.22 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 47], 10.00th=[ 61], 20.00th=[ 84], | 30.00th=[ 113], 40.00th=[ 186], 50.00th=[ 255], 60.00th=[ 302], | 70.00th=[ 359], 80.00th=[ 449], 90.00th=[ 586], 95.00th=[ 701], | 99.00th=[ 955], 99.50th=[ 1106], 99.90th=[ 2245], 99.95th=[ 4948], | 99.99th=[39060] bw ( MiB/s): min= 620, max=18323, per=100.00%, avg=5341.30, stdev=167.45, samples=3807 iops : min= 4960, max=146584, avg=42728.87, stdev=1339.54, samples=3807 lat (usec) : 10=0.01%, 20=0.74%, 50=29.73%, 100=27.76%, 250=15.88% lat (usec) : 500=17.73%, 750=6.27%, 1000=1.45% lat (msec) : 2=0.36%, 4=0.04%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=2.14%, sys=34.94%, ctx=3703676, majf=0, minf=1193 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5123361,5126233,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5337MiB/s (5596MB/s), 5337MiB/s-5337MiB/s (5596MB/s-5596MB/s), io=625GiB (672GB), run=120002-120002msec WRITE: bw=5340MiB/s (5599MB/s), 5340MiB/s-5340MiB/s (5599MB/s-5599MB/s), io=626GiB (672GB), run=120002-120002msec #### 2 x RZ2 TEST: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16 ... fio-3.33 Starting 16 processes TEST: Laying out IO file (1 file / 51200MiB) TEST: (groupid=0, jobs=16): err= 0: pid=126591: Tue Jan 23 16:34:59 2024 read: IOPS=44.0k, BW=5495MiB/s (5762MB/s)(644GiB/120001msec) clat (usec): min=9, max=517210, avg=71.93, stdev=725.16 lat (usec): min=9, max=517210, avg=72.07, stdev=725.18 clat percentiles (usec): | 1.00th=[ 18], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 40], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 69], 80.00th=[ 84], 90.00th=[ 111], 95.00th=[ 143], | 99.00th=[ 375], 99.50th=[ 603], 99.90th=[ 1352], 99.95th=[ 1958], | 99.99th=[16057] bw ( MiB/s): min= 569, max=16003, per=100.00%, avg=5507.80, stdev=148.91, samples=3776 iops : min= 4556, max=128026, avg=44060.00, stdev=1191.23, samples=3776 write: IOPS=44.0k, BW=5498MiB/s (5765MB/s)(644GiB/120001msec); 0 zone resets clat (usec): min=11, max=558734, avg=285.64, stdev=1649.22 lat (usec): min=12, max=558736, avg=288.63, stdev=1653.79 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 48], 10.00th=[ 61], 20.00th=[ 83], | 30.00th=[ 111], 40.00th=[ 159], 50.00th=[ 194], 60.00th=[ 255], | 70.00th=[ 326], 80.00th=[ 408], 90.00th=[ 545], 95.00th=[ 701], | 99.00th=[ 1074], 99.50th=[ 1303], 99.90th=[ 2802], 99.95th=[ 7373], | 99.99th=[46924] bw ( MiB/s): min= 594, max=16122, per=100.00%, avg=5510.54, stdev=149.27, samples=3776 iops : min= 4758, max=128979, avg=44081.83, stdev=1194.12, samples=3776 lat (usec) : 10=0.01%, 20=0.76%, 50=28.80%, 100=27.51%, 250=21.78% lat (usec) : 500=14.50%, 750=4.46%, 1000=1.40% lat (msec) : 2=0.69%, 4=0.06%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=2.06%, sys=36.94%, ctx=3856051, majf=0, minf=2711 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5275369,5277810,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5495MiB/s (5762MB/s), 5495MiB/s-5495MiB/s (5762MB/s-5762MB/s), io=644GiB (691GB), run=120001-120001msec WRITE: bw=5498MiB/s (5765MB/s), 5498MiB/s-5498MiB/s (5765MB/s-5765MB/s), io=644GiB (692GB), run=120001-120001msec #### 2 x Mirrors TEST: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16 ... fio-3.33 Starting 16 processes TEST: Laying out IO file (1 file / 51200MiB) TEST: (groupid=0, jobs=16): err= 0: pid=129855: Tue Jan 23 16:41:52 2024 read: IOPS=53.6k, BW=6695MiB/s (7020MB/s)(785GiB/120002msec) clat (usec): min=9, max=498247, avg=54.13, stdev=680.47 lat (usec): min=9, max=498247, avg=54.27, stdev=680.67 clat percentiles (usec): | 1.00th=[ 16], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 39], 50.00th=[ 44], 60.00th=[ 49], | 70.00th=[ 57], 80.00th=[ 67], 90.00th=[ 84], 95.00th=[ 102], | 99.00th=[ 182], 99.50th=[ 258], 99.90th=[ 506], 99.95th=[ 668], | 99.99th=[ 1205] bw ( MiB/s): min= 732, max=17188, per=100.00%, avg=6694.69, stdev=212.62, samples=3824 iops : min= 5858, max=137508, avg=53556.98, stdev=1700.98, samples=3824 write: IOPS=53.6k, BW=6697MiB/s (7023MB/s)(785GiB/120002msec); 0 zone resets clat (usec): min=12, max=511087, avg=239.21, stdev=1255.91 lat (usec): min=13, max=511088, avg=241.85, stdev=1256.20 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 47], 10.00th=[ 58], 20.00th=[ 77], | 30.00th=[ 103], 40.00th=[ 151], 50.00th=[ 188], 60.00th=[ 237], | 70.00th=[ 269], 80.00th=[ 310], 90.00th=[ 453], 95.00th=[ 660], | 99.00th=[ 1037], 99.50th=[ 1123], 99.90th=[ 1319], 99.95th=[ 1532], | 99.99th=[ 6915] bw ( MiB/s): min= 718, max=17313, per=100.00%, avg=6697.25, stdev=212.88, samples=3824 iops : min= 5748, max=138506, avg=53577.49, stdev=1703.03, samples=3824 lat (usec) : 10=0.01%, 20=1.49%, 50=32.52%, 100=27.95%, 250=19.67% lat (usec) : 500=14.09%, 750=2.36%, 1000=1.30% lat (msec) : 2=0.60%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=2.57%, sys=39.09%, ctx=5234864, majf=0, minf=2850 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6426923,6429400,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6695MiB/s (7020MB/s), 6695MiB/s-6695MiB/s (7020MB/s-7020MB/s), io=785GiB (842GB), run=120002-120002msec WRITE: bw=6697MiB/s (7023MB/s), 6697MiB/s-6697MiB/s (7023MB/s-7023MB/s), io=785GiB (843GB), run=120002-120002msec