Donny Davis
Contributor
- Joined
- Jul 31, 2015
- Messages
- 139
I am hoping someone can help me understand how to extract the absolute best performance possible from my system. The purpose of the system is to function as my primary storage array for NFS based home shares. The one I have is ageing, and needs to be upgraded. After careful thought and consideration on what the system needs to perform its absolute best for its purpose, I had to custom build something that does exactly what i need it to at light speeds. So comes to my question.
All test are performed locally, NFS is not yet in the equation. I have 12 drives in 2 RAIDz2 (6x6)
Why when my PCIE NVME drive performs as a SLOG, does it not write as fast as when it functions as a normal drive.
To give an example, here is what the SLOG device looks when used as a drive in its own pool. Which yields performance in line with the rest of the setup(network).
However when its placed in the larger pool full of spinny rust, this is what fio has to say
I am pretty sure I am misunderstanding how a SLOG is used, or how to configure things to do what I am looking for.
What I want to do is send all writes to my SLOG, which in turn should give me the performance of the raw device (nearly).
Its entirely possible I have this all wrong, and if that is the case, please help me understand.
Thanks
~Donny
All test are performed locally, NFS is not yet in the equation. I have 12 drives in 2 RAIDz2 (6x6)
Why when my PCIE NVME drive performs as a SLOG, does it not write as fast as when it functions as a normal drive.
To give an example, here is what the SLOG device looks when used as a drive in its own pool. Which yields performance in line with the rest of the setup(network).
Code:
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64 fio-3.0 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [w(1)][93.8%][r=0KiB/s,w=417MiB/s][r=0,w=107k IOPS][eta 00m:02s] test: (groupid=0, jobs=1): err= 0: pid=14342: Sun Sep 23 14:47:04 2018 write: IOPS=34.5k, BW=135MiB/s (141MB/s)(4096MiB/30413msec) bw ( KiB/s): min=54818, max=416429, per=95.60%, avg=131839.60, stdev=80313.94, samples=60 iops : min=13704, max=104109, avg=32959.57, stdev=20078.63, samples=60 cpu : usr=3.02%, sys=37.72%, ctx=334582, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4096MiB (4295MB), run=30413-30413msec
However when its placed in the larger pool full of spinny rust, this is what fio has to say
Code:
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64 fio-3.0 Starting 1 process Jobs: 1 (f=1): [w(1)][90.9%][r=0KiB/s,w=209MiB/s][r=0,w=53.5k IOPS][eta 00m:10s] test: (groupid=0, jobs=1): err= 0: pid=17178: Sun Sep 23 14:59:00 2018 write: IOPS=10.6k, BW=41.2MiB/s (43.2MB/s)(4096MiB/99310msec) bw ( KiB/s): min= 1872, max=488836, per=92.93%, avg=39248.60, stdev=40088.66, samples=198 iops : min= 468, max=122209, avg=9811.83, stdev=10022.19, samples=198 cpu : usr=1.07%, sys=9.28%, ctx=829506, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=41.2MiB/s (43.2MB/s), 41.2MiB/s-41.2MiB/s (43.2MB/s-43.2MB/s), io=4096MiB (4295MB), run=99310-99310msec
I am pretty sure I am misunderstanding how a SLOG is used, or how to configure things to do what I am looking for.
What I want to do is send all writes to my SLOG, which in turn should give me the performance of the raw device (nearly).
Its entirely possible I have this all wrong, and if that is the case, please help me understand.
Thanks
~Donny
Last edited: