Brezlord
Contributor
- Joined
- Jan 7, 2017
- Messages
- 189
I'm having a hard time understanding whats going on with the flash pool below.
I have attached the data sheet of the SSDs I'm using. The SLOG is an Intel Optane 900p 280GB PCI card version.
The SSD data sheet states that Write IOPS (max IOPS, random 4K) 24,000 and this is what I'm getting roughly. Every test I do matches the data sheet, but the data sheet is for a single disk I have 12 disks arranged in 6 x 2 way mirrors so I would of thought that the speed would of scaled more than a single disk. Full spec of server in my signature.
Test done inside a CentOS 7 VM over NFS, sync.
Test done inside a CentOS 7 VM over NFS, sync.
As can be seen the NVMe SLOG has no detrimental impact when in play. The SLOG shows 12% busy in gstat while the disks are at near 100% and only get 1200 IO/s.
Are there any tunables I should be looking at for an all flash pool. Surely this pool should preform better or am I wrong?
Code:
root@nas1:~ # zpool status vol3 pool: vol3 state: ONLINE scan: scrub repaired 0B in 00:04:39 with 0 errors on Sun Oct 25 00:04:41 2020 config: NAME STATE READ WRITE CKSUM vol3 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/5dcdd488-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5ee2636b-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gptid/5f604727-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5fcbc4a4-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gptid/5fdf2d58-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5fd6316a-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 gptid/5e0f3c1f-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5fe2580d-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 gptid/5e9980e5-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5f7733ed-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 gptid/5e57b6f6-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 gptid/5fbf3133-f99c-11ea-aab7-90e2ba3a89c4 ONLINE 0 0 0 logs gptid/b0ceec12-15c0-11eb-85d6-00074333ba50 ONLINE 0 0 0 errors: No known data errors
I have attached the data sheet of the SSDs I'm using. The SLOG is an Intel Optane 900p 280GB PCI card version.
The SSD data sheet states that Write IOPS (max IOPS, random 4K) 24,000 and this is what I'm getting roughly. Every test I do matches the data sheet, but the data sheet is for a single disk I have 12 disks arranged in 6 x 2 way mirrors so I would of thought that the speed would of scaled more than a single disk. Full spec of server in my signature.
Test done inside a CentOS 7 VM over NFS, sync.
Code:
[root@cloud ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.7 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=78.0MiB/s][r=0,w=20.2k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=5120: Sun Oct 25 12:37:45 2020 write: IOPS=18.5k, BW=72.4MiB/s (75.9MB/s)(4096MiB/56603msec) bw ( KiB/s): min=56352, max=92368, per=99.91%, avg=74029.83, stdev=4936.87, samples=113 iops : min=14088, max=23092, avg=18507.44, stdev=1234.26, samples=113 cpu : usr=9.99%, sys=28.69%, ctx=376272, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=72.4MiB/s (75.9MB/s), 72.4MiB/s-72.4MiB/s (75.9MB/s-75.9MB/s), io=4096MiB (4295MB), run=56603-56603msec Disk stats (read/write): dm-0: ios=0/1043192, merge=0/0, ticks=0/3570970, in_queue=3571337, util=99.93%, aggrios=18/1043212, aggrmerge=0/5394, aggrticks=122/3569746, aggrin_queue=3569753, aggrutil=99.91% xvda: ios=18/1043212, merge=0/5394, ticks=122/3569746, in_queue=3569753, util=99.91%
Test done inside a CentOS 7 VM over NFS, sync.
Code:
[root@cloud ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.7 Starting 1 process Jobs: 1 (f=1): [w(1)][98.3%][r=0KiB/s,w=70.1MiB/s][r=0,w=17.9k IOPS][eta 00m:01s] test: (groupid=0, jobs=1): err= 0: pid=5232: Sun Oct 25 12:41:34 2020 write: IOPS=18.1k, BW=70.9MiB/s (74.3MB/s)(4096MiB/57785msec) bw ( KiB/s): min=64328, max=85800, per=99.85%, avg=72477.19, stdev=3393.64, samples=115 iops : min=16082, max=21450, avg=18119.23, stdev=848.40, samples=115 cpu : usr=9.92%, sys=28.96%, ctx=384178, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=70.9MiB/s (74.3MB/s), 70.9MiB/s-70.9MiB/s (74.3MB/s-74.3MB/s), io=4096MiB (4295MB), run=57785-57785msec Disk stats (read/write): dm-0: ios=1/1045455, merge=0/0, ticks=2/3650287, in_queue=3650931, util=99.92%, aggrios=19/1043307, aggrmerge=0/5363, aggrticks=46/3643350, aggrin_queue=3643369, aggrutil=99.90% xvda: ios=19/1043307, merge=0/5363, ticks=46/3643350, in_queue=3643369, util=99.90%
As can be seen the NVMe SLOG has no detrimental impact when in play. The SLOG shows 12% busy in gstat while the disks are at near 100% and only get 1200 IO/s.
Are there any tunables I should be looking at for an all flash pool. Surely this pool should preform better or am I wrong?