NVMe pool tuning

[CookiesLikeWhoa]

Hey all,

Freenas 11.1-4u
2 e5-2623v3
32gb RAM

This might be a bit of a stretch but I was hoping to get some help with regards to tuning an all NVMe pool. I have a supermicro 2028U-TN24R+T with 4 intel 900p and 4 intel p3520.

I was testing both in a striped pool just to see what kind of performance I could get and I was honestly disappointed. When running dd with bs:2048k and compression off I was seeing 1.1-1.3 GB/s writes and 1GB/s reads for both pools. I upped the bs to 1M and performance actually dropped to around 900MB/s. Each drive is capping out at around 300MB/s for read and writes. CPU usage when running the test was at 80% for 2048k and down to 25% for 1M.

Any advice on how to squeeze more out of this?

Thank you in advance!


Sent from my iPhone using Tapatalk

 

[Ericloewe]

How are these SSDs attached? If they're hanging off the PCH, it could be something as silly as the uplink to the CPU bottlenecking things.

 

[CookiesLikeWhoa]

It’s attached through the backplane on the chassis. I know the connections are good. Tested with windows server. Hence me wondering if there is some tuning I need to do.


Sent from my iPhone using Tapatalk

 

[Ericloewe]

For future reference, this system seems to use two backplanes, each with a PLX 9765 PCIe switch configured with a PCIe 3.0 x16 uplink. (Not the more recent all-in-one SATA/SAS/NVMe stuff, like the LSI SAS 9400 and associated backplanes).

How are you running these tests, exactly? What's the exact dd command?

 

[CookiesLikeWhoa]

It’s a single backplane with two plx chips. The second backplane is for two SATA drives in the rear of the chassis if I recall correctly.

I can grab the commands when I get back from work and post them here.

Thank you for the help!


Sent from my iPhone using Tapatalk

 

[zambanini]

which performance mode is enabled within the bios?

 

[CookiesLikeWhoa]

As in? Do I have the cstates enabled or?


Sent from my iPhone using Tapatalk

 

[CookiesLikeWhoa]

So here is the code and some of the results. This is with the 4 Intel 900Ps striped.

I had gstat open in another shell to monitor the drive usage. During the first write test the drives maxed out at about 75% usage, and CPU hit 80% with 1.4GB/s writes. The next test was the reads. Drives hit 60% usage and CPU hit 30% usage with 1.3 GB/s reads. Next write test was done with 1M block size. Drives hit 55% usage during the write test and CPU spiked to 55% then dropped down to 40% with 1.8GB/s writes. After that we have the 1M block size reads. Drives hit 50% usage and CPU was at 20% usage with 1.36GB/s reads.

Code:

root@freenas:~ # dd if=/dev/zero of=/mnt/test/tmp.dat bs=2048k count=80k
81920+0 records in
81920+0 records out
171798691840 bytes transferred in 120.296428 secs (1428127962 bytes/sec)

root@freenas:~ # dd if=/mnt/test/tmp.dat of=/dev/null bs=2048k count=80k
81920+0 records in
81920+0 records out
171798691840 bytes transferred in 131.082277 secs (1310617234 bytes/sec)

root@freenas:~ # dd if=/dev/zero of=/mnt/test/tmp.dat bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 23.600602 secs (1819854979 bytes/sec)

root@freenas:~ # dd if=/mnt/test/tmp.dat of=/dev/null bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 31.544821 secs (1361544365 bytes/sec)

Next up is the P3520. I didn't bother to do the 2048K block size since it seemed that the 1M block size worked better for this one. First test was writes which maxed at 40% drive usage 40% CPU usage and 1.95GB/s writes. Then did the read test and saw 95% drive usage with 10% CPU usage and 900MB/s reads.

Code:

root@freenas:~ # dd if=/dev/zero of=/mnt/test1/tmp.dat bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 21.942736 secs (1957352666 bytes/sec)

root@freenas:~ # dd if=/mnt/test1/tmp.dat of=/dev/null bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 47.276556 secs (908477220 bytes/sec)

And finally all 8 Drives striped together. CPU usage was never above 5-10% for either test. The write test saw all the drives hit around 20-30% usage and 1.9GB/s, while the read test saw the p3520's hit 90% usage while the 900p's were sitting at 15% and 1.4GB/s.

Code:

root@freenas:~ # dd if=/dev/zero of=/mnt/test/tmp.dat bs=1M count=50k
51200+0 records in
51200+0 records out
53687091200 bytes transferred in 28.401039 secs (1890321400 bytes/sec)

root@freenas:~ # dd if=/mnt/test/tmp.dat of=/dev/null bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 30.062220 secs (1428692634 bytes/sec)

So this is where I stand. The 900P's are capable of nearly 10GB/s reads and 8GB/s writes when striped, yet I'm barely seeing 20% of that. The P3520's are capable 6.4GB/s reads and 5.2GB/s writes when striped yet I'm seeing around 40% of that. Odd thing is the numbers are really close to each other no matter how I set up the drives.

So just for kicks I decided to do just a single 900P pool and see what the performance was. And guess what? It was exactly the same as if it was a striped pool! So it seems I'm only getting one drives performance out of a pool...No idea what gives about that.

Code:

root@freenas:~ # dd if=/dev/zero of=/mnt/test/tmp.dat bs=1M count=50k
51200+0 records in
51200+0 records out
53687091200 bytes transferred in 36.308151 secs (1478651213 bytes/sec)

root@freenas:~ # dd if=/mnt/test/tmp.dat of=/dev/null bs=1M count=40k
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 32.836506 secs (1307985465 bytes/sec)

 

[Stux]

Guessing CPU/memory bottleneck.

 

[CookiesLikeWhoa]

Possibly, but wouldn't the CPU be hitting 100% during these tests? Even then, I don't know how much more CPU I can throw at it. It has 8 cores, 16 threads running at 3.0Ghz. I have two E5-2689V4's used in an ESXi host that are 10 core 3.1Ghz CPU's that I could try but that seems like an awful lot of CPU.

RAM wise it's running 32GBs of Samsung DDR4 2400.

When I run the same set up on Windows Server 2016 I can get 9GB/s reads 7.75GB/s writes with the 900P's and 6GB/s reads and 3GB/s writes with the p3520's (locally). Over the network the Windows' Server gives 1.3GB/s read/writes on a 10Gb/s and I haven't tested on my 40Gb/s network yet. The same set up on FreeNAS will give 900MB/s reads and 350MB/s writes on 10Gb/s network. Both of these are through iSCSI. So I'm leaning towards something I'm missing in FreeNAS. Either I've managed to really miss on configuring the pool or there is tuning I need to learn how to do.