Still on FreeNAS-11.3-U5 and looking to push performance

[orddie]

Hi all!

Looking to ensure I'm where I "should" be with my hardware performance. I recently came accross some Tech Talkers talking about Tunables within Freenas and getting major performance boots. Trying to see if they are still valid and something I should be looking into.

System Specs.
Intel i3-8100 CPU @ 3.60GHz (on average, less than 2% usage)
64GB memory
All Flash (SSD) Pool consisting of 8 mirrored V-Dev's
1x Log Drive which is a optane SSD 900P
2x Cache drives that are NVME.
2x 10GBE interfaces that are in a LAG for 20Gbe connections.

My use case is ESXi only datastore that is still NFS version 3. Each ESXi server is connected via 10GBE

Storage network is a flat /24 network among all hosts.

Wondering if the following tunables would help improve the below benchmarks.
references

Configure Your Own Jellyfish Fryer All-Flash NAS

We show some of the behind-the-scenes on how we made the Jellyfish Fryer and setup a FreeNAS box that was pushing over 40GbE speeds

www.servethehome.com

How to Tune a NAS for Direct-from-Server Editing of 5K Video

At 45 Drives , we make really large capacity storage servers. When we first started out, our machines were relatively slow, and focused on ...

45drives.blogspot.com

Code:

    kern.ipc.maxsockbuf = 16777216
    net.inet.ip.intr_queue_maxlen = 2048
    net.inet.tcp.recvbuf_inc = 524288
    net.inet.tcp.recvbuf_max = 16777216
    net.inet.tcp.recvspace = 4194304
    net.inet.tcp.sendbuf_inc = 32768
    net.inet.tcp.sendbuf_max = 16777216
    net.inet.tcp.sendspace = 2097152
    net.route.netisr_maxqlen = 2048

Benchmarks

Code:

root@freenas[~]# dd if=/dev/zero of=/mnt/pool/testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 7.786966 secs (5386313495 bytes/sec)

and from within a VM

 

[Rand]

The problem here is that you're maxing out your 10G connection. A LAG will not magically make 2 10G connections a 20G one, it just offers either resilience or more total/aggregate bandwith.
If you start a second process generating traffic then the aggregate performance likely would be higher

The tunables mostly cater for faster networking, thats not your issue.

You could move to nfs 4 and dual link (mount with 2 ips) the FreeNas box (not on LAG, put them on separate vlans), to potentially increase total performance

 

[orddie]

I did run two test on two VM's on seperate hosts the same time. Looking at the reports, I think I hit a CPU limit. Reporting within FreeNAS shows I hit 9GPS. Guessing this means my hardware can perform to the limit of the network?

 

[Rand]

Well without any screenshots to look at its hard to say, but its possible o/c.

 

[orddie]

what screen shots and I show you?

 

[Rand]

top ideally while you run two concurrent tests

 

[orddie]

top ideally while you run two concurrent tests

Still working on getting this setup. does the local DD showing above help with this? are you trying to confirm If i'm CPU bound?

 

[Rand]

A single threaded process may utilize 100% of one core but leaving others free; an aggregated vierw of the utilization will show sgnificant free resources while you're still cpu bound... t

 

[orddie]

hopefully this is what you are looking for. i took the TOP screen capture at the lowest point

 

[orddie]

here is the FreeNas side of things.

 

[Rand]

Well if you look at the top output you're at 350% cpu usage , so basically 3,5 cores@100%

I would say thats indeed close to maximum what the CPU can do, so at this point thats your limit.
Your network seems to have some headroom, so you might be able to squeeze some more out of it if you replace your CPU (8350k maybe) but I don't think thats worth it at this point, your components seem to be well matched.