Still on FreeNAS-11.3-U5 and looking to push performance

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
Hi all!

Looking to ensure I'm where I "should" be with my hardware performance. I recently came accross some Tech Talkers talking about Tunables within Freenas and getting major performance boots. Trying to see if they are still valid and something I should be looking into.

System Specs.
Intel i3-8100 CPU @ 3.60GHz (on average, less than 2% usage)
64GB memory
All Flash (SSD) Pool consisting of 8 mirrored V-Dev's
1x Log Drive which is a optane SSD 900P
2x Cache drives that are NVME.
2x 10GBE interfaces that are in a LAG for 20Gbe connections.

My use case is ESXi only datastore that is still NFS version 3. Each ESXi server is connected via 10GBE

Storage network is a flat /24 network among all hosts.

Wondering if the following tunables would help improve the below benchmarks.
references
Code:
    kern.ipc.maxsockbuf = 16777216
    net.inet.ip.intr_queue_maxlen = 2048
    net.inet.tcp.recvbuf_inc = 524288
    net.inet.tcp.recvbuf_max = 16777216
    net.inet.tcp.recvspace = 4194304
    net.inet.tcp.sendbuf_inc = 32768
    net.inet.tcp.sendbuf_max = 16777216
    net.inet.tcp.sendspace = 2097152
    net.route.netisr_maxqlen = 2048


Benchmarks
Code:
root@freenas[~]# dd if=/dev/zero of=/mnt/pool/testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 7.786966 secs (5386313495 bytes/sec)


and from within a VM
freenas.png
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
The problem here is that you're maxing out your 10G connection. A LAG will not magically make 2 10G connections a 20G one, it just offers either resilience or more total/aggregate bandwith.
If you start a second process generating traffic then the aggregate performance likely would be higher

The tunables mostly cater for faster networking, thats not your issue.

You could move to nfs 4 and dual link (mount with 2 ips) the FreeNas box (not on LAG, put them on separate vlans), to potentially increase total performance
 

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
I did run two test on two VM's on seperate hosts the same time. Looking at the reports, I think I hit a CPU limit. Reporting within FreeNAS shows I hit 9GPS. Guessing this means my hardware can perform to the limit of the network?
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Well without any screenshots to look at its hard to say, but its possible o/c.
 

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
what screen shots and I show you?
 

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
top ideally while you run two concurrent tests
Still working on getting this setup. does the local DD showing above help with this? are you trying to confirm If i'm CPU bound?
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
A single threaded process may utilize 100% of one core but leaving others free; an aggregated vierw of the utilization will show sgnificant free resources while you're still cpu bound... t
 

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
hopefully this is what you are looking for. i took the TOP screen capture at the lowest point
 

Attachments

  • 2021-02-26_17-51-41.png
    2021-02-26_17-51-41.png
    84.4 KB · Views: 149
  • 2021-02-26_17-54-30.png
    2021-02-26_17-54-30.png
    44.8 KB · Views: 153
  • 2021-02-26_17-54-49.png
    2021-02-26_17-54-49.png
    54.1 KB · Views: 156

orddie

Contributor
Joined
Jun 11, 2016
Messages
104
here is the FreeNas side of things.
 

Attachments

  • 2021-02-26_18-07-00.png
    2021-02-26_18-07-00.png
    628 KB · Views: 161
  • 2021-02-26_18-07-15.png
    2021-02-26_18-07-15.png
    601.5 KB · Views: 143
  • 2021-02-26_18-09-26.png
    2021-02-26_18-09-26.png
    1.3 MB · Views: 149

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Well if you look at the top output you're at 350% cpu usage , so basically 3,5 cores@100%

I would say thats indeed close to maximum what the CPU can do, so at this point thats your limit.
Your network seems to have some headroom, so you might be able to squeeze some more out of it if you replace your CPU (8350k maybe) but I don't think thats worth it at this point, your components seem to be well matched.
 
Top