mstang1988
Contributor
- Joined
- Aug 20, 2012
- Messages
- 102
There are a lot more nuances to performance than the above, but yes, I agree. VM benchmarking isn't standard practice as the variability induced but I'll also disagree, it's not pointless. More on that in a second.
Typically performance is run on bare metal with a known (and consistent) hardware configuration. A statistically meaningful run cycle is determined and usually you choose to lop off the outliers and standard deviation intervals (not just the high/lows). You also typically disable caching to not only minimize it's affects but to isolate it out as well.
A few other bad things about my benchmarking, I'm not benchmarking directly from an NFS client and I'm instead utilizing a VM disk (without data) and windows. This introduces a few more uncontrolled variables. I'm also not assessing across different NFS clients to identify if the bottle necks are on the client or server side.
With all that said, I've only given Freenas 32GB of RAM (awaiting my replacement DIMM's) and windows 8GB. I don't off the top of my head know how much NFS caching is doing but it's pretty minimal. I see a performance decrease between 512M and 1G and I think this is likely the ESX NFS cache. 16GB seems sufficiently large in this environment to have little to no meaningful impact on performance. Also, with sync writes there should be no cache.
ESX does sync writes by default for NFS.
Aside from all the test methodology issues there is meaningful data to be had from the VM's here. We aren't just at the mercy of the hypervisor (ready time represents this well).
Without having > 10Gb/s line speeds I can't test the throughput of sequential I/O. If I keep it local to the host I'm not bound by my network speeds and latencies (although they still come into play based on the virtual NIC/switch performance as well).
I also get to understand what the real world performance is of VM's which will be utilizing this datastore. This is equally important as I won't be utilizing bare metal clients in many cases. Data like this can also be used to pinpoint bugs that may impact our real world scenarios. We can see that things like TSO offload on VMXNET3 devices affects us due to a bug in freebsd.
Odd results like the above where we see nearly an order of magnitude performance difference (in the wrong way) also implies there is some investigation warranted. It may be VM related but there is likely some system behavior present instead.
I have to do some rewiring of one of my machines to test bare metal performance (And I might do so) but the above tests also tell me that I'll stick to a SLOG and sync right based on the current client/usage.
If I were to contrive the test I'd want to try with unlimited time/resources:
- Bare Metal 5900x Ryzen single 10Gb-T NIC (my home infrastructure doesn't really have a great way of running fiber)
- Linux installs of ubuntu, RH and a clean install of ESX
- Disablement of all client caches (and during on sync writes for clients)
- 128GB RAM with 64GB being utilized as a RAM disk for read/writes to/from TrueNAS
- Repeat each experiment 50x and have two data points filtering within 2 standard deviations (95%) and within 3 standard deviations (99.7%)
- Validate with various NFS block sizes
- NFSv3 vs NFSv4
- and go through a significant amount more of NFS tuning
I'm sure I've missed a few things but I really don't have time to get into this level of effort in a home project.
