Slow Nested VM's Performance

[vm-guy]

Hi all
I’m having a strange performance issue which I can’t explain, hopefully someone can help resolving it.

I’ll start with the hardware specs of my Host
Workstation model: Dell Precision T6710
CPU: 2 x Intel Xeon 2670 v2 10c/20HT @ 2.5GHz
RAM: 192GB DDR Quad Channel
SCSI RAID Controller: LSI SAS2 2308
Disk: Samsung Evo 860 1TB SSD
Hosting OS: Windows 10 Pro 21H1
VM Workstation Pro: v16.2.0 build-18760230
TrueNAS Core: v12.0 U6 (Given 2 CPU's and 16GB of RAM, Lots of free memory and the CPU spikes at times)


I basically use above workstation for my "fairly" complex VMware lab (all working though), which I’ll explain shortly. Few things to mention before we start:

1. I’m not a TrueNAS expert, so please be easy on me when it comes into what to do.
2. I have 3 more (different) disks than the one above, but they out of the scope of my issue.
3. Stated Samsung Evo 860 SSD on LSI SAS2 2308, gets the exact same benchmark results compared to any standard onboard SATA3 PC Controller.

The Lab Layout
My lab consists for several primary and nested VM's to simulate Prod environut functionality, all the VM's are running on stated Samsung Evo 860 SSD, the primary VM's list are:
3 x Windows Server (different versions) Acting as Domain Controller, NFS3 and NFS4.1 storage targets
1 x Windows 7 Pro, acting as a management and PowerShell station
12 x virtualized ESXi 7 VMHosts
OpenFiler v2.99 (is primary iSCSI storage target for my nested VM's)
TrueNAS v12.0 U6 (is the secondary iSCSI storage target, I'd like to migrate to)

And nested VM's list are:
vCenter VCSA
8 Mixed Windows and Linux VM's

Issue Description
The purpose of TrueNAS is to be and only be an iSCSI storage target for above 12 virtualized ESXi 7 VMHosts. However I'm currently still using a very old OpenFiler 2.99 as my iSCSI storage target due to performance reasons. But before that let me show you first few benchmark results which may indicate better performance outcome on TrueNAS. Please note that all below tests were performed on the Nested VM's (please see above)

• Cloning 50GB VM, takes 7:18 (mm:ss) on OpenFiler, while it took 1:32 (mm:ss) on TrueNAS (great!)
• Migrating 50GB VM while generating heavy virtual disk IO workload, took 8:01 (mm:ss) on openFiler, while it took 3:42 (mm:ss) on True NAS
• Provisioning 40GB Thick "Eager-zeroed" virtual disk on a VM, took 8:15 (mm:ss) on OpenFiler, while it took 2:54 (mm:ss) on TrueNAS

So the the winner is TrueNAS and I should migrate immediately to it, right? No, somehow the internal file system on my nested Windows 2019 test VM (NTFS) is actually mush slower on TrueNAS than OpenFiler, for example:

• Copying 3GB (15k files) between 2 virtual disks on a nested VM on TrueNAS took 8:28 (mm:ss), while on OpenFiler it took 7:35 (mm:ss) which is not expected!
• Also Crystal Disk Mark 7.0.0 x64 sequential results were 85.56MB/s (Read) and 87.03 MB/s (Write) on OpenFiler, while on TrueNAS I got 20.97MB/s (read) and 35.23MB/s (Write)

Please note that all above tests were done on the same nested VM clone, the only difference is I placed one on OpenFiler and the other one on TrueNAS. Somehow TrueNAS flexes its muscles on nested VM's operations under VMFS (VMware file system) iSCSI LUN's, but once it drills down to the nested VM file system (NTFS in my case) it slows down heavily, despite the fact that everything is running on the same SSD (Samsung), and that OpenFiler is performing much better under the exact same environment.

Any idea? and thanks in advance.