FreeNAS Build with Terrible Performance, please help

[xsparta11x]

Hey Everyone,
So I built a Fairly Robust FreeNAS server and I am getting absolutely terrible performance. So here are the Specs:
Dell R720xd
H710 mini monolithic 5CT6D with LSI 9207-8i P20 IT Mode
128 GB of RAM
12x 4TB SAS 7200RPM HGST Hard Drives
1TB Samsung EVO 970 M.2 NVMe SSD (for Cache)
10Gb Nic
64GB SanDisk Class 10 v3 SD Card for FreeNAS boot
Running FreeNAS- 11.3-U3.2

The server is built for Production VM Storage with VMware. From my research, i discovered that a Mirrored pool is what would be best for my setup. I setup 6 Mirror Polls across the 12 Drives and used the NVMe Drive for Cache (screenshot attached of design). I am sharing this pool via iSCSI to VMware across a 10Gb switch and all devices on 10Gb.

I setup the 10GB network link aswell as the primary NIC. the major problem I am facing is I am only getting in the neighborhood of 50 MiB/s on speed MAX. I have checked alot of things but I cannot figure out why it is so terribly slow. If you need any more information, please feel free to ask, I would be happy to share.

 

[Samuel Tai]

Where do you have your system dataset? Look in System->System Dataset. This should be on your Mirror_Pool_22TB, not freenas-boot. Your boot media is much too slow to handle the system dataset.

 

[Samuel Tai]

Also, you've filled out your dataset past the recommended 50% point for iSCSI. See https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

 

[xsparta11x]

Where do you have your system dataset? Look in System->System Dataset. This should be on your Mirror_Pool_22TB, not freenas-boot. Your boot media is much too slow to handle the system dataset.

its in the Mirror_Pool_22TB

 

[Yorick]

I am only getting in the neighborhood of 50 MiB/s on speed MAX.

Read? Write? Both?

I'm assuming these are sync writes from VMWare, is that so? With sync writes, you likely want a SLOG. https://www.servethehome.com/buyers...as-servers/top-picks-freenas-zil-slog-drives/

For your L2ARC, what do your ARC stats tell you? Does your read data fit into ARC mostly? If you are using the L2ARC, how much of your working set fits into it?

Read, I'd expect much better speeds as long as you can service from ARC / L2ARC.

Write, if that's sync, the situation you have is expected.

 

[xsparta11x]

Read? Write? Both?

I'm assuming these are sync writes from VMWare, is that so? With sync writes, you likely want a SLOG. https://www.servethehome.com/buyers...as-servers/top-picks-freenas-zil-slog-drives/

For your L2ARC, what do your ARC stats tell you? Does your read data fit into ARC mostly? If you are using the L2ARC, how much of your working set fits into it?

Read, I'd expect much better speeds as long as you can service from ARC / L2ARC.

Write, if that's sync, the situation you have is expected.

I tried googling, (sorry I am new to freeNAS) but I cannot find the info on ARC. where do I need to go in order to take a look at ARC info?

 

[Yorick]

You could run, from ssh command line:

zfs-stats -A - for ARC
zfs-stats -L - for L2ARC
zfs-stats -a - for all of the stats