Performances vs raid selection and number of drives

[Francis D]

Hi all, I have made some upgrades to my freenas server and in fact, performances have decreased. Maybe anyone have any cues on real causes.

I run freenas on a ESXI setup with a LSI 9211-8i card passthrough

First setup;
2 cores of my xeon E5520 2.26gHz, 8 gb ram, 4 drives, 2 stripes mirror ( raid 10 equivalent )
I was able to reach 90-100Mo/sec with samba share

Second ;
up to 6 cores of my new xeon X5670 2.93gHz, 24gb ram, 6 drives Raid Z3
I can only reach 60Mo/sec max.

Here are my observations;
-more than 4 cores does nothing
-from 8 to 24gb of ram, I saw an increase of 10Mo/sec with every 8gb of ram I add. This is really expensive to reach 100Mo/sec ...
-my cpu upgrade had no impact
-now with Raid Z3, sometimes one thread use 40% of my cpu for days. I don't know what it does.

Anyone has a clue about what is the bottleneck ? I see that it's not the CPU. Maybe it's my cheap LSI 9211-8i that is not able to handle well more than 4 drives ? RaidZ3 is a bad choice ? With my first setup, even with a limited hardware, I was able to use 100% of my Gb network, and now with 2 more drives, I only receive complains of users. Any other ideas ?

 

[Spearfoot]

What version of FreeNAS are you using? What version of VMware ESXi?

What VMware ethernet NIC are you using? E1000? VMXNET?

Your LSI 9211 card is fine and, in fact, is recommended hardware.

Yes, RAIDZ3 is a bad choice with only 6 drives; half of your disks are being used for parity. Consider using RAIDZ2 or mirrors instead.

Have you locked all of the FreeNAS VM's memory in ESXi?

FWIW, I use FreeNAS 9.10-STABLE as a VM on ESXi v6 with 2 vCPUs and 16GB RAM with very satisfactory results.

 

[Mirfster]

As @Spearfoot mentioned you want to lock the RAM. In your first setup, even though you had less RAM FreeNas had total control over it; whereas as a VM it does not hence the need to lock in the RAM.

RaidZ3 is a bad choice ?

Depends, but the initial answer is "Yes"

I can only reach 60Mo/sec max.

What are you using as your benchmark tool(s)?
Are you checking over CIFS, NFS, iPerf, etc?
Is ESXi running any other VMs?

Maybe it's my cheap LSI 9211-8i that is not able to handle well more than 4 drives ?

Doubt it, per LSI:

Connect up to 256 SAS and SATA devices with 8 internal 6Gb/s SAS and SATA ports

First setup;
2 cores of my xeon E5520 2.26gHz, 8 gb ram, 4 drives, 2 stripes mirror ( raid 10 equivalent )

Second ;
up to 6 cores of my new xeon X5670 2.93gHz, 24gb ram, 6 drives Raid Z3

Not a fair comparison, you are using one vDev in RaidZ3 which basically is the through put of one (1) drive; but has a lot more writing and checking to do... 2x Mirrors are going to give you more IOPS...

Suggestions:

1. Set the CPU to "High" on the VM
   
2. You can "isolate" a NIC on a vSwitch just for FreeNas (if desired)
   • Heck could even pass-through a NIC if you wanted (I did in some testing with 10GB and got 100% increase; but still have more testing to do...)
3. If using a "VMXNet3" emulated adapter, see this thread for using updated vSphere drivers (instead of the ones in FreeNas)
   • It is page 11; last post on that page