FreeNAS & ESXi performance issues

[plissje]

Hi guys,
Would appreciate some insights into some of the information I see on my NAS server.

My NAS setup is used for an VMWare lab for about 10 employees with around 30 active VMs.
We have 2 ESXi servers, connected via a dedicated vlan to a dedicated 1Gbps nic on the NAS.
The NAS server itself is built on ASRock 2750D4I board with 16 GB of RAM. with 1TBx4 (7200) disks in a RAID 10 that serve as the main datastore for the VMs via NFS, and another 2TB disk for just a general CIFS share.
As of now, I have no ZIL setup, and have sync writes disabled. I know the risks of data loss and i understand and accept them, but doesn't mean I wouldn't love to fix this issue in the future.
Also, there is no need for real time performance. I just need the lab to be operational & OK. I would be more then happy if I can just saturate my current 1Gbps link.

Anyway, now back to the issues :). While checking the reporting tab, I see numbers that I cant seem to understand and this is where I hope you guys can come in and help out:
1) When looking at the network tab, I usually don't go over 500 at peaks, and stay at around 300 on a regular workload. (igb0 is storage, igb1 is the CIFS)

I thought this might be an issue with my networking, but running iperf from one of the ESXi hosts via the storage network shows that everything is normal.

So I guess the first question is, is this normal? shouldn't I see spikes that fully saturate my link when I have a lot of VMs running and doing stuff?

2) The second issue is with regards to the performance of my RAID. It seems to be quite low, in regards to the link saturation I see at (1). I'm talking specifically about the RAID as it seems that when I tried running a test on a temp NFS I've created on my additional single 2TB disk, I get far better performance. From the same VM I've tested with a RAID disk, and my secondary disk and these are the results:
RAID:

single disk:

So we obviously see that I can easily saturate my single disk, which for some reason is not even close on my RAID. Even if I consider that I have around 300MBps regular usage on it, I still need to see something in the region of 50-60 MBps right? Am I missing something here?


So any thoughts and suggestions would be more then welcome. I'm really trying to figure out when is missing :(.

Thanks!

 

[jgreco]

How full is your pool, and how fragmented is it? ("zpool list" from command line)

16GB is on the low side for iSCSI, even for a tiny pool such as yours.

Your pool is also quite small. ZFS is a Copy-on-Write filesystem, so as you rewrite old VM disk blocks, they are assigned new locations on the pool. This eventually leads to a lot of fragmentation, which ZFS designers typically battle by leaving GOBS of free space available, which dramatically increases write speeds, and then having sufficient L2ARC to cache the working set, which dramatically increases apparent read speeds. Being able to access pool free space works better when ZFS can cache more metadata, so the ARC (RAM) size impacts both read and write speeds. Block storage is one of the stressiest things to do with ZFS.

https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/

 

[plissje]

How full is your pool, and how fragmented is it? ("zpool list" from command line)

16GB is on the low side for iSCSI, even for a tiny pool such as yours.

Your pool is also quite small. ZFS is a Copy-on-Write filesystem, so as you rewrite old VM disk blocks, they are assigned new locations on the pool. This eventually leads to a lot of fragmentation, which ZFS designers typically battle by leaving GOBS of free space available, which dramatically increases write speeds, and then having sufficient L2ARC to cache the working set, which dramatically increases apparent read speeds. Being able to access pool free space works better when ZFS can cache more metadata, so the ARC (RAM) size impacts both read and write speeds. Block storage is one of the stressiest things to do with ZFS.

https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/

here's the pool list:

But please note that I use NFS for my storage and not ISCSI. The 50% rule with my pool size kind of left me with no choice here.
Is NFS also affected by the fragmentation issue?

 

[tvsjr]

The 50% full rule has to do with any type of block storage... it applies to a VM workload whether running NFS or iSCSI. A 55% fragmented, 79% full pool trying to do any type of block storage/VM workload is going to suck mightily.

 

[jgreco]

NFS or iSCSI are pretty much the same from the fragmentation perspective. At 55% fragmentation, 80% utilization, and 16GB of RAM on only four disks, you're actually getting pretty good performance in my opinion.

Add more disk space, either by adding more larger disks, or replacing the existing disks with larger disks. You will notice a massive performance difference if you go to some 4TB drives.

If you look at that, you'll notice performance is MUCH better at 20% utilization than 80%. We don't really expect block storage to do well beyond around 50-60% and that graph is just proof of why.

Keeping a similar amount of data to what you have on the pool now, if you were to go to 8 x 2TB drives and 32GB of RAM, you would see a huge jump in performance, and at 12 x 4TB drives with 64GB of RAM, you would think it was probably SSD. Unfortunately the 16GB DDR3 ECC UDIMM for the C2750D4I are basically gold-plated unobtanium.