FN 8.0.3p1, iSCSI, Perc 5i, VMWare ESX performance

[sabreofsd]

Hello all,
Well, I have combed through the forums and several online articles, but I have not been able to solve my problems. Here is my setup:

FreeNAS Server:
Dell PE 2900 (32G RAM, dual quad core xeons, Perc 5i w/256MB cache, 6x7200 1TB drives)
RAID controller setup with a RAID 5 of the 6 disks above with a 512k stripe
Exported the block level storage extent using iSCSI
64 connections max
64 sessions max
CHAP configured for user and initiator

VMWare ESXi 5 on a dual quad core DL380 with 64G
1VM running (Windows 2008 x64, 8G RAM, 160GB drive)

MTU set to 9000 on FreeNAS and ESXi server (was originally on 1500, but I upped it)
1Gbps dedicated connection between the two and I double checked the negotiation settings.

Problem:
iSCSI transfers are abysmal. To transfer a 680MB file up to ESX takes ~4 minutes. Boot speeds on the VMs is very slow.

During transfer notes:
The ESX server is almost idle, waiting for data from the NAS
The NAS is almost idle, almost no CPU time at all being used (makes sense as it has a dedicated RAID controller).
Using top, I see 600k (yep, k) devoted to cache on the NAS box and 30G free!
Read/Writes locally using dd are quick (100+MB/s)

Any ideas?

 

[louisk]

One of the things I found out (about 5 years ago when I was given a ESX/iSCSI setup to manage) was that there is a big difference between IO throughput and IOPS. SATA can provide OK IO throughput, but it provides miserable IOPS. When you're running another OS virtually, most of your IO is going to be small, low latency, not large file transfers. SATA drives are typically good for around 75 IOPS. 15k SAS is around 175 IOPS. This will make a very noticeable difference.

For high performance, you want to use RAID10, not RAID5.

Does your switch also support jumbo frames? Is it configured the same on both ends? Configuring only 1 end won't make a difference.

I would actually recommend that you use NFS instead of iSCSI.