iSCSI (device extent and file extent) + ESXi 5.0 low speed

[Essential]

Hi all.

My freenas hardware:
Pentium(R) Dual-Core CPU E6600 @ 3.06GHz
4047MB
Raid level 10 Adaptec 2405
Network Intel pro
FreeNAS-8.0.2-RELEASE-amd64 (8288)
ESXi 5.0.0 (469512)

When i create iSCSI target with file extent with vmfs5 at UFS volume i have this speed from esxi host:

Code:

/vmfs/volumes/4ed5fb55-c55975ac-96de-001b21c3f854 # time dd if=/dev/zero of=ddfile bs=1024K count=1000
1000+0 records in
1000+0 records out
real	0m 15.02s
user	0m 1.50s
sys	0m 0.00s
68 MB/s
/vmfs/volumes/4ed5fb55-c55975ac-96de-001b21c3f854 # time dd if=ddfile of=/dev/null bs=1024K count=1000   
1000+0 records in
1000+0 records out
real	0m 14.51s
user	0m 5.45s
sys	0m 0.00s
70 MB/se

This is test at device extent (hardware raid 10) with VMFS5

Code:

/vmfs/volumes/4ed72483-8e87d0a0-8b4e-001b21c3f854 # time dd if=/dev/zero of=ddfile bs=1024K count=1000
1000+0 records in
1000+0 records out
real	0m 52.12s
user	0m 2.00s
sys	0m 0.00s
19.6 MB/s
/vmfs/volumes/4ed72483-8e87d0a0-8b4e-001b21c3f854 # time dd if=ddfile of=/dev/null bs=1024K count=5000   
5000+0 records in
5000+0 records out
real	1m 15.10s
user	0m 25.36s
sys	0m 0.00s
68 MB/s

Maybe somebody know why write speed so low when i use device extent? This is because i use hardware controller or maybe some else issue?

 

[jgreco]

Slow disk, no write cache, and a crummy choice of RAID controller and level? I'm just guessin'. Test on the FreeNAS host *before* introducing the challenge of iSCSI into the equation. You won't get good performance from ESXi until you get good performance from FreeNAS at the CLI.

You will want to see if write cache is enabled on the drives. You probably have some (128MB) write cache on your controller, but it may not be configured to use it. Check. 128MB is a piddly amount, most modern controllers have more like 512MB or more, but the difference between 128MB and 0MB is still quite stunning. Turning write cache on everywhere *helps*. Significantly.

Using RAID10 offered by your hardware is quite possibly a bottleneck. Try taking apart the RAID10 and using a single drive. See what speeds you get. Should be great, almost no drives these days truly suck. Then try using ZFS and RAIDZ or RAIDZ2, see how that goes. Understanding these tradeoffs is crucial to getting max IOPS out of your NAS system. Once you have all that working as you wish, then set up iSCSI and you'll still see a performance hit, since iSCSI is invariably a fat layer in between your storage and your VM's. By default, for example, an SSD-backed iSCSI device on FreeNAS (where dd from FreeNAS console runs around 260MBytes/sec) only writes at about 38MB/sec from an ESXi VM. Some of that can be tuned better, but some of it is just iSCSI.

 

[Essential]

Sorry that i dont explained clearly. When i created UFS volume with file extent i used the same RAID 10.

I have one array and different speed when i use file and device extent. Device extent is more slow then file.

This test from freenas cli at raid 10 array

Code:

 /mnt/rd10ufs# dd if=/dev/zero of=ddfile bs=1024K count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.299040 secs (197880377 bytes/sec)
188MB/sec