Interesting issue with VMWare, iSCSI and IOMeter

[prodigi]

So i'm having an interesting problem with running IOMeter. I'm testing out some new equipment before deploying it to production.
Here is my basic setup:
Dell PowerEdge C2100
12x 450GB 15K SAS
ZFS mirror using all drives
Perc R700 - adaptive reading/write through
VMWare 5.1 connected with iSCSI on a dedicated vlan and switch

I have the entire ZFS LUN presented to VMWare and I carved out a 100GB disk at attached it to a test virtual machine. I'm running IOMeter with an 8k 70% read and random writes. I am consistently pulling 15,000 total IOPS and about 90MB/s read and 70MB/s write. These numbers stay at this level until I reboot the FreeNAS. The very first test I run will see IOPS starting around 3,000 and eventually finishing at 10,000. Reads and write start out very low and go up as well.

The strange part is if I run a second test, the performance goes back up to normal numbers.

 

[eraser]

I would guess that you are seeing the effects of the ZFS ARC cache warming up. The first tests after reboot are directly from disk, while subsequent tests benefit from reading data already in the ARC.

I am a bit concerned about your PERC R700 card. (I think you meant H700 as I can't find a R700) for two reasons:

• I don't think the H700 offers a JBOD mode. See - http://forums.freenas.org/index.php?threads/dell-r710-perc-h700-in-jbod-mode.9570/ . Also you will want to confirm that the H700 does propagate disk errors up to the OS so that ZFS knows when a disk is starting to fail before it is too late.
• The "adaptive reading" setting may be affecting your test results depending upon how much cache is installed on the card. ZFS performs its own kind of pre-reading into the ARC (known as "prefetch" or "prefetching").

 

[prodigi]

That makes sense. Yes I meant an H700. And I do know that it doesn't have a JBOD mode. I created 12 individual virtual disks in the raid controller. I'll have to do some research on propagating the disk errors.
Just as an FYI to all. I have two identical boxes and was testing performance once with ZFS mirror and one with hardware raid10. The performance was surprisingly identical. That shows the power of ZFS.

 

[eraser]

Cool. That is a sweet little setup you have.

In case you don't already know you can run arc_summary.py and arcstat.pl from a command shell to see some more details about your ARC.

 

[cyberjock]

Yes I meant an H700. And I do know that it doesn't have a JBOD mode. I created 12 individual virtual disks in the raid controller.

This may be the single biggest mistake you can make with a FreeNAS box.

 

[eraser]

CJ - please provide additional information to support your comment or just delete it.

 

[cyberjock]

Sorry but the manual my presentation, and the forums already include the answer to the question.