I've been lurking, researching, and testing ZFS (for many months with OmniOS with Napp-IT, FreeNAS, and a bit of Nexenta) as datastore for ESXi with NFS or iSCSI; and the wall I am running into is EXACTLY like what jgreco, cyberjock and several other experience users on this forum and others had said thousands of time.
You want good performance? (given that you meet the requirement of good amount of RAM and several mirror vdev)
- Add super fast ZIL for sync write or forget about the whole project. << I am out finding for a suitable SSD right now and the Intel S3700 gives hope. In fact, I'd just created a thread in the Hardware portion of this forum asking for help.
http://forums.freenas.org/threads/ssd-suitable-for-zil-intel-dc-s3700.15130/ I'm going to order the Intel S3700 to test. If it doesn't helps much, I will return it and eat the 15% restocking fee. Then I'll just use iSCSI, with redundant UPS at co-location, and stop wasting more time testing ZFS datastore project.
- If you can live with NFS sync=disabled or iSCSI sync=standard which are the same thing, then no need for ZIL. Better be sure you have redundant power supplies AND UPS.
You can see whether your ZIL is being used by SSH to your FreeNAS box and do a "zilstat"... forgot the rest of the options. Look it up. :) If you see your ZIL being constantly used (lots of number moving), add ZIL device or performance will never be what you expect/want. I did this when I did a Veeam replication and my ZIL was constantly hit. Then I did a sync=disable, zilstat showed zeroooos...my peformance and IOPS skyrocketed.
As for ZIL device with super capacitor for power loss protection, from my research, currently the Intel S3700 is the best hope that is somewhat afforable but I have to test. No one said much about the new Seagate 600 Pro. Or ZeusRAM if you got big $$$ ($2500 for 8GB).