will this be enough for 10Gbe NFS Storage

[ibmg]

Hey there

I'm about to build a NFS storage for two ESXi hosts

The Box will based on an Intel Server Chassis

Hardware

2x Intel Xeon 5620 2,4GHz
48GB DDR3 ECC Ram
12x WD 1TB RE4
2x Intel 320 SSD 120GB (on internal SATA)
LSI SAS 9211-8i IT Firmware
Intel 10Gbe CX4 Dual Port
8GB SATA DOM (FreeNas boot)

This box will be connected via a HP 2910al with 4 10Gbe CX4 Uplinks
In the first instance i will just use on port of the card to connect the storage
Each ESXi will have a 10Gbe link to the same switch as well

I hope to get dencent speed out of this box
Does anybody tryied an aggregated link with two 10Gbe interfaces on Freenas ?

Since i have two ESXi should I split my storage in two seperat datasets for each ESXi to share ? will this help on I/O
Which RaidZ Level ?

I'm primary using NFS, so what would be the best to use the 2 SSD's for
LOG or Cache on ZFS ?

 

[louisk]

In order to actually get 10G performance, you will likely need SSD for cache, and you will certainly need something like http://stec-inc.com/product/zeusram.php for your LOG (I think best practice would be to mirror them in case one fails).

I haven't tried multiple 10G connections on FreeNAS, I seem to recall from last VMworld that ESXi recommendations were to limit to 2x 10G connections.

If you're going to create 2 zpools for storage, you will need more spindles. I'm confident that there is no combination you can configure 6 spindles that will produce 10G throughput. I also suspect there is no combination you can configure 12 spindles that will give you 10G.

Also, if you're serious about pushing 10G with ESXi, you should really get SAS. SATA will slow you down. And FWIW, if you can accept SATA storage, I don't think you need 10G. We have about 500VMs on one ESXi cluster at work and we're only pushing about 2G, which you could accomplish with a 4x LAGG (LACP) and have room to spare.

 

[ibmg]

Louis,

Thank You !
I think you just answered exactly what i wanted to know.
My goal is not to push 10G, i just want a rocksolid platform and i wasn't sure about Freenas
I had Nexenta and Freenas to choose from, but i like to use Freenas since it is nanoBSD base and can run off the SATA DOM

But regarding LAGG, i was under the impression ESX does not support LAGG's
Anyway i will setup the system

10G was more like a "nice to have toy" and the budget would allow it.
I could go with LAGGS, but i more like a single pipe, even if i can't fully use it.

 

[louisk]

I've never tried a LAGG in a VM on ESX. We're just starting a 10G rollout at work (Juniper QFX - converged Ethernet/FCoE). I've never needed to push more than 1G on ESX.

I'm using FreeNAS at home, and I haven't pushed it past 600Mbit (75MB/s). My only tests are to/from a laptop or a mac mini, and neither will push data terribly fast.
At work, we have a similar setup and we're able to push 925MB/s through it (36 2T SATA spindles, ZFS, 9 spindle vdev RAIDZ, 24G RAM) on a single 1G link.