Best setup for ESXi datastores

[kobazik]

Hi,

I'm building my first FreeNAS box with focus on using it for ESXi datastores. There will be 3 ESXi servers accessing storage on FreeNAS. Each server will have 2x 1Gbit links and FreeNAS itself will have 4x 1Gbit links.

There will be 10 - 20 small/mid spec VMs running on each ESXi servers.

FreeNAS hardware:

Chasis: Supermicro SC836 BE16-R920B
Motherboard: Supermicro X8DTi
CPU: 2x Xeon E5620/4C@2.4GHz
RAM: 24 x Kingstone DDR3 1333 MHz 2GB (24GB)
HBA: IBM ServeRaid M1115

I'm planning to have two pools:

6 x 1TB 5400 rpm WDC WD10SPCX-22H - 16mb cache avg. latency 5.5ms, internal transfer rate (max) 140MB/s

6x 500GB 7200rpm

L2ARC: Sandisk X110 256GB (MLC) - 81k / 51k IOPS, 515/465 MB/s - 80 TBW
ZIL: Sandisk U110 (SLC) - 128GB - 470/380 MB/s - 60 TBW

I've spent few days researching and still I'm not sure about few things:

1. Some people preffer NFS over iSCSI for ESXi storage. I would thought iSCSI is better and faster as you can do MPIO. I know ESXi sends everything async so I will have to set zfs to always do sync.

In my case I will be using 2 x 1Gbit links per ESXi server so in theory it should give me ~250MB/s between ESXi and FreeNAS box.

2. Is SLC SSD a must for ZIL ? Can I use MLC with higher TBW? Can you recommend any high endurace SSDs other then Intel S3500 and ZeusRAM?

3. There are mixed opinions about mirroring ZIL accross 2 SSD drives. What happens to a pool if I lose ZIL SSD drive?

4. Should I use datasets or just create zvol for each ESXi datastore?

5. What is the best way to backup VMs stored on FreeNAS? I know you can do snapshots or replication. Will it work with NFS datasets and/or iSCSI?

Any comments and suggestions are apriciated!

Thanks

 

[L]

I don't know if you have tried a nightly build of 9.3, but there are some nice tools in there for esxi. There is a new utility symmetrical snap which communicate with the vm when you take a snapshot. It also asks in the wizard at install time "what are you using this for?" and config's the storage for you.

I always mirror zil.

I just recently read an article on ssd endurance, http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte ...

 

[HoneyBadger]

Nice first post, it looks like you've done a good bit of research. Is this for home use or business?

"1. Some people preffer NFS over iSCSI for ESXi storage. I would thought iSCSI is better and faster as you can do MPIO. I know ESXi sends everything async so I will have to set zfs to always do sync. In my case I will be using 2 x 1Gbit links per ESXi server so in theory it should give me ~250MB/s between ESXi and FreeNAS box."

iSCSI with sync forced will be limited by your SLOG write speed, just like NFS. You will get MPIO benefits from reads though. For either scenario though you will want to use a separate set of switches for your storage traffic. For the 2x 1Gbit links per ESXi server, is that dedicated to storage? If not, I'd highly suggest trying to get additional NICs (for a total of four ports) so that you can dedicate a pair for storage and a pair for VM traffic. If you want to add VMotion in there, best practice would be to isolate that as well, and now you're up to six ports per host.

"2. Is SLC SSD a must for ZIL ? Can I use MLC with higher TBW? Can you recommend any high endurace SSDs other then Intel S3500 and ZeusRAM?"

You'll get better performance but it isn't a must. The key for SLOG devices is to have fast writes, low latency, and power-failure safety. That ZeusRAM is a complete monster, being battery-backed RAM, but you're going to pay a lot of money for them. Aside from the DC S3500, good consumer SATA SSDs are the DC S3700, the old Intel 320 (it's slow though) and the Seagate Pro 600. Whichever one you pick, you'll only need to provision about 8-10GB of it for use as the SLOG; leave the rest of the drive unallocated to help with wear-leveling and consistent write speeds.

"3. There are mixed opinions about mirroring ZIL accross 2 SSD drives. What happens to a pool if I lose ZIL SSD drive?"

Best practice is to mirror the SLOG devices. If you lose the device, your pool speeds will nosedive and those VMs are going to be incredibly unhappy when their write latency skyrockets. In addition, if you have a power failure and your SLOG device doesn't come back up, your VM will think that it wrote the data to safe storage (which it did, the SLOG) but ZFS won't be able to replay the transaction group. You'll have to roll back, which means that data is lost.

"4. Should I use datasets or just create zvol for each ESXi datastore?"

If you're going to use iSCSI, use zvols. The new iSCSI target in 9.3 will perform better with them as well as (I believe?) supporting important features like SCSI_UNMAP to reclaim deleted space.

"5. What is the best way to backup VMs stored on FreeNAS? I know you can do snapshots or replication. Will it work with NFS datasets and/or iSCSI?"

As Linda mentioned above, 9.3 will bring some improvements for VMware snapshotting (it will take a VMware snap before doing a ZFS snap) which should guarantee that you'll have crash-consistent VMDKs in your ZFS snapshots. There's always the option of getting third-party backup software (eg: Veeam) to handle this task as well.

Hardware thoughts coming.

 

[HoneyBadger]

Hardware thoughts now.

Board/CPU - Good, but older generation. Is this parts on hand already, or a particular reason you aren't going with at least the LGA1155 Xeons?

RAM - You've got 24x 2GB DIMMs listed but only 24GB total. Typo perhaps? Either way I'd strongly suggest you go with 48GB as a minimum here especially if you want to use L2ARC.

Disks - Your pool layout looks like you're going for RAIDZ2? That's fine for bulk data storage (eg: those 1TB 5400rpm drives) but not what you want backing a general VMFS datastore. Random I/O is king for virtualization and while L2ARC/SLOG will help you out, you want to give your pool the best chance it can to keep up. Use a mirrored setup (3x2) for the 500GB 7200rpm drives for the system datastores. If you need the space badly, you can use a RAIDZ2 setup and store data on it, but try to keep it for sequentially accessed data.

L2ARC - Chosen SSD is fine. Size-wise though, if you only have 24GB you don't have enough RAM to make proper use of L2ARC. Even with 48GB, a 256GB L2ARC is possibly too large. You can always provision a smaller chunk (128-192GB) though and still make use of the increased speed you'll get from a bigger SSD.

ZIL (SLOG) - This isn't actually an SLC disk; it just has a small section of the MLC NAND that is behaving like SLC in order to accelerate writes. Also no power-loss safety. Not a good one here, look for one of the Intels suggested in my previous post, the Seagate if your budget's tight, and go enterprise SLC/ZeusRAM if it's not. ;)

Hope I haven't lost you in that wall of text. Cheers!

 

[kobazik]

Many thanks for your reply. Please find my answers below.

Nice first post, it looks like you've done a good bit of research. Is this for home use or business?

Business use.

"5. What is the best way to backup VMs stored on FreeNAS? I know you can do snapshots or replication. Will it work with NFS datasets and/or iSCSI?"

As Linda mentioned above, 9.3 will bring some improvements for VMware snapshotting (it will take a VMware snap before doing a ZFS snap) which should guarantee that you'll have crash-consistent VMDKs in your ZFS snapshots. There's always the option of getting third-party backup software (eg: Veeam) to handle this task as well.

Hardware thoughts coming.

I'm planning to to build this setup next month so hopefully 9.3 stable will be out by then ;)

 

[HoneyBadger]

"Business use."

Awesome, because what that means to me is "we have a bigger budget." Try to stretch it to include a better pair of SLOG SSDs as I suggested.

"I will be using 2 x Juniper EX4200 linked by a VCP cable. Each ESXi server will have dedicated 2x 1Gbit ports going to a different switches."

If you have only 2x 1Gbps ports on each server in total, I strongly suggest you buy an additional dual-port NIC for each host and wire up a separate pair of switches to use for iSCSI. Mixing regular network and storage traffic leads to Bad Things.

"I'm planning to to build this setup next month so hopefully 9.3 stable will be out by then ;)"

BETA is coming very soon. Not sure if STABLE will be out by then but I hope so, since I'm eagerly waiting for that to upgrade my office setup.

 

[kobazik]

Hardware thoughts now.

Board/CPU - Good, but older generation. Is this parts on hand already, or a particular reason you aren't going with at least the LGA1155 Xeons?

Budget constrains:) I will change them to x5650 to support 1333mhz RAM I got already.

RAM - You've got 24x 2GB DIMMs listed but only 24GB total. Typo perhaps? Either way I'd strongly suggest you go with 48GB as a minimum here especially if you want to use L2ARC.

Indeed I've got only 24GB (12 x 2GB).

Disks - Your pool layout looks like you're going for RAIDZ2? That's fine for bulk data storage (eg: those 1TB 5400rpm drives) but not what you want backing a general VMFS datastore. Random I/O is king for virtualization and while L2ARC/SLOG will help you out, you want to give your pool the best chance it can to keep up. Use a mirrored setup (3x2) for the 500GB 7200rpm drives for the system datastores. If you need the space badly, you can use a RAIDZ2 setup and store data on it, but try to keep it for sequentially accessed data.

I will rather scarify space and gain performance for VMs so I'm thinking of going mirror aka RAID-10

L2ARC - Chosen SSD is fine. Size-wise though, if you only have 24GB you don't have enough RAM to make proper use of L2ARC. Even with 48GB, a 256GB L2ARC is possibly too large. You can always provision a smaller chunk (128-192GB) though and still make use of the increased speed you'll get from a bigger SSD.

Even with 48GB ram would you suggest having l2arc with my setup? Will it have much performance impact for ESXi datastores?

Perhaps I will be better of with without l2arc at all? I've seen somewhere that l2arc will use arc to store index so 256GB l2arc could potentially use up even 10GB RAM.

 

[kobazik]

If you have only 2x 1Gbps ports on each server in total, I strongly suggest you buy an additional dual-port NIC for each host and wire up a separate pair of switches to use for iSCSI. Mixing regular network and storage traffic leads to Bad Things.

Sorry I should be more specific:) Each server will have a quad-port Intel NIC with 2x 1Gbit for iSCSI and 2x 1Gbit for other traffic.

 

[kobazik]

ZIL (SLOG) - This isn't actually an SLC disk; it just has a small section of the MLC NAND that is behaving like SLC in order to accelerate writes. Also no power-loss safety. Not a good one here, look for one of the Intels suggested in my previous post, the Seagate if your budget's tight, and go enterprise SLC/ZeusRAM if it's not. ;)

I looked at Sandisk X110 specs and it seems they have power loss protection.

http://www.sandisk.com/Assets/docs/Unexpected_Power_Loss_Protection_Final.pdf

"SanDisk X and A series include the following unexpected power loss protection mechanisms:

IO Write Commit
SanDisk X and A series use the safe location mechanism to protect existing data in the lower
page while writing new data to the upper page. It also includes the mechanism to detect
unsuccessful writes during SSD power up after unexpected power loss to attempt recovery
from the safe location."

Correct me if I'm wrong but according to above X110 SSD series should have some sort of power loss protection. In addition write performance seems really good for this model so I believe it can be a good candidate for mirrored ZIL setup.

Sandisk X110 128GB - 80 TBW- 81.000 / 51.500 IOPS - 515/465 MB/s

vs

Intel DC S3500 80GB - 45TBW - 70.000 / 7.000 IOPS - 340 / 100 MB/s

So far only competition seems to be Seagate 600 PRO 120GB with amazing 209 TBW - 80.000 / 20.000 IOPS - 520 / 300 MB/s (burst 600MB/s)

Price to value I would go for Sandisk X110 as mirrored ZIL. I'm worried Intel S3500 write performance might not be enough considering my pool and network should be able to saturate ~500MB/s plus TBW seems low and price is higher than X110.

What does everyone think ?

 

[jgreco]

Perhaps I will be better of with without l2arc at all? I've seen somewhere that l2arc will use arc to store index so 256GB l2arc could potentially use up even 10GB RAM.

We typically suggest maintaining a 4:1 or 5:1 ratio of L2ARC:ARC to make sure the ARC isn't overly stressed, but of course actual reality dictates that better.

 

[kobazik]

I'm bit worried that my 6x 1TB 5400rpm and 6x 500GB 7200rpm drives pools won't be able to provide good performance for ESXi servers. Ideally I would like to be able to for each pool to saturate 2Gbits link over iSCSI.

I'm considering now having a cheap RAID-5 SSD pool (I don't need a huge capacity) that would provide more IOPS and be able to saturate 2Gbits.

My choices are:

3x Kingfast F9 256GB (8yrs - 100GB/day TBW) - 59000 / 3300 IOPS - 510 / 180 MB/s

or

3x Intel 520 256GB (1.2mln MTBF) - 25.000 / 40.000 IOPS - 550 / 500 MB/s

 

[jgreco]

Think I just finished advising against RAIDZ SSD for ESXi datastore in another thread.

 

[kobazik]

Think I just finished advising against RAIDZ SSD for ESXi datastore in another thread.

Can you elaborate please or point me the thread

 

[jgreco]

upper right corner, search, ssd raidz esxi author jgreco ...?

 

[kobazik]

Think I just finished advising against RAIDZ SSD for ESXi datastore in another thread.

OK so two mirrored vdev will be better otherwise write performance might suffer...

 

[jgreco]

also read, though suffers less

 

[L]

IMHO, if I want to run freenas commercially, i am running supported truenas.

 

[jgreco]

So... you're advocating pirating TrueNAS?

 

[L]

advocating buying a truenas system :)

 

[cyberjock]

LOL. Pirated TrueNAS. Isn't piracy supposed to be the sign of a successful software product?

I can see it already.. right next to the "very legit" Windows 8.1 install CDs is the "very legit" TrueNAS install CDs.