ESXi_5.5+FreeNAS_9.2.1.5 migration performance

[TimTeka]

Guys, i have dedicated vlans/nics for each vmotion/iscsi/vm networks, plus i'm using red WD's 3TB in raid1/zfs on each FreeNAS storage. Still getting rather poor performance on migration (consider double 1GB nics per FreeNAS host with doulble dedicated nics on esxi with round robins):
read stats:

write stats:

 

[cyberjock]

Can you post hardware specs for both sides please....

 

[TimTeka]

IMHO, the hardware is excessive :)
FreeNAS1 (the source):
double Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
RAM 24546MB
zvol 2.50T with lz4

FreeNAS2 (the receiver):
double Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
RAM 24554MB
zvol 2T "inherit" (suppose it's also lz4)

The esxi host is too heavy compared to the FreeNAS hosts. Two modern Xeons with 128GB RAM.

 

[cyberjock]

So FreeNAS is a VM on ESXi1?

 

[TimTeka]

Nooo. Sorry for the typo. I've described 3 servers. Two with FreeNAS installed and one - esxi 5.5.

 

[cyberjock]

You are going to need MUCH more hardware on your FreeNAS side. You are very likely to need a ZIL, an L2ARC, and a pool of mirrors if you want good performance. The forum is literally full to the brim of users with problems with VMs being slow. Read up on them to understand better, but your hardware is very inadequate. ;)

 

[TimTeka]

Hm. Double Xeons with 24GB ram is too few for the FreeNAS? Suppose you are not kidding, dear cyberjock?
As for ZIL and L2ARC i'll RTFM. Thanks for advice

 

[cyberjock]

Yeah.. you should look at going to 64GB of RAM for starters to support an L2ARC. You'll also need a ZIL.

CPU will be fine as you don't need alot of processing power. What you need is lots and lots of caching of stuff for ESXi

Big problem that boils down to ZFS being protective of your data it requires serious hardware to safely handle your data. Then ESXi makes some overly protective choices by forcing all writes to be sync writes and you've got a recipe for poor performance.

 

[TimTeka]

But how should i guess where's the bottleneck? You've written in the Guide that usually one shouldn't use ZIL initially but rather perform RAM upgrade.
So here's the chart from the receiver FreeNAS:

How should i understand that I lack RAM whilst clearly reading "Free : 12.7G min" ? :))

 

[TimTeka]

BTW, compared to e.g. that product:
http://www.synology.com/en-global/products/overview/RS3412xs
They support only 8GB RAM! (and still declare 100,000 IOPS) Maybe the FreeNAS isn't suitable for esxi currently?

 

[brashquido]

Synology can make a storage unit, but I wouldn't be taking their perf figures on face value. Synology claim 100,000 IOPs on a RAID 5 array over dual aggregated 10GbE connections, but make no mention of the disks used or their testing methodology. Storage Review reviewed that unit and during a 4k random write test 'only' got 26,000 IOPs followed by 46,000 IOPs for a read test on a RAID 10 array with 10 x SSDs. I'd be considering building a storage server around multiple LSI 9271 controllers using CacheCade before getting that Synology unit.

I'll be very interested to hear how you get on with this one.

 

[cyberjock]

BTW, compared to e.g. that product:
http://www.synology.com/en-global/products/overview/RS3412xs
They support only 8GB RAM! (and still declare 100,000 IOPS) :) Maybe the FreeNAS isn't suitable for esxi currently?

Don't take this as me sounding like an a-hole... but please don't compare us to Synology. We are professional/enterprise grade. We do ZFS. We live to protect your data.

Do some googling and you'll find that Synology has their own special kind of screwups and lost data. They care, but only so far. We're like rabid dogs when it comes to protecting your data. That protection is not free however. It requires you to have more knowledge with regards to storage systems than a Synology and it also requires more hardware to get exceptional performance.

 

[TimTeka]

Mmm...Ok. understand now
Guru, last question please. How could one decide the "proper" amount of RAM for the host? (i mean my case, when using FreeNAS in conjunction with iscsi+esxi). What are the common considerations?
In various manuals/presentations you talk about "not less" values, like 16GB, 24GB, now 64GB. How much RAM should I invest into my system?

 

[cyberjock]

There are no "thumb rules". It's really about understanding the fundamentals of how ZFS works and how to properly interpret data from multiple sources to come to the proper answer. Generally, if your VMs are "too slow to keep you happy" then research is needed.

I know that with VMs you can easily find yourself needing 64GB+ of RAM, so don't buy a bunch of 4GB sticks for cheap and how to make it work with "just" 64GB of RAM.