Questions about using FreeNAS NFS share for ESXi data store

Status
Not open for further replies.

steven6282

Dabbler
Joined
Jul 22, 2014
Messages
39
Ok, so if you saw my other thread recently, you know I'm working on a new FreeNAS build. While doing some more research before buying my hardware I came across some statements that eluded to a ZFS based NFS share for an ESXi datastore would have poor performance due to the way ZFS works.

It was suggested that the solution to this is ZILs and L2ARCs. I had previously decided that I did not need these based on Cyberjock's newbie guide, however now I'm questioning that decision. I do intend to be using an NFS share for my ESXi's data store.

Now I've seen Cyberjock on more than one occasion say that you shouldn't use L2Arc until you have 64GB of ram, but after a few hours of searching I have been unable to find any reasoning behind this. I could bump my build up to 32 GB of ram, but I don't think it's even possible for me to go to 64 GB of ECC currently with the board I am planning to use (A1SAM-2550F, http://www.supermicro.com/products/motherboard/Atom/X10/A1SAM-2550F.cfm) The board requires unbuffered memory, and I have not seen any ECC 16 GB strips available as unbuffered yet. Changing the board isn't something I really want to do right now. This box needs to stay as low powered as I can make it. If you must know the reason, I like our environment so I do what stuff I can to help be clean, thus my ESXi and NAS boxes + networking equipment (the stuff that runs 24 x 7) all run primarily off a small solar array in my backyard. It of course has line power backup, but I'd prefer to stay within the confines of my solar array's capacity.

So anyway, I'm trying to figure out at what point is performance bad for VMs stored on an NFS data store from FreeNas? Is it something that is only seen once you get a bunch of VMs running, or if I'm just running a small number of 4 to 5 VMs the majority of the time, will I see the problems? Sometimes I may be running as many as 10, but there are almost always at least 4 running. Are the problems more related to write speed thus needing a small ZIL, or read speed?

Also, I've seen cyberjock say in this thread: http://forums.freenas.org/index.php?threads/at-what-point-does-l2arc-make-sense.17373/
...so you don't want a 60GB L2ARC because that will take up about 12GB of your ARC...

Where is this information derived from? I haven't been able to find anything concrete on this, but I have seen two other websites earlier while researching the problem estimate it closer to 2GB of ARC to index 100GB of L2ARC. I of course can't find those sites again atm cause I've literally went through 50 to 100 different pages this morning trying to find more information lol. But if I run across them again, I'll come back and add the links to them. I'm trying to figure that out because I think that is his primary basis for recommending 64GB before doing L2ARC.

So the question really boils down to if I even need to L2Arc with 4 to 5 VMs the majority of the time, and if so can I get away with a 60GB L2Arc with 32GB of system ram without any seriously adverse side effects? And also, if I need to set up a small ZIL for this situation.

Cyberjock, I know you are likely to reply to this based on previous encounters so I'm going to pre-reply to you :). I ask that if you do reply, please have links for sources for the information you provide. You definitely appear to be knowledgeable when it comes to FreeNAS / ZFS stuff, I'll grant you that, but I can't simply take one person at their word to be definitely correct. As to your common "you can search for it", I've been searching for it all morning and unable to find it lol. As for your other common "you can do it that way if you want but don't cry to me when you have problems", yeah... don't worry I wont cry to you, but I'd really rather have the facts up front before I decide how I'm going to do it. :)


Thanks in advance for any help!
 
Last edited:

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Whether you need an L2ARC or ZIL is totally dependent on the kind of performance you need. If they run fast enough to make you happy then you don't need them. If they are painfully slow you'll probably need them.

2GB of ARC indexes about 100GB of L2ARC... I might believe that. I think 3-4GB is probably more accurate. This is MUCH more complicated than you think though. So either push the "I believe" button and keep your l2arc size to 1/5th of your arc size or read up on how all of this works internally to ZFS. Sorry but I don't really want to hash this out all over again. Done it plenty of times in the past. All ARC is not available to index the L2ARC. The short answer if you do all the math and understand the technology is to not exceed about 5x your ARC as you may not even be able to use the L2ARC because of limitations internal to ZFS.

The 64GB of RAM thing comes about with L2ARC because the L2ARC is meant to supplement the ARC. If you don't have a good sized ARC to start with you aren't supplementing anything and you are trying to substitute. The L2ARC sucks at substituting for the ARC and it won't do it. If your ARC is that small adding an L2ARC is only going to stress the ARC even more because the L2ARC takes up ARC space. If you stress out the ARC performance of your pool will actually drop by adding an L2ARC. I've seen people all over the forum do this and get upset over it. And from watching all of these people complain I'm convinced that an L2ARC isn't too useful (except in very very specific situations and with tuning) unless you have 64GB of RAM.

So remember.. L2ARC = 5x your ARC and 64GB of RAM to start. Can't go wrong. ;)

As for your ZIL, you should look at getting one. The Intel S3700 are the ones you should eyeball. Many of the cheaper SSDs don't have a performance curve that makes them a good choice. Several people have tried cheaper stuff from Samsung and said that they didn't seem to help much until they spent their money on the Intel.
 

steven6282

Dabbler
Joined
Jul 22, 2014
Messages
39
Ok, now I'm confused. So you are saying you might believe 2GB per 100 but think 3-4 GB is more accurate. Ok, that is still a ways off where I quoted you saying a 60GB would take 12GB??? Even at 4GB per 100, a 60GB L2Arc would take 2.4GB to index, no? Also, L2Arc 1/5th arc size, or 5x Arc size, which one? A 60GB SSD is much smaller than 5x what the ARC size would be on a 32 GB ram system, but it's quite a bit larger than 1/5th the size =/

I'm not looking to rehash anything, I just want some links to sources of documented, quantified, objectified, whatever you have, to read about it. I mean I read that OpenZFS is open source, so sure I could go grab the source and scour it for a few days to figure it out directly from the source, but I don't really have time to commit to that lol.

As for the Intel S3700, the smallest one I see is 100GB. I've seen multiple places including you I think say a ZIL didn't need to be more than a few GB. Is the S3700 choice then due solely to performance? Also, the S3700 is an MLC drive, everything I see says a ZIL should be a SLC drive. I haven't looked real hard into why but I believe it was due to reliability since data in the ZIL can be lost and cause pool corruption if the ZIL fails, and SLCs are more reliable. Is this possibly just an older rule and newer MLCs are reliable enough?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Ok, now I'm confused. So you are saying you might believe 2GB per 100 but think 3-4 GB is more accurate. Ok, that is still a ways off where I quoted you saying a 60GB would take 12GB??? Even at 4GB per 100, a 60GB L2Arc would take 2.4GB to index, no? Also, L2Arc 1/5th arc size, or 5x Arc size, which one? A 60GB SSD is much smaller than 5x what the ARC size would be on a 32 GB ram system, but it's quite a bit larger than 1/5th the size =/

I'm not looking to rehash anything, I just want some links to sources of documented, quantified, objectified, whatever you have, to read about it. I mean I read that OpenZFS is open source, so sure I could go grab the source and scour it for a few days to figure it out directly from the source, but I don't really have time to commit to that lol.

As for the Intel S3700, the smallest one I see is 100GB. I've seen multiple places including you I think say a ZIL didn't need to be more than a few GB. Is the S3700 choice then due solely to performance? Also, the S3700 is an MLC drive, everything I see says a ZIL should be a SLC drive. I haven't looked real hard into why but I believe it was due to reliability since data in the ZIL can be lost and cause pool corruption if the ZIL fails, and SLCs are more reliable. Is this possibly just an older rule and newer MLCs are reliable enough?

Tiny SSD are tiny in many other ways - including performance. NAND is painfully slow (as your average USB drive demonstrates), but can be used in parallel to make the SSDs we know and love.

Partition the size you need and forget the rest exists. The drive will use the whole capacity (plus reserves) for wear leveling, so it's not completely wasted. Since there's room to write without erasing first, an almost empty drive will also have better performance than after it's filled.
 
Status
Not open for further replies.
Top