Small NAS build for NFS to ESXi

[jwhitt]

I'm in the process of building a nas that will provide NFS storage to 2-4 small exsi hosts. This is a small home lab, currently consisting of 2 esxi hosts (6core 16GB desktops) with the planed expansion to 4 hosts.

Network hardware consists of Delkl 2824 and dell 2016 switches.(gig-e)

The hardware im using for the build is
SuperMicro MBB-X9SCM-F-O
Pentium G2020
IBM ServeRaid M1015
Starting with 2 8GB Sticks of DDR3 1333 ECC Ram (Planing on upgrading to 32GB as money allows)
Harddrives consist of 6 1TB SATA3I was planning on raidz1
I have 2 64GB SSD's that i was planning on useing as a zil
I also have a few more ssd's that could use the on-board SATA connectors

As this is a lab environment the primary use will be to deploy vm environments as fast as possible. These will have hardly any load as its mostly for testing, spin up setup a few test accounts etc.

My question is where should i start on performance tuning of Freenas to meet this objective? Should i use the zil? Should i look into a l2arc cache device?

 

[cyberjock]

First, you aren't going to get far with VMs with just 16GB of RAM. 32GB is probably the lowest to start(and unfortunately, is the max you can use). If you are just testing you can simply do sync=disabled and realize that if the server crashes your VM should be redone from scratch. ZIL may be a waste of time.

You shouldn't even try an L2ARC until you get 32GB of RAM. And even then, anything bigger than 64GB is probably crazy.

RAIDZ1 is borderline irresponsible(but so is sync=disabled). But again, if its just for testing VMs and you are willing to deal with the possible rebuild of VMs from scratch, it's okay.

As for performance tuning, you'll get almost no support on that. I wouldn't even try to tune your system until you have 32GB of RAM. Tuning is an art and a science. It's hard as hell for newbies, and even harder when you are short on resources such as you are.

What you need to do is read around the forums at all of the other threads for people that have done VMs on ZFS and see their responses. These topics have been discussed to death.

 

[jwhitt]

Thanks for the reply. Yeah unfortunately that motherboard only supports 32G, I may look into a different mobo in the future, however all the server class single socket boards seem to only have 4 ram slots.

Would i be better served using ufs instead of zfs?

 

[cyberjock]

UFS support is going to be removed after 9.2.x, so UFS is a bad long-term choice. It's also very limited in what it can do as it has no caching system at all.

The socket 2011 boards have more RAM slots and let you use MUCH more than 32GB of RAM.

Your best bet is probably to go to 32GB of RAM, add a mirrored zil that is as small as possible(even 8GB is absurdly huge as you'll never hit 200MB), and go with an L2ARC that is 64GB and hope for the best(in that order!). You also should get rid of the RAIDZ1 and go with mirrored vdevs for higher iops. Other than that, you aren't going to get much more from that hardware without more RAM, which means a whole new system.

Welcome to the world of ZFS, where system requirements skyrocket as soon as you say "I want to run a VM".

 

[ZFS Noob]

I'm doing testing on a FreeNAS machine I built for 3 Xenserver hosts.

My guess is that IOPS matter more for your performance than raw throughput, and as such you should be building your server with performance in mind. Off the top of my head:

• Ditch RAIDZ1. As Cyberjock said it's irresponsible, because according to a quick & dirty calculator on the web, your chance of successfully resilvering after a drive failure is 62% in that configuration. In other words, when you try and replace a failed drive there's a 38% chance you'll lose the entire pool. Your data (and your peace of mind) deserve better.
• As IOPS matter, you should probably be considering RAID10 instead (or a stripe of mirrored vdevs, or whatever we're supposed to say in ZFSlandia). RAIDZ1 in ZFS gives ~ the same speed as a single drive; RAID10 does much better. Plus it's more resilient.
• In a quick test on my new server I was getting > 9,000 IOPS using a read/write pattern consistent with my databases before any real optimizations. This is awesome, but it only happened because I have enough RAM to cache frequently used data in an ARC. If you're running VMs, you'll want more RAM.
• Using a separate SSD for your ZIL is good for NFS connections, because NFS with VMWare is 100% synchronous writes. But use the right SSD (one with a super-capacitor is best, minimum should be an Intel 320), and add it correctly (which means partition it to be the correct size and add it from the command line.)
• Don't use an L2ARC. ARC > L2ARC, and L2ARC reduces the amount of ARC available because tracking all that data on the SSD requires memory. Worse, you don't really have enough memory to support any reasonable amount of L2ARC, so just leave it alone. Add RAM first, and when that's maxed if you need more cache then it's time to talk L2ARC.

 

[ZFS Noob]

Would i be better served using ufs instead of zfs?​

Nope. Part of the magic of ZFS is that you end up with a hybrid storage system, where rarely used data is stored on hard drives and the regularly accessed stuff is in RAM or on an SSD reserved for the purpose. UFS gives that up.​

Plus, ZFS + an SSD SLOG handles synch writes better than UFS possibly can, and you'll be dealing with nearly 100% synch writes which are unbearably slow otherwise.​

 

[jwhitt]

I like the raid 10 idea, With the 6 drives i currently have that would give me enough space to start, and i can replace them if need be with larger drives in the future. It sounds like best course of action for the zil device would be to partition it and add it manually. Should i still go with a 8GB Slice even with 32G of ram? Or would 16G be a better choice?

Thanks for all this info.

 

[jwhitt]

Any thoughts on multipathing network traffic? I do have several Dual port pci-e Intel and Broadcom nic's hitting 125Mbps is for chumps :)

 

[cyberjock]

I like the raid 10 idea, With the 6 drives i currently have that would give me enough space to start, and i can replace them if need be with larger drives in the future.

Good luck with that. I gave you my recommendation already.

It sounds like best course of action for the zil device would be to partition it and add it manually. Should i still go with a 8GB Slice even with 32G of ram? Or would 16G be a better choice.

I already answered that question...

Your best bet is probably to go to 32GB of RAM, add a mirrored zil that is as small as possible(even 8GB is absurdly huge as you'll never hit 200MB), and go with an L2ARC that is 64GB and hope for the best(in that order!).

Do you actually intend to either ignore my recommendations or reword your question hoping I'll give a more desired answer the second time?

I also said to read around the forums, and if you actually stop and do that you'll see people that are running VMs with 64GB+ of RAM, 6+ disks in mirrors, bigger L2ARCs, etc.

/faceplam

 

[ZFS Noob]

Should i still go with a 8GB Slice even with 32G of ram? Or would 16G be a better choice?

Smaller is better. Again, n00b speaking here, but as I understand it:

• ZFS is transaction based. It will sit there with the pending changes in memory, then write them all at once every 6 seconds or so by default.
• SSDs wear out, so if you underallocate your SSD when you add it as your log its internal logic will do its wear-leveling thing to maximize its life. The more unused space it has, the more wear leveling it can do and the longer it should last.
• SLOGs don't need to be that big. The maximum size that will ever be used is two transaction groups - anything larger than that is by definition wasted. So if you're going with 3 mirrored vdevs, each of which can write 120 MB/second, then you're looking (120MB/s) * (3 vdevs in the stripe) * (6 seconds in a transaction group) * 2 (transaction groups) = 4,320 MB as the maximum size that will ever possibly be used. That's your ceiling. Now, look at network connectivity. If you're connecting via 1 GbE connection over a SOHO switch that doesn't support jumbo frames then you'll possibly see: (80 MB/s) * (6 seconds in a transaction group) * (2 transaction groups) = 960 MB as your max size. That's your floor. Now does cyberjock make sense when calling your proposed sizes huge? It'll never be used.

(And if cyberjock and I have different recommendations, defer to him. I'm new to all this, and he's got a much greater understanding of all this stuff. I'm just trying to answer questions because I think I explain things well... So if he says 32G RAM is enough for an L2ARC and that's the way to go, then it probably is.)

 

[jwhitt]

CyberJock, I apologize for the improper wording of my response. I did not understand the math that ZFS Noob later explained.
ZFS Noob: Thanks for your response.

The only reason i was asking about the 16G slice was that i had read that the zil can only go half as large as the amount of ram in the system.

My switches aren't super awesome, however their the basic Power Connect series and advertized capacity of 48 and 32Gbps. I do have the capability of LACP Bonding, however that's useless in this situation as im talking 2-4 hosts. I think the best shot is multi-pathing and i will look into that setup.

Also im slightly kicking myself for not buying this board
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182341

Just wasn't in the budget. Thanks for your input guys.

 

[cyberjock]

Your ZIL has multiple limits that determine its final size. Normally, it stores something like 1 second's worth of writes based on your pools performance. The way it comes to that size is not exactly optimal, but its supposed to be a quick and dirty guestimate. So unless you have a REALLY fast pool, even 1GB is pretty darn big.

Keep in mind that everything in the ZIL is always also in RAM. So if you have 16GB of RAM there's no way you could possible have more than maybe 2GB in an "absolute biggest write cache you could imagine regardless of the cost to get there" scenario. The ZIL is not meant to replace the data that is to be written to the pool. It's meant to be a non-volatile store for data that is to be written to the pool. That's all. So your average ZIL will be almost 100% writes over its lifespan.