SOLVED Hardware Recommendations for FreeNAS and ESXi

[brando56894]

I did another test with the same file, within FreeNAS, from dataset to dataset using rsync and got the same results again.

Sent from my AOSP on dragon using Tapatalk

 

[joeschmuck]

Thanks for letting us know what your use cases were, I didn't see anything that required high transaction rates. Even without the testing we could have told you that the SLOG and L2ARC would not be a good idea, however I know you have stated that you like proof so I guess you are validating it for yourself.

Since you like to test, if you devote the bulk of your ESXi RAM to FreeNAS (48GB+) then you could try to connect the SSD as an L2ARC ONLY in pass-through and see what those results are. Keep in mind what an L2ARC is for because if you are not opening up the same file repeatedly then you will have no gains and will actually introduce losses. Do the testing with 48GB+ RAM first, then add the L2ARC and test again. I'm not sure what software suit you are using to test with either but the Intel NAS performance suite works well and gives you consistency. I actually have a posting that was primarily directed at Intel NIC vs. RealTek NIC but it covered all the basic speed tests, both internal and external. Here is the link in case this can lend some assistance.

 

[brando56894]

FreeNAS already has the majority of my RAM at 46 GB, I'm slowly adding more to it as I monitor how much RAM the other VMs need. The whole problem with this testing is that I can't pass through my SSDs without detaching one of my pools since ESXi won't allow me to pass through SATA ports, only PCI devices.

For the L2ARCs I generally do open the same files on SafeKeeping and on Storage I figured it would be useful for caching all of the metadata for Kodi (episode info and artwork) and Plex since they are used every day and the same files are always scrolled through. I know having too big of an L2ARC when you don't have enough RAM can hurt your performance, so I was cautious of making them too big. SafeKeeping's is at 40 GB and Storage's is at 60 GB. Looking at my ARC stats L1 is at 89% hit ratio and is 41 GB, L2 is at 4% Hit Ratio and is 60 GB. So not much use out of the L2ARC, but then again my uptime is only 1 day and 22 hours so far.

I haven't been using any test suites, just simple copy tests. I didn't even know that product existed so I'll give it a go, thanks!

Sent from my AOSP on dragon using Tapatalk

 

[toadman]

Makes sense. He'd probably gain more benefit by expanding his RAM to 128GB than by using an L2ARC, too.

Yea, more ram always helps. But I run basically the same set of workloads on a 32GB system and it's pretty lightly loaded (60% memory and normally 10% of a 6 core cpu) on average. The FreeNAS vm that stores all the data on a single pool (vs. the three he has) only has 8GB. Performance is fine. (Were I running a windows VM on there it would clearly benefit from a local SSD datastore. As of now everything but the FreeNAS VM is on an NFS datastore served from the FreeNAS.)

So I'd save that money unless there is an identifiable performance issue. And if there is spare equipment, please just send it to me. :)

 

[toadman]

Looking at my ARC stats L1 is at 89% hit ratio and is 41 GB, L2 is at 4% Hit Ratio and is 60 GB.

Given that, yes, any more $ on RAM will benefit you (for ARC) more than for L2 at this point I suspect.

 

[joeschmuck]

If you want to give the L2ARC a test I would suggest you disconnect your "SafeKeeping" pool and use one of those SATA ports.

But to be honest here, I would just lower your FreeNAS RAM to 16GB, lock it, and call it a day, no L2ARC or SLOG. While this might be a good learning experience but there is definitely risk to your data by playing around. I am running a few Windoze 7 VMs (4GB RAM allocated) using iSCSI from my FreeNAS VM, no problems at 16GB RAM so far but my use case isn't compiling programs or any heavy lifting. So far my system is very stable. If the iSCSI became unstable then I can afford to allocate more RAM to FreeNAS. Using the VX3 NIC driver and having everything on the same machine makes the Windoze VM boot up very fast. Of course placing your Windoze VMs on a SSD attached as a datastore will be very fast too and might be the right path for you.

 

[brando56894]

I'll probably remove the L2ARCs and keep the SLOGs off, but I see no reason to reduce my RAM by 30 GB since all the other VMs collectively use only about 10 GB.

Sent from my AOSP on dragon using Tapatalk

 

[joeschmuck]

I'll probably remove the L2ARCs and keep the SLOGs off, but I see no reason to reduce my RAM by 30 GB since all the other VMs collectively use only about 10 GB.

Sent from my AOSP on dragon using Tapatalk

Very well. FreeNAS will be very happy with all that RAM.

 

[toadman]

I'll probably remove the L2ARCs and keep the SLOGs off, but I see no reason to reduce my RAM by 30 GB since all the other VMs collectively use only about 10 GB.

Yea, if you already have it, use it. It's just not clear adding any more would provide a demonstrable performance increase. (Definitely make sure the Freenas VM is reserving all it's memory on boot in the ESXi config. I can't remember if you said you did so.)

I think joeschmuck is suggesting that given your workload you may not see a demonstrable decrease in performance if you lower the freenas RAM (and say use it elsewhere, some other server or something). Obviously if you don't have another use for it, keep it with freenas. :)