FreeNAS/TrueNAS Best Practices for VMware NFS datastore [2018]

[JalapenoNimble]

What are the FreeNAS/TrueNAS Best Practices for VMware Datastores?

Current Draft Plan:

• 22 mirrored vdev Setup with 10TB Helium SAS drives (220TB usable before overhead, 44 total drives)
• Running on Dual Xeon 10 Core CPUs with 256GB RAM and a P4800X ZIL and dual 40GBE.
• 12 x VMware ESXI 6.7 Hosts environment: Running RHEL Linux VMs
• 16-64kb record size with compression on, dedup off. [workload analysis required]
• NFS for faster throughput but iSCSI for lower latency?

Anyone using FreeNAS/TrueNAS at work as a VMware NFS datastore? Curious what hardware your using and challenges you have had.
All feedback is welcome.
Especially interested in professional experience with 24+ drive NFS datastore environments?
How fast is she under load and any lessons learned?



Tips found so far:
Increase FreeNAS # NFS Servers set to >24, perhaps 48
https://forums.freenas.org/index.ph...nalyzer-workers-any-tuning-suggestions.62762/

I believe NFS is now recommended over iSCSI for VMware datastores in contrast to this 2015 thread:
https://forums.freenas.org/index.php?threads/esxi-nfs-datastore-what-are-best-practices.26947/

Do these Oracle ZFS ESXI Best Practices apply to FreeNAS/TrueNAS, anyone try them?
Best Practices for Configuring Oracle ZFS Storage Appliance and VMware vSphere 6.x with NFS Protocol White Paper [2017]
https://community.oracle.com/docs/DOC-1013472

 

[dlavigne]

This whitepaper is in need of updating but does contain some useful info: https://www.ixsystems.com/TrueNAS_for_VMware_WhitePaper_PDF . If you have questions on some of its recommendations, we can queue them up to be addressed when the paper is updated.

 

[zambanini]

do not use nfs

 

[HoneyBadger]

This whitepaper is in need of updating but does contain some useful info: https://www.ixsystems.com/TrueNAS_for_VMware_WhitePaper_PDF . If you have questions on some of its recommendations, we can queue them up to be addressed when the paper is updated.

I'd suggest something in there regarding the performance you can gain/lose by setting the proper recordsize on your dataset to match the type of work you'll be hitting it with.

@JalapenoNimble - As alluded to in the Oracle ZFS+NFS whitepaper, 64KB is a bit of a large record unless you're storing DB log files - if you store actual DB data files which might use 8KB pages on top of a 64KB recordsize dataset, you're going to horribly punish your random I/O. Match it as closely as possible to your known workload.

For "generic workloads" it's a balancing act - making recordsize smaller tends to trend towards lower small random I/O latency, but worse streaming/sequential performance; the inverse applies for larger recordsize. For what it's worth, the default iSCSI zvolblocksize is 16K on FreeNAS.

The other thing you'll want to ensure is that you reformat your P4800X drives to use 4K native sectors.

 

[JalapenoNimble]

do not use nfs

Ok, I'm curious why do you say that? I've read in some testing that NFS destroyed iSCSI for VMware workloads. A good ZIL is required of course.

One example:

Still in the initial testing phase, but swapping to 48 NFS servers on our pure SSD array (4x 2TB samsung 850 pros with an Intel p3520 ZIL) shows almost a DOUBLING of IOPs and halving of latency over previous tests (iSCSI included). Before iSCSI was a marginal victor, now NFS is absolutely smashing it.

Thanks for the tip! Anything else you got up your sleeve? Any gotcha's regarding running a high server count for NFS?

https://forums.freenas.org/index.ph...nalyzer-workers-any-tuning-suggestions.62762/

 

[JalapenoNimble]

This whitepaper is in need of updating but does contain some useful info: https://www.ixsystems.com/TrueNAS_for_VMware_WhitePaper_PDF . If you have questions on some of its recommendations, we can queue them up to be addressed when the paper is updated.

So what is the recommendation for TrueNAS supporting VMware ESXI datastores with choosing NFS/iSCSI? The white paper does not directly address this but appears to lean iSCSI.
I understood that M40/M50 NVDIMM for the best caching performance required NFS communication.

 

[atakacs]

So what is the recommendation for TrueNAS supporting VMware ESXI datastores with choosing NFS/iSCSI? The white paper does not directly address this but appears to lean iSCSI.
I understood that M40/M50 NVDIMM for the best caching performance required NFS communication.

Did you get anywhere with this setup ?

 

[K_switch]

I would love the feedback from more seasoned experts than myself on this comment :) It worries me to see 22 mirrored VDEVs... in my experience (mind you, in small to medium sized businesses) whenever i use a large number of mirrored VDEVs I always run into an issue with disk failure and the Silvering process taking the other drive with it. with the sheer amount of drives wouldn't it make more sense to run several RaidZ2/Z3 VDEVs? I know that Mirrored is the best possible performance configuration, but two disks failures could ruin your BEAUTIFUL Pool.

Thanks in advance!

 

[HoneyBadger]

but two disks failures could ruin your BEAUTIFUL Pool.

"That's what backups are for." ;)

The performance difference between mirrored vdevs and raidzN can be absolutely massive for certain workloads (eg: random I/O) and in short it's considered worth the risk of running the large number of mirrors.

 

[K_switch]

That's what backups are for

You are 100% correct, Thank You!!!