*Should* I FreeNAS it? Esxi / NVME / High ram.

[Labtester]

I have a stable Esxi 6.5 host running with a small SSD boot drive and a 1TB Samsung 960 Evo for the datastore, running half dozen windows VM's for a home lab. The VM's currently all use virtual NVME controllers to talk the their VHD's on the datastore. I have a large amount of ECC RAM (by home lab standards, or at least my own)-- 256 GB-- and am not CPU limited either for typical usage.

I'm experimenting with increasing the responsiveness of the VM's, including reboot times. I'm evaluating FreeNAS as a VM on the same host.

Typical VHD size is < 100GB, but that is heavily duplicated across the VM's. With efficient deduplication I could run them all out of system RAM ; but Esxi apparently only supports very small RAM caches. Of course, Windows does file caching itself, but that's very RAM inefficient (doesn't take advantage of the heavy reduduncy between the VM's, and doesn't help with reboots).

So, I've discovered FreeNAS. It may be overkill for my applications: I'm still only planning on using the one NVME drive! But, FreeNAS appears to have a stack of useful features: Efficient deduplication and RAM caching. I can passthru the whole NVME drive to FreeNAS and give it dedicated (reserved) RAM on the holst, and I'm willing to work with write back caching mode for my applications. OTOH, a FreeNAS VM on the same machine might (?) be bottlenecked by its own VMXNet3 virtual NIC.

So the question is: Is this likely to speed up the performance of the VM's ? Is it worth the additional complexity of installing FreeNAS and migrating the VHD's to it (say, as ISCSI)?

Thanks,
LT

 

[m0nkey_]

So the question is: Is this likely to speed up the performance of the VM's ?

Unlikely. You may end up doing just the opposite and slow things down.

Is it worth the additional complexity of installing FreeNAS and migrating the VHD's to it (say, as ISCSI)?

Again, no it's not. You're still going to be limited to the IOPS of the device. You can increase IOPS by adding a second device and put them in a mirror.

 

[Labtester]

Hold on... the principal advantage I am seeking from FreeNAS is the RAM caching ("ARC"). Under circumstances where RAM dedicated to the FreeNAS VM approximates (likely, is greater than) the working aggregate hard drive sizes of the VM's, wouldn't the IOPS of the ARC (the RAM ) be what matters? Obviously, FreeNAS isn't going to make my NVME hardware itself any faster (without RAID, etc.) Effectively, I'm hoping to hide that "slow" performance behind a giant RAM cache. The surprising thing (to me) is that ESXi won't do that itself.

Another consideration is using a Windows 2016 server VM as an iSCSI server.

 

[m0nkey_]

You're adding unnecessary overhead by adding a virtual layer to your VM storage. You can by all means try it, but I really don't think you're going to get any performance gains.

 

[Ericloewe]

Why... do you need half a dozen Windows VMs?

 

[Stux]

IOPS shouldn't be a major concern with 960
Evo as a drive.

Nor should the ZIL I guess.

Passing through the drive is certainly the way to do it in a data safe fashion

I'd use iSCSI.

Might be worth considering dedup and Compression on the pool.

You can replicate the pool to spinning rust as a backup.

 

[tvsjr]

Why... do you need half a dozen Windows VMs?

I've got almost a dozen :p

 

[Labtester]

The system is already pretty fast running esxi with the evo as datastore, it's a great drive. But nothing beats RAM for latency (or throughput). I'll give it a test run with freenas on top of esxi using 128gb ram to cache a 64gb iscsi disk (this, a crude ram disk) for benchmarking (vs using existing vm drive). The intent will be to test the iscsi/vmxnet3 stack.