VMWARE Tools and Freenas 8

[cyberjock]

Put simply. you don't. There isn't space to install it on the OS partition.

 

[Mordock]

All right, let me try to be polite and make this perfectly clear. The VMware tools are not optional. This is not a matter of opinion that you and I can argue about, it is plain fact. VMs will not operate properly in a VMware ESXi environment without the VMware tools installed. If you have been seeing issues with VM implementations, I have no problem what-so-ever concluding that at least half of those problems were due to not having the tools installed, or not installed properly or up to date. The number one troubleshooting response for all VMware issues, particularly performance issues, is to verify that the tools are installed and UP TO DATE. To include a section in the FreeNAS manual on installing FreeNAS in a VMware VM without installing the VMware tools, or in a Hyper-V VM without installing the Hyper-V tools, is not just a minor oversight, it is a major, major, MAJOR PROBLEM!! This is not debatable, this is an absolute requirement. So, VMware needs to be completely removed from all documentation leaving only a "not supported" statement, or somebody needs to spend a few minutes to write up step-by-step instructions on how to install and update the VMware tools. And don't make any assumptions on what the user knows about Linux/Unix. Remember that while there are maybe 10% of people that know some Linux, there are almost no Unix experts left out there in the real world. So these instructions should not assume that anyone knows even how to use a simple mount command; state it and every command necessary down to the cd commands to change directories.

BTW, since the VMware tools installation script requires Perl and compat6x-amd64, it would be reasonable to include these in the basic FreeNAS install package as part of a VMware support initiative. In conjunction with that, it would be nice to see some documentation on scripting the creation and deletion of iSCSI LUNs and other activities whether with Perl or some other scripting tools.

So as Yoda would say, there is no try. Either support VMware or don't and that means tools.

 

[cyberjock]

Can you elaborate on why you said this:

The VMware tools are not optional. This is not a matter of opinion that you and I can argue about, it is plain fact. VMs will not operate properly in a VMware ESXi environment without the VMware tools installed.

I'm asking because I've had FreeNAS running in VMWare Workstation 8 and 9 without problems(aside from ones I created for experimenting) without installing the tools. Also, I did do some experimenting with VMWare ESXi 5.1 in December and my hardware didn't support PCI passthrough(which is a MUST for anyone trying to use ZFS with ESXi) and I don't remember having any performance issues or any other problems to speak of. Of course, I never got to the "use this setup in a production environment" stage so I may not have found any problems because I never fully implemented and used it extensively.

When you say "not work properly" can you elaborate? What in particular is broken?

Also, even if some things don't work right, I'd still say that a virtual machine is an amazing tool for someone that has never touched any Unix or Linux based OS. Most people wouldn't be quick to run out and buy new hardware just to experiment with an OS they've never touched and may not work properly for their given application.

If you read through the forums, it's made pretty clear that virtualizing(especially ZFS) shouldn't be done in production environments without being an advanced user of virtualizing. I'm not saying that the manual says this(it doesn't afaik) but there are alot of people that think they can virtualize FreeNAS without any ill side effects and they complain when they lose their data. Quite often they lose data because they used RDM with ZFS and that's really not a good idea at all. To be fair though, the manual makes no mention at all about how you should(or shouldn't) create your zpool in a virtualized environment, or if it even recommended. It only provides steps to installing FreeNAS and then stops there.

 

[ProtoSD]

No one ever did answer the basic question. How do you install the current version of VMware tools in FreeNAS?

All this bickering, bitching, and whining... I know this topic has been discussed in another thread:

Open a command prompt and tell us what this command shows:

ls -ld /usr/local/lib/vm*

open-vm-tools has been included with FreeNAS at least since version 8, how the hell else are they going to test stuff without lots of extra bare metal???

 

[cyberjock]

But wasn't the place where it was discussed(I vaguely remember now) that it didn't include all of the vmware tools?

 

[Mordock]

When you say "not work properly" can you elaborate? What in particular is broken?

First, compared to ESXi with vCenter, VMware workstation is a toy. But even in workstation, every time you download an update to workstation, you are also getting an update to the tools that needs to be installed in all of your VMs.

ESXi with vCenter is massive. I teach 2 different 5 day (10 hour a day) bootcamps (Administration and Advanced) on VMware. I wrote our 4 day bootcamp on desktop implementation (VMware View) and our 4 day class on VMware Troubleshooting as well as collaborating on the other 2 classes.

The VMware tools provide a number of features to the ESXi environment starting with all the drivers for keyboard, mouse, video, network, disk, memory, and all of the virtuallized motherboard components such as IDE, PCI, PS2, and SIO controllers, etc.

In the case of ESXi, there have been 9 updates to the VMware tools for VMware ESXi 5 since its release 17 months ago, of which 6 were deemed as "Important BugFix"s, 3 as "Important Security" and 1 to add "Moderate Enhancements". There were also a number of "Critical Security" updates to the 4.0 and 4.1 VMware tools in the same time frame including one released 2/7/2013. So far there have only been 2 to ESXi 5.1, 1 each "Important BugFix" and "Important Security", both released 12/20/2012. While I don't know what version of drivers is supplied with FreeNAS, I am certain they do not include these most recently released “Important BugFix” s.

Of most obvious concern are Memory, Disk, and Network, in that order. Keep in mind the first 3 steps when creating/optimizing a VM for ESXi are:
1. Install the tools
2. Install the Paravirtual SCSI controller
3. Install the VMXNET3 network controller

Memory: VMware ESXi provides 3 levels of memory management beyond that of normal operating systems. Of course it has a basic swap file system like all operating systems, but this is considered only a last resort after the other 3 have been expended. Two of these are handled totally transparent to the VM (transparent page sharing and memory compression). Note: memory compression is only done to avoid swapping as compression would be faster than going to disk. The one of concern here is what is commonly called the Balloon driver that is part of the tools. This driver allows the VMware kernel when under memory contention, to request that the guest operating system in the VM (FreeNAS in this case) swap some of its memory rather than have the kernel try to guess what memory would be best to swap from the outside. The example I often use in class is: If the kernel were to try to guess from the outside, it might determine that a block of memory that has not recently been used and would be a candidate for swapping might actually contain the keyboard drivers since no-one has logged into the VM recently and used the keyboard. No OS would knowingly swap out the keyboard drivers. Only the guest OS has the knowledge and expertise to manage its own memory, thus the balloon driver in the tools. To be honest, if I were using FreeNAS for production, I would recommend reserving all of its memory so that it could not be swapped at all by either the balloon driver or the kernel.

Disk: Two issues really here. One is in response to some of the comments earlier in the thread. There were problems with the paravirtual SCSI controller in ESXi 4.0. Most of them were resolved by 4.0 update 2 and the paravirtual SCSI driver has further been enhanced since then. In addition, VMware now has a fully supported tools release for FreeBSD which they did not have that long ago. I have seen some comments in other threads about using 3rd party tools, which is a really bad idea now that VMware is fully supporting FreeBSD. Of course this is all predicated on having ESXi fully patched and having the tools updated to the current version. Again, I have no idea what drivers are in FreeNAS, whether they are third party or VMware official drivers. The VMware client is telling me they are "3rd-party/Independent" which could mean actual 3rd party or simply that not all the pieces are there (independent).

The second issue is the ability to quiese the disk. Others have mentioned this as an issue with backup, but that is just one manifestation. One of the primary features of VMware is DRS (Distributed Resource Scheduling). This allows VMware vCenter to balance the CPU and Memory load across multiple hosts by moving VMs from one host to another automagically with VMotion. VMware will give you a warning if it is not able to quiese the disk during a VMotion as this slows down the VMotion process and makes it more susceptible to problems. The disk is also quiesed for snapshots and other operations.

Network: VMXNET2 supported 64-bit OSs and VMXNET3 add support for various new features including some of the offload capabilities to physical NICs. Having the most recent version of the tools guarantees that these operations occur efficiently and properly. This could very well be the cause of some of the performance issues I have seen in the forums comparing FreeNAS network performance against other versions of Linux and Windows.

Now I didn't specifically answer your question about what is broken. Of course you can look at the release notes of the various updates but here is an excerpt from one of them:

PR 837005: Large number of UDP packets are dropped when you use the VMXNET3 adapter with a Linux guest operating system installed on an ESXi 5.0 host.

PR 837008: RHEL 6 virtual machine configured with Raw Device Mapping (RDM) devices through Paravirtualized SCSI (PVSCSI) adapter might encounter I/O failure after bus reset on an ESXi 5.0 host. During reset, the PVSCSI adapter causes the I/O to fail instead of executing the retry command.

PR 838575: The network scheduler uses a single queue to allocate bandwidth between all virtual machines sharing the same network resource pool. If multiple virtual machines send network data at the same time, the available bandwidth might get exhausted before it is allocated proportionally to all virtual machines.

The improvised algorithm now uses a time based scheduling scheme to optimally allocate bandwidth to all virtual machines sharing the same network resource pool.

PR 902238: The virtual machine fails during a specific backup scenario involving quiesced snapshot, this happens because the synchronous manifest file copy operation fails due to an inconsistency in callback process and forces the system to enter an invalid state.

 

[Mordock]

open-vm-tools has been included with FreeNAS at least since version 8, how the hell else are they going to test stuff without lots of extra bare metal???

Bottom line, open-vm-tools is not the current supported tools. No OS consistently ships with the current version of any driver, VMware or otherwise. The mechanisms have to be there to upgrade the drivers as new ones are released.

Would you install any other OS, even from the latest downloaded distribution without checking for newer drivers from every manufacturer of components in your machine? Why should FreeNAS in a VM be any different? Are you really telling me that if you were installing FreeNAS on a physical machine that you would not go to the RAID card manufacturer site and download the latest drivers for FreeBSD and install them before putting the server into operation? In the case of a VM, all the latest drivers are in the tools whether it be VMware tools or Hyper-V tools. One of the big advantages of using VMware is that it is a one stop shop for all drivers. Install the latest tools and you have every driver in one operation. None of this checking the Video card site and the network card site and the disk controller site and the motherboard site.

 

[cyberjock]

First, if you run the command protosd mentioned above the files that are installed are:

/usr/local/lib/vmware-tools/modules/drivers/linker.hints
/usr/local/lib/vmware-tools/modules/drivers/vmblock.ko
/usr/local/lib/vmware-tools/modules/drivers/vmhgfs.ko
/usr/local/lib/vmware-tools/modules/drivers/vmmemctl.ko
/usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko

I'm not sure of their version, but I'd assume that FreeNAS doesn't have the absolute latest version.

Second (and correct me if I'm totally off my rocker on this) but wasn't the vmware tools for FreeBSD open sourced and made the open vm tools project that is on sourceforge at http://sourceforge.net/projects/open-vm-tools/files/. Those files, if you look are version 9.x(which seems to imply a 9.x VMware Workstation) are relatively old. Even if they aren't for VMWare Workstation(or at least not exclusively) the latest non-developmental version is dated Nov 7, 2012. Nov 7th is older than the date you mentioned(Dec 20, 2012). There is a developmental version dated 12-28-2012, but there doesn't appear to be any version released since then. So my thoughts (if I'm not completely screwed up here.. I am outside what I know about vmware tools on FreeBSD) is that the fixes you mentioned don't appear to have been released on vmware tools...yet.

Third, I'm not sure what version of the vmware tools are running. I did download the 3MB file of open vmware tools and there are alot more modules available than just the couple of .ko files I listed above. So at this point I'm somewhat questioning if openvmwaretools is the correct project and if its even well maintained(assuming that it is the right project for this discussion).

Fourth, by doing a little googling I found some websites that don't appear to mention open-vmware-tools for FreeBSD, but just vmware-tools. It may be that there is a licensing issue with including the vmware-tools with the FreeNAS installation.

And with that I'd like to say most of this conversation with what tools and modules are/aren't installed are pretty much out of my league. I've used VMWare tools with Workstation since 1.0 back in the late 90s up to 9.0.1 with workstation. I know that tools can provide some serious performance improvements for virtual machines. But I'm just not sure what version we have, what version is the latest, what/if there is a difference between open-vmware-tools and vmware-tools, etc.

Perhaps a developer with more knowledge will chime in with some information. Unfortunately, I'm going to also fallback on what I have said before... VMs are great for experimenting with FreeNAS, but not generally a good decision for 99% of admins out there. You are likely the exception to this rule, and you know enough about VMWare to make me look like I just learned my ABCs, but since FreeNAS is currently trying to implement features that help the most users for the least amount of development I'd say VMware tools isn't exactly a high priority.

If this is something you really want I'd recommend you put in a ticket at support.freenas.org. Of course make sure there isn't one already. If you can provide links, patches, code, etc to make the job easier for the developers to implement you have a very good chance of seeing it implemented quickly. I provided some source code for a driver in FreeBSD last year and it was implemented in less than 48 hours.

I know I could really use a class on ESXi. During my 2 weeks of experimenting with it I was impressed. Of course, without PCI Passthrough being supported with my server hardware there was no way I'd trust my data to a virtualized FreeNAS installation with zfs.

 

[cyberbot]

i installed FreeNAS on ESXI 5, however the NIC isn't up because it need Vmware tools for the drivers,
hopefully someone can advise how to get those installed

 

[jgreco]

And with that I'd like to say most of this conversation with what tools and modules are/aren't installed are pretty much out of my league. I've used VMWare tools with Workstation since 1.0 back in the late 90s up to 9.0.1 with workstation. I know that tools can provide some serious performance improvements for virtual machines. But I'm just not sure what version we have, what version is the latest, what/if there is a difference between open-vmware-tools and vmware-tools, etc.

Mordock is simply wrong in saying that VMware Tools are not optional. They are most CERTAINLY optional; VMware's whole business model revolves around being able to emulate a PC. VMware Tools are like lubricant that help things work better together. For example, memory management is significantly enhanced with VMware Tools, and the hypervisor can signal the VM to shut down, etc. Open VM Tools lacks a fair bit of functionality but does implement the most basic and important tools functionality in a friendly-license format.

Put differently, if VMware Tools weren't optional, Open VM Tools would not work, because it only implements a minor subset of the VMware Tools functionality.

VMware Tools also add certain drivers to the base system to make VM I/O more efficient. However, it also adds some random brokenness, so many of us gave up on vmxnet, use em, and never looked back. There are nasty ordering dependencies involved in the system rc scripts that make life painful if you're in an environment where networking startup is done in a different order than FreeBSD default, and there are some odd problems with vmxnet going deaf, stupid, and/or dumb in complex network environments.

However, anyone who wants to run a virtualized FreeNAS is likely to generally agree that tools of some sort are desirable if for no other reason than to allow managed control over system shutdown.

I believe - but haven't confirmed - that VMware Tools are under a license that would preclude inclusion/distribution in FreeNAS. However, it would be nice to have a way to add them to the system.

 

[jgreco]

i installed FreeNAS on ESXI 5, however the NIC isn't up because it need Vmware tools for the drivers,
hopefully someone can advise how to get those installed

Shut down your virtual machine, delete the vmxnet Ethernet interface(s), and add E1000 based interface(s). Problem solved.

 

[cheezehead]

Shut down your virtual machine, delete the vmxnet Ethernet interface(s), and add E1000 based interface(s). Problem solved.

It can be done, it took about 20 minutes tonight to figure it out.

http://www.liquidobject.com/index.php/2013/03/13/freenas-8-3-and-vmware-esxis-vmxnet3-adapter/