vmxnet3.ko for FreeNAS 9.X in ESXi 5.5

[David E]

The vmxnet3.ko in VMware's ESXi 5.5 tools does not work with FreeNAS 9.2.X or 9.3, I have recompiled the (unmodified) source code for this driver, and am making available a ready to go kernel module for the FreeNAS versions listed below. You will want to temporarily use the E1000 NIC type so you can copy the driver into FreeNAS, then change it to VMXNET3 on the next boot.

For instructions on how to install the driver, see the latter part of the following post:
http://itanalyses.blogspot.com/2013/02/jumbo-frames-in-virtual-freenas-831-on.html

Instructions for compiling yourself
Rough guide of the necessary steps, substitute 9.2.1-RELEASE below with the FreeNAS version you are interested in. Updated to include ajohnson's faster build below:

• Set up a FreeBSD 9.2 VM in VirtualBox
• Create and attach a second hard drive to the VM (20G or so)
  • Find the disk, ls /dev look for ada*
  • zpool create tank ada1
• Install Dependencies
  • cd /usr/ports/devel/git
  • make install clean
  • cd /usr/ports/sysutils/cdrtools
  • make install clean
  • cd /usr/ports/ports-mgmt/pbi-manager
  • make install clean
• Check out FreeNAS
  • cd /tank
  • git clone git://github.com/trueos/trueos.git
  • cd trueos
• Find version to build against
  • git tag -l
  • git checkout 9.2.1-RELEASE
• Unpack vmware tools
  • scp the tools to the vm in /root (vmware-freebsd-tools.tar.gz)
  • cd /root
  • gunzip vmware-freebsd-tools.tar.gz
  • tar xf vmware-freebsd-tools.tar
  • tar xf /root/vmware-tools-distrib/lib/modules/source/vmxnet3.tar
• Build
  • cd /tank/trueos
  • make buildenv
  • cd /root/vmxnet3-only
  • SYSDIR=/tank/trueos/sys make

Resulting kernel module will be in /root/vmxnet3.ko

Determining your kernel version for FreeNAS 9.3
As of 9.3 FreeNAS has moved to a much more frequent release process, that is revving the kernel across patches, but not updating the overall version number. To determine what version of the TrueOS kernel you are using, you'll need to find the MANIFEST file for the version of FreeNAS you are using. Start here: http://download.freenas.org/9.3/ pick the release train, ie STABLE, then the date of the ISO you installed, then the MANIFEST file. Inside it look for the line:

git@gitserver.ixsystems.com:/git/repos/freenas-build/trueos.git 4663c83

The last group of letters and numbers indicates the git revision, and corresponds to a revision at the Github repository https://github.com/trueos/trueos

Armed with this revision, follow the same instructions above for building, but instead of checking out a tag revision, check out the exact revision for your build, ie:

git checkout 4663c83

Available Kernel Modules

• ESXi 5.5 FreeNAS 9.3-STABLE-201503270027 (Git rev e006356) shasum: 0ec8c6b4992432cc438a1f03b201eac10645eabe
• ESXi 5.5 FreeNAS 9.3 x64 shasum: 4ea6e8dc8069fbcbb735599bf76286bb8a58b721
• ESXi 5.5 FreeNAS 9.2.1.6 x64 shasum: ff156321029d71f353f8752878a2e6d157f93f22
• ESXi 5.5 FreeNAS 9.2.1.3 x64 - use the module from 9.2.1.1
• ESXi 5.5 FreeNAS 9.2.1.1 x64 shasum: 378cf7b6d4267991b31e412cc90ddc4b1a6afb59
• ESXi 5.5 FreeNAS 9.2.1 x64 shasum: 69baa7d0621ee604277cb6768a504f79cb72b872
• ESXi 5.5 FreeNAS 9.2.0 x64 shasum: 831f99c967713c88a02fa240c2542cf3b41cbd7d

 

[ser_rhaegar]

Quick and easy to load, thanks!

 

[FREAKJAM]

Can someone verify that it also works with FreeNAS 9.2.1 & ESXi 5.5?
I'm getting "no configured interfaces were found" after booting :(

 

[AkumaKuruma]

this will only work with FreeNAS 9.2.0. a fresh compile needs to be made against 9.2.1 to make it work there.

I would definitely be interested in a write up of the steps you are taking to create the file. some of us would like to be able to compile other variations or need to do an in house compile for QA reasons.

 

[David E]

Ya I'm working on that, as well as a compile for 9.2.1, however Github is currently checking out a multi-100 MB repository to me at ~30KiB/s.. so it'll be awhile :|

 

[cyberjock]

I know several people would appreciate a how-to on compiling this stuff. This is often asked for in various fashions. Usually it's someone using "less than the latest version" of FreeNAS with the "less than the latest version" of ESXi. In my case, I'm using 9.2.0 with ESXi 5.1. ;)

 

[David E]

Added a module for 9.2.1 x64, and rough instructions for doing it yourself. Its possible I missed steps since I made the instruction list a couple weeks after doing it myself, let me know and I'll be happy to update the original post.

@cyberjock: You may want to try the 5.5 tools build with 5.1, they tend to try to stay pretty backwards compatible with the OS-level drivers.

 

[cyberjock]

And the 5.5 vmxnet3.ko does work with 5.1. Woohoo. Not too useful for my purposes. But it does work!

 

[FREAKJAM]

I can confirm the latest vmxnet3.ko works with FreeNAS 9.2.1/ESXi 5.5. Thanks alot!

 

[ajohnson]

Thanks for documenting all of this David. I have a compile setup on my end for this as well. It's working quite nicely.

I do have one question -- when a FreeNAS update occurs, do you do a full make clean / make release cycle or have you worked out some way of git pulling and compiling updates without needing to do a make clean (would be much faster in theory).

 

[David E]

I was planning on testing this tonight for the 9.2.1.1 release, I think I can just stash/pull --rebase/stash pop on the parent repo, delete the os-(I forget the name here) build directory, and fetch/checkout the new release tag for the two sub repos. I'll give it a go and report back.

 

[ajohnson]

Great, thanks much.

Also, I wanted to contribute the method I was using before I read this thread. It actually used 'make buildenv" to set the environment up so that it builds using the FreeNAS/FreeBSD toolchain in such a way that you can build the driver outside of the source directory. I don't know that this is really more useful than your method, but I'll put it here in case it is of interest. I guess the one benefit of this approach is that you don't actually need to build the full FreeBSD kernel / FreeNAS release if you don't want to -- you can just build the kernel module.

(Before performing these steps, follow David's instructions up to and including "Check out FreeNAS". I use a slightly different directory for the FreeNAS source below, so just consider that.)

Instructions for creating a vmxnet3 driver for FreeNAS from outside the FreeNAS build directory:

Code:

# Set the environment up to use the FreeNAS/FreeBSD build toolchain instead of the system's default toolchain:
cd /usr/local/freenas/FreeBSD/src
make buildenv
 
# Assuming your vmware-tools-distrib is extracted in a directory structure as below, do the following:
cd /usr/local/vmware-tools-distrib/lib/modules/source
tar xvf vmxnet3.tar
cd vmxnet3-only
SYSDIR=/usr/local/freenas/FreeBSD/src/sys make
# The .ko file should now live in /usr/local/vmware-tools-distrib/lib/modules/source/vmxnet3.ko

 

[David E]

So fetching then checkout of the new tag, combined with deleting the os-base directory worked fine for me, and I added 9.2.1.1.

I also confirmed building your way worked great, I figured there was probably a far cleaner way to build just what was needed, but my FreeBSD knowledge is pretty limited so I went with the hacky method :)

 

[ser_rhaegar]

Are you guys cloning the FreeBSD source into the FreeNAS directory? I was following David's instructions however there is no FreeBSD folder under FreeNAS nor any submodule in the FreeNAS git that I can see.

 

[spidernas]

I've been trying over the past few days to compile the module myself as well for 9.2.1.3, but without success. Seems that some paths have changed as well as the build environment.

If anyone could help with updated instructions or a new binary module posted, I would really appreciate it.

Thanks.

 

[David E]

I recompiled for 9.2.1.3 and it had the same shasum as 9.2.1.1, so I assume there were no kernel changes that affected its compilation, so feel free to use 9.2.1.1 for 9.2.1.3.

wrt to compilation, probably the easiest thing to do at this point is to directly git clone the kernel source at https://github.com/trueos/trueos, check out the tag for the release, then follow ajohnson's build instructions.

 

[spidernas]

I'll try the 9.2.1.1 module. Thanks David!

 

[ddimick]

Annoyingly the 9.2.1.3 update appears to have blown away the module in /boot/modules. Keep a copy of it somewhere else on the NAS locally or you'll wind up adding an E1000 again just to get it back on there.

 

[ser_rhaegar]

Annoyingly the 9.2.1.3 update appears to have blown away the module in /boot/modules. Keep a copy of it somewhere else on the NAS locally or you'll wind up adding an E1000 again just to get it back on there.

Every update will blow away any file system changes to the root file system.

 

[pbucher]

Every update will blow away any file system changes to the root file system.

Very easy workaround...I've been doing this since version 8.3. Download the ISO and attach it to the VM and upgrade using the ISO image. It keeps the files in /boot/modules every time.

Also the stock binaries with 5.5 work great with 9.2.1.2 and newer. I've pumped several TB of data through them using jumbo frames.