FreeNAS 9.3 VM datastore on VMware ESXi 6 VMXNET3 adapter unreliable under heavy load with MTU=9000

[Spearfoot]

TL;DR: my testing shows it is not safe to use an MTU of 9000 with the VMXNET3 network driver on a FreeNAS 9.3 VM running on VMware ESXi 6. The same configuration works very well with the default MTU value.

I've been testing FreeNAS over the last month and a half on the following hardware:

• Supermicro X10SL7-F, Intel Xeon ES-1241v3 @3.5GHz, 32GB Crucial ECC RAM
• Motherboard LSI 2308 controller passed through to FreeNAS VM via VT-d per best practices
• Disk configuration: 7 x 2TB HGST 7K4000 drives set up as a single RAIDZ2 vdev for a 10TB pool
• VMware ESXi v6.0.0 (build 2715440) booting from a USB stick
• FreeNAS FreeNAS-9.3-STABLE-201506162331 installed with the mirroring option on a pair of SSD datastores
• FreeNAS configured with 2 vCPU and 16GB RAM

I configured a separate data network using Ben Bryan's excellent guide here:

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

Part of this setup included using an MTU of 9000 on the VMware virtual switch as well as the attached FreeNAS VMXNET3-based NIC. As we will see, this turns out to be a problem.

I configured two Windows VMs (one Windows 7 w/ 1 vCPU + 2GB RAM, the other Windows 2012R2 w/ 1vCPU + 1GB RAM) and installed the CrystalDiskMark and ATTO disk benchmarks on both.

I used these VMs to test FreeNAS as a VMware datastore. I tried both NFS and iSCSI; with and without a SLOG and/or L2ARC SSD device; with and without 'sync' enabled, etc.

What I found is that, in all cases, the FreeNAS-based datastore would break down under a heavy load, in this case, when simultaneously running the ATTO benchmark on both Windows VMs. The VMs would appear to freeze and the vSphere client would display the datastore using the italicized typeface indicating it was offline. Usually the datastore would eventually 'wake up'; the VMs would become responsive; and the benchmarks would finish, though with abysmal performance.

On the FreeNAS system log I would see this sequence of messages for iSCSI-based datastores:

Code:

WARNING: 10.0.58.1 (iqn.1998-01.com.vmware:554fa508-b47d-0416-fffc-0cc47a3419a2-2889c35f): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 10.0.58.1 (iqn.1998-01.com.vmware:554fa508-b47d-0416-fffc-0cc47a3419a2-2889c35f): connection error; dropping connection

There is an unresolved Bug report about this type of problem here:

https://bugs.pcbsd.org/issues/7622

NFS-based datastores behaved exactly the same way, though without any log entries.

Note that, when run singly, the Windows VM benchmarks indicated absolutely stellar performance - I/O rates over 1000MB/s. I wanted this kind of performance for my VMs! But reliability comes before performance in my book, and these VMXNET3-based datastores simply weren't reliable. So I kept testing...

I tried re-compiling the VMXNET3 driver, per David E's excellent instructions detailed here:

https://forums.freenas.org/index.php?threads/vmxnet3-ko-for-freenas-9-x-in-esxi-5-5.18280/

But all to no avail.

When I switched over to the E1000 driver, everything worked fine. No failures, but with reduced performance - especially write performance.

Finally, I tried reconfiguring the system with the VMXNET3 driver but with the standard MTU value. Voila! No datastore failures!

Here are ATTO benchmark results from a single run. Pretty nice I/O rates for spinning rust!

Here is the same benchmark, run simultaneously with another VM. There is a clear reduction in speed, but performance is still quite acceptable:

 

[Spearfoot]

FWIW, I re-tested my iSCSI datastore setup with the default MTU value this afternoon, and it still fails as I described above. So NFS seems to be the only good choice if you want to use the VMXNET3 driver w/ FreeNAS on ESXi 6.

 

[fede2222]

Thanks for sharing the expirience... i will build a system for homeuse going directly to NFS, Default MTU, VMXNET3

 

[Spearfoot]

@fede2222 - FYI, after posting this I discovered that FreeNAS 9.3 doesn't support VMXNET3 at all and you should use the E1000 driver instead. I have done so on my all-in-one system and it works very well.

 

[depasseg]

Did you install vmware tools?

 

[jgreco]

While you should be able to add VMXNET3 to FreeNAS 9.3, I've found it to be inconsistent over the years and seems to have a variety of issues or problems when used in certain ways, large MTU being one of those.

There's very little good reason to use jumbo, though. https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/

 

[Spearfoot]

Did you install vmware tools?

Yes I did, when first I installed FreeNAS. A few weeks later, after more research, I re-installed FreeNAS because I found out that the tools are installed automatically - sans VMXNET3.

 

[fede2222]

@fede2222 - FYI, after posting this I discovered that FreeNAS 9.3 doesn't support VMXNET3 at all and you should use the E1000 driver instead. I have done so on my all-in-one system and it works very well.

Oh.. No problem i'm on freenas 8.x. I'm not totally convinced for 9.x

 

[wreedps]

Why not run Freenas on physical hardware like the documents say?

 

[fede2222]

Why you don't read official statements?
http://www.freenas.org/whats-new/2015/05/yes-you-can-virtualize-freenas.html

And.. I'm running freenas with a dedicated hardware controller with passthrough

And... The advantage of this setup is the speed of the virtual network between the esxi datastore and the virtualized freenas

 

[jgreco]

The official opinion and the forum opinion vary somewhat. Do feel free to take the official approach and also make sure to get your official support from the official blogger for the officially sanctioned technique. In the meantime, read my stickies for a more pragmatic and realistic sense of what you are likely to find help with here on the forums.

 

[wreedps]

Why you don't read official statements?
http://www.freenas.org/whats-new/2015/05/yes-you-can-virtualize-freenas.html

And.. I'm running freenas with a dedicated hardware controller with passthrough

And... The advantage of this setup is the speed of the virtual network between the esxi datastore and the virtualized freenas

I like the big WARNING.

 

[jgreco]

I like the big WARNING.

I like the parts that appear to be rewrites of my virtualization stickies.

 

[fede2222]

Jgreco...I know the official and the forum statements about that... The official statement is a business statement

My reaction was for the arrogant "read the documentation"

I take all the precautions to run virtualized freenas... Passthrough, esxi, reserved memory, ecc, etc that were commented on the forum

We are discussing about the mtu, etc... Is not necessary to throw that..

My reaction was for the arrogant reply..

 

[jgreco]

Which part is an arrogant reply? Because the way it usually works is that either we tell people things they already know and they take offense at our making sure they've seen things we think they need to know, or they just quietly do $badthing and then come back and curse us for not warning them -- after it breaks and their data is lost.

There are definitely paths forward to virtualizing FreeNAS safely, but here in the forums it is a normal progression of events for $newuser to come in with an existing box and preconceived notions and to disregard the existing documentation here. I wrote that stuff as a way to save people who lacked the necessary skills from making an awful mistake, and also to help those who are pros to learn the pitfalls without making all the normal mistakes themselves.

We are not paid iX employees here, so please try to remember that the community is made up of people who are simply trying to help you out.

 

[fede2222]

It's not against you jgrego... I read a lot of you and cyberjock the las 2/3 years.. And learn a lot..

It,s for wreedps..

 

[cyberjock]

@fede2222

You need to understand a few things:

1. That "official" document you linked is written by the FreeNAS devs (and had input from me). I made it very clear I was very against the document because people would then say "well, must be safe, very stable, etc" like you did. That is not entirely true. If you follow every recommendation we have in the forums, it *can* be stable. It may still not be, which leads me into #2.
2. Read the forum rules.. in particular this section:

Topics that are likely to go unanswered: Some topics tend to be ignored either because they are already explained thoroughly in forum stickies and the documentation or they regularly turn into arguments about technology. These topics include:

• Virtualization of FreeNAS on Type-1 Hypervisors such as ESXi, XenServer, and Hyper-V. In short, if you can't do this without help, you shouldn't be doing it.

The reason is that it often isn't stable, and we don't have the manpower, resources, or experience to help people that do choose to virtualize. Dozen of people have had error messages and other problems (some of them fatal and involved losing the zpool permanently) that do NOT exist in non-virtualized environments. That brings me to #3.

3. The de-facto answer by most everyone with experience on this forum is to NOT virtualize. If you aren't enough of an expert to do virtualization and handle all of FreeNAS on their own, they're probably better off without virtualizing. This should be obvious from the whole comment about "don't do this in production, only as an experiment, etc.".

4. Virtually 100% of users that use FreeNAS want a solid, stable, reliable storage that performs very well and uses ZFS to ensure data isn't being corrupt or lost. So the warning about "don't do this in production, only for experiments" applies to roughly 100% of the users that come here. Nobody comes here and says "help me build the most unreliable server I possible can because I don't make backups and want to lose my data!"

Virtualization has been a very heated topic. When we rewrote the forum rules late last year I wanted a forum rule that said "any threads or posts about virtualizing will be deleted without warning" as ultimately 1 (or more) of 3 things end up in the threads.

1. Hotly debated and contested with a flamewar often resulting.
2. Pissed off end-user that didn't get the solution.
3. Pissed off forum admins because the pissed off end-user goes and complains to iXsystems, which then asks what is going on and why we let the thread escalate to a flamewar.

Also, keep in mind that while you are on 8.x, nobody is using 8.x anymore. So coming here for support on your system is really not going to be beneficial to you... at all. And back in 8.x there were lots of problems that were fixed with the 9.x kernel, making virtualizing more stable. So you're actually at higher risk because you are on such an old version. If you ever have a problem and your post starts with "running 8.3.0" or something else equally old everyone will probably turn away immediately because they don't want to deal with such an old version.

Yes, I did virtualize. I had to use 3 different sets of hardware to get one that was stable for more than a week without problems. I also deliberately stopped using virtualization because I didn't want to be the next guy to have a lost zpool because of virtualizing. The problem are too sudden, too permanent, and far too often for my own taste. And I do TrueNAS/FreeNAS support for iXsystems. So take that for what its worth. ;)

 

[jgreco]

It's not against you jgrego... I read a lot of you and cyberjock the las 2/3 years.. And learn a lot..

It,s for wreedps..

As a practical suggestion in forum survival, please consider that many people here are of varying skill levels, varying levels of fluency in English, and most of the people who bother to reply honestly are just trying to help, or to understand your issue better, or learn something themselves. You will keep your sanity intact longer if you attempt to read posts without trying to infer negativity.

For example I write stuff all the time that can be taken several ways, often it is terse and short because I'm typing it on a smartphone with tapatalk (like now).

My take on it is that the blog post is no more a blessing of virtualization than the stickies are. I work on both VM and bare metal FreeNAS and have probably been involved with more virt threads here than anyone. Best practice is bare metal unless you're sufficiently goo-rue to go virty. Part of that is probably tolerating people asking why you don't just bare metal it.

 

[jgreco]

Figures CJ was composing a frickin book while I was tapping a reply into tapatalk.

 

[fede2222]

OK.. I don't defend Virtualization
After 3 years using it, it's work. I don't pretend that all others think like me.
... And I share all that you say, both (jgreco & cyberjock). I use it with all the caveats, warnings and trouble s with inexperience until I do things right

The best suggestion was
"Part of that is probably tolerating people asking why you don't just bare metal it."

No bad feelings from here.
Cheers