vmxnet3.ko for FreeNAS 9.X in ESXi 5.5

Status
Not open for further replies.

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Software defined storage is coming on strong and has to eventually follow the path of software defined hardware and networks. It is a HUGE market that will be addressed by products like TrueNAS at some point. Someone will get it right. I see no reason why that couldn't be an opensource/zfs based project. The rigid hardware solutions are getting their margins crushed. Partially by folks like TrueNAS, but also by those with even more flexible architecture. It is short sighted to think that TrueNAS could not function perfectly without baremetal access. Type 1 hypervisors are a known quantity and getting more efficient with every iteration. The big money is driving this model hard.

Here is an interesting perspective: http://www.virtual-strategy.com/2014/07/15/storage-turf-wars?page=0,0

Frankly, I'd far prefer the TrueNAS pure model that I could run on a strong hypervisor. There is certainly a strong business model, support, and hardware opportunity there.

I realize we aren't there yet. I also support you protecting noobs from themselves and enjoy reading your opinions. I fully recognize my thoughts may be "out there" but I have to push the edge in the lab before I can provide creative and safe solutions. Working through the challenges is how we improve systems and models.

I promise to never cry about data loss, and thank the folks around here for the insight. I keep hoping you'll catch the bug for ESXi, cyberjock, and help get this thing bulletproof to where you'd trust it.

Regards,

P.S. This module works great. Thanks OP.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It is short sighted to think that TrueNAS could not function perfectly without baremetal access. Type 1 hypervisors are a known quantity and getting more efficient with every iteration.

Nice in theory but not so much in practice.

Any virtual SAN product is a different beast than your typical VM. In some cases, they will be installed as part of the hypervisor platform itself. In others, they will require specific hardware and configurations. I can't think of any case where a vSAN product intended to store many TB of data is simply installed as just an ordinary VM without any conditions or caveats.

Pretty much anyone can set up a VM that contains FreeNAS, and a virtual disk taking up 100% of each of several nonredundant datastores. It works fine. The problem is not that this doesn't work, but that it can go unexpectedly sideways when a disk fails. The virtualization emulated SCSI controller may freeze up, either for that one disk, or totally. The emulation fails to report on underlying datastore disk health - and the people trying these configurations typically do not have a vCenter Server that alarms as to the problem, so there's no early warning. People have tried using (officially unsupported, hacky) SATA RDM on hardware that doesn't support VT-d, and then had all hell break loose when mappings came undone when something failed.

So the caveat here is that you can virtualize FreeNAS in such a manner but you put your data at severe risk because you've compromised parts of the underlying protection model in various ways. It'll seem to work great right up until you someday have a problem.

I'll even be happy to take it one step further: if you virtualize FreeNAS by putting your virtual disks on typical ESXi datastores, your pool will pick up the reliability and safety characteristics of the underlying datastores:

  1. If you have four nonredundant 1TB disks each set up as a datastore, your RAIDZ pool will be healthy right up to the point where one fails, at which point it OUGHT to be recoverable, but otherwise-knowledgeable-appearing people have done this and had it blow up on them, buyer-beware.
  2. If you have eight 1TB disks set up as four sets of RAID1 data stores, and you put a vmdk on each resulting RAID1 datastore, and then create a RAIDZ pool out of those four virtual disks, I would expect you to never have any trouble as long as you diligently paid attention to the health of the datastores.

The problem is that that second solution represents a heavier investment in resources than your typical cheapskate home user wants to make. It also contains significantly more moving parts, in terms of spindles, and also some sort of ESXi-friendly RAID monitoring.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
Perhaps you should have said virtualizing storage (NAS, SAN, etc) instead of file server.

Virtualizing file servers is quite common.

Well, considering that virtually no file server out there recommends it be virtualized we are hardly alone
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Perhaps you should have said virtualizing storage (NAS, SAN, etc) instead of file server.

Virtualizing file servers is quite common.

That's a fantastic point to make, come to think of it! While FreeNAS is incredibly competent as a fileserver, one of the things that is clear to me is that having an extremely complex storage appliance doing ten different things is also a real pain to reconstruct in the event that there's an upgrade or significant change necessary.

Running a FreeNAS VM instance on your virtual cloud to serve files has some ups and downs. Assuming your datastores are already suitably reliable to your business requirements, a FreeNAS VM with a single virtual disk will still be able to detect corruption that the hypervisor's storage system may not identify, which shoots you in the foot because then ZFS refuses to cough up the corrupted data. You can disable checksums of course but that seems bad.

The problem is that there isn't a clear delineation between what's a "fileserver" and what is a serious large scale NAS. And we already know that many n00bs invariably feel that they're exempt from reality. And we know that we can say "redundant datastore" and somehow this becomes "SATA RDM" in the home power user's deployment. And you can say things like "unable to recover data" and well we know how it goes.

However, if you're interested in a highly reliable FreeNAS VM for small scale file storage, and you're willing to commit appropriate resources in the form of VM datastores, there are options available.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
mjws00-

Well, considering that virtually no file server out there recommends it be virtualized we are hardly alone. ESXi doesn't even recommend file servers in its training documents.

I then recommend you come out of your hole. VMware makes a virtual storage product themselves(I'm not saying I'd recommend it) and at one time heavily promoted for the SMB market. They are now more into their virtual SAN product which looks much better but it's still a 1.0 product. Anyways virtual NAS/SAN appliances are big business these days for small to mid sized companies. If you want proof then check out HP's StoreVirtual VSA, there are ton's of small players in this market also. They are pretty much all Linux based systems and now ZFS is finally starting to show up on these Linux based appliances and be embraced(aka they are tired of the promise of btrfs).
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Wow.. what a deep argument.... not.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
That's a fantastic point to make, come to think of it! While FreeNAS is incredibly competent as a fileserver, one of the things that is clear to me is that having an extremely complex storage appliance doing ten different things is also a real pain to reconstruct in the event that there's an upgrade or significant change necessary.

Running a FreeNAS VM instance on your virtual cloud to serve files has some ups and downs. Assuming your datastores are already suitably reliable to your business requirements, a FreeNAS VM with a single virtual disk will still be able to detect corruption that the hypervisor's storage system may not identify, which shoots you in the foot because then ZFS refuses to cough up the corrupted data. You can disable checksums of course but that seems bad.

The problem is that there isn't a clear delineation between what's a "fileserver" and what is a serious large scale NAS. And we already know that many n00bs invariably feel that they're exempt from reality. And we know that we can say "redundant datastore" and somehow this becomes "SATA RDM" in the home power user's deployment. And you can say things like "unable to recover data" and well we know how it goes.

However, if you're interested in a highly reliable FreeNAS VM for small scale file storage, and you're willing to commit appropriate resources in the form of VM datastores, there are options available.
Small scale server consolidation is really what interests me. The common use case being dumping antiquated fileservers. I can throw RAID at datastores it's what we've always done. But the thought of dumping hardware raid and gaining the benefits of ZFS is compelling. In addition if the 10Gb ethernet drivers function reliably, we should be able to get fantastic network and shared data performance within the host. Ideally I'd like to have 10Gb SAN like performance to the collection of VM's running iop intensive apps and skip the wire. Plus add replication and all the other ZFS goodness. It is very easy to make a strong business case for the esxi host, and additional hardware necessary when so many functions can be centralized.

Vt-d so far gives me a pretty strong comfort level and allows me to migrate from hardware RAID. Virtualized overhead is minimal. The ability to boot to baremetal with full pool access seems like a great solution. It also allows easy performance comparisons between virtualized and baremetal implementations. Your passed through redundant datastores is an interesting thought in terms of performance though the "magical" loss of data pools bothers me. Things fail for a reason.

In an smb world. A virtualized freenas solution hits many of the targets. ZFS, AD, ISCSI, AFP, replication, and backup services all wrapped up in a decent gui. I just want it to play nice in my already virtualized environment. There is also no hope in hell of rolling out something that can randomly refuse to mount a pool.

I was going to start "Known Good SAFE and FAST esxi thread" hoping that the gurus would pitch in. Throw it in an experimental or performance section (whatever). But didn't really want to incur cyberjocks wrath.

So instead of improving the product and figuring out how to tweak it. We have to all retreat to our own little holes and never speak of the innovation and cool things we pull off.

Thanks for all your posts, jgreco. We've wandered OT, but this is by far the most active esxi thread.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
But the thought of dumping hardware raid and gaining the benefits of ZFS is compelling.

That makes me nervous; the hardware RAID for primary disk VM storage (and for ESXi boot/vm conf files) seems like a real good idea. This veers off into the world of VM design philosophy though, and in my philosophy a VM should be designed such that it has a base system disk (in some ways quite similar to the FreeNAS install image) and then add on additional disks for application use - or, better, NFS mounting of application data. So a lowish performance RAID controller - ironically still the favored M1015 around here, except this time with IR firmware - is still desirable from my point of view.

In addition if the 10Gb ethernet drivers function reliably, we should be able to get fantastic network and shared data performance within the host.

Unnecessary. Try running iperf on two E1000's on two VM's on the same host. It already goes faster than gigE. There's some debate as to which option is better, vmxnet or E1000...

Vt-d so far gives me a pretty strong comfort level

It might give you a less strong comfort level if you run into some of the gotchas I've seen. Things suddenly not working correctly after an upgrade (of either ESXi or FreeNAS!) comes to mind.

Your passed through redundant datastores is an interesting thought in terms of performance though the "magical" loss of data pools bothers me. Things fail for a reason.

Well that's easily enough explained. So you're some guy on a budget who has managed to install ESXi on something that vaguely qualifies as a compatible whitebox. You have several disks, a 1TB, and two 2TB's, attached to the motherboard SATA ports. You think "gee I could get FreeNAS goodness." So you make a VM with three 1TB VMDK's in RAIDZ1 on your unprotected datastores.

Now maybe you know or maybe you don't know, what happens with ESXi when it loses access to a datastore. Typically it freezes VM I/O. So a failed disk that's appearing via the ESXi I/O abstractions because it is an ESXi datastore can cause the FreeNAS VM to freeze. This isn't what you intended or expected. So you also had the ESXi host mounting NFS from the FreeNAS VM in a wonderful loopback fashion, cuz, you know, you're SO clever and look how smart you are ... ZFS-protected VM's. Except that now all your VM's using the NFS datastore also freeze. And in a panic you cannot tell what's going on and reset the ESXi host.

How much of a betting person are you? Because we've seen stuff like this and it's resulted in pool damage. My best estimation is that the freezing of the I/O is somehow very bad and the reset of the hypervisor leaves the pool in an inconsistent state (and we can agree that SHOULDN'T happen, but ... seen it.)

As you say, things fail for a reason. So when you listen to me say "if you use vmdk's on redundant datastores I think you'd be likely to be OK" you better not be thinking "this guy's a chump and I'll show him by using nonredundant datastores and oh by the way ta-da, it works." Because duh of course that does work - in normal operation. It's the planning for failure part that's important, though.

There is also no hope in hell of rolling out something that can randomly refuse to mount a pool.

So your alternative is just not to run anything at all? Because pretty much every platform for services out there will refuse to mount its filesystems or pool if there's corruption. The problem with ZFS is that consistency isn't just a per-disk thing; all the disks have to be consistent with each other. There's room for things to go awry there.

We discourage n00bs from trying virtualization because they invariably seem to wind up looking something like the story above. This makes us sad. We're also hesitant to discuss the ways that it CAN be made to work because the inevitable problem is that if I say "you can put a single FreeNAS vmdk on a datastore that's protected with redundancy" they often hear something more like "yeah just go do whatever ghetto crap you can dream up and it'll work just fine and we even bless your configuration" ... and even have the temerity to claim that we told them to do it.

I was going to start "Known Good SAFE and FAST esxi thread" hoping that the gurus would pitch in.

Not that much interest. Typically the people who really know their ESXi will look at my "absolutely must virtualize" post, go "well DUH," and happily move on with their lives. I kinda hang around when I have a little time because I like dispensing information sufficient to get people to that "aha!" moment.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Thanks for the insight. I certainly appreciate it and I'm sure others do as well. It looks like there is such potential for an elegant solution. But then.... <puke>.

So from a minimalist point of view it sounds like you'd consider, 2 vmdks, on mirrored hardware raid, mirrored in FreeNAS more reliable than simply passing through the controller and building the mirrored pairs. Even though we lose the ability to boot to baremetal and access the pool. I thought vt-d was far more robust than it is given credit for here.

I'm fine with having to run Intel gear, or known good controllers, HCL only etc. But I'd sure like to throw a little extra money at the hardware once and have a flexible (virtualized) environment. Versus adding more boxes with fairly high specs (xeon, ecc, 32GB+) that are still capped at the same 1Gb my old servers or a pos desktop can saturate. 10Gb switches are out of reach for most of my clients. A fast esxi box and a backup is easy. I almost always have to keep a windows box (virtual is fine).

I think I'll set up both vt-d and the redundant datastores passed in for testing. I'd love to know what on earth actually goes wrong. All the anecdotal one off failures drive me nuts when it comes to evaluating risk. "It broke, we can't reproduce it, and we don't know why." I don't mind throwing gear and time at it to test. Unfortunately, it is tough to break in a reasonable time frame. Failure 2 years from now is meaningless.

Thanks to all the trailblazers.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
So from a minimalist point of view it sounds like you'd consider, 2 vmdks, on mirrored hardware raid, mirrored in FreeNAS more reliable than simply passing through the controller and building the mirrored pairs. Even though we lose the ability to boot to baremetal and access the pool. I thought vt-d was far more robust than it is given credit for here.

But that's the thing. You don't need VT-d in order to run FreeNAS on top of vmdk's. That should ALWAYS work just fine as long as there's no fault. The issue there is what happens in a failure. If you have a failure of an underlying nonredundant datastore, you hit one kind of wall in that the hypervisor freaks out and freezes I/O. Bad. So if you have an underlying REDUNDANT datastore, that's still not sufficient because if FreeNAS sees a bitrot error on that data, it has no way to request the redundant copy being made by the hardware. So you need to provide redundancy at the FreeNAS layer by having two vmdk's (mirrored) or multiple vmdk's (RAIDZ). And those need to be on hardware-RAID datastores.

I am reasonably comfortable stating that this will be at least as reliable as the underlying datastores (more, because you get the bitrot protection etc). The PROBLEM with this is simply that some virtualization/hardware RAID n00b will try this and will still manage to hose up a datastore when a disk fails and the correct hardware-RAID recovery process isn't followed. But at that point, this becomes an issue of my carefully qualified statement "as reliable as the underlying datastores" - if you compromise their reliability, then you risk the FreeNAS.

And now we can replay the posts above where I talk about how n00bs will take my carefully considered and cautiously constructed environment and just make whatever changes they see fit to make to it and assume that their configuration is also blessed.

Not.

So if you need large NAS/SAN storage, use VT-d and just "doitrite." But if you are a virtualization pro and you are using reliable datastores (DAS RAID, or SAN RAID, etc) and you understand everything I've said in this thread, and the virtualization stickies, you now understand the primary risks sufficiently that you may make informed decisions about non-VT-d configurations.
 

zang3tsu

Dabbler
Joined
Feb 17, 2013
Messages
14
Is the 9.2.1.6 vmxnet3.ko module compatible with 9.2.1.7 or should it be recompiled?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I wouldn't guess it to be a problem, but it is always safest to recompile.
 

bkvamme

Dabbler
Joined
Jun 26, 2014
Messages
16
Just though I'd shoot in my experience:

9.2.1.6 vmxnet3.ko module works for 9.2.1.7, and with ESXi 5.0 U3.
 

Joop Worst

Dabbler
Joined
Oct 14, 2014
Messages
10
I want to copy vmxnet3.ko to /boot/modules but it says read-only filesystem. Is there any way around this?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yes... see the FAQ and/or documentation for FreeNAS as it explains how to get around it. ;)
 

Joop Worst

Dabbler
Joined
Oct 14, 2014
Messages
10
Yes... see the FAQ and/or documentation for FreeNAS as it explains how to get around it. ;)
I' am sorry but i have been searching all over the place and can't seem to find it. I've been looking under shell in the documentation, but no luck. Could you maybe point me in the right direction?

EDIT: nevermind! found it! mount -uw / does the trick.

The Vmxnet3.ko for Freenas 9.2.1.6 works fine with 9.2.1.8
 
Last edited:

Vijay

Dabbler
Joined
Mar 5, 2014
Messages
18
Stefan,

Thank you, for the upload. I found out from another post, that vmware supplied vmxnet3.ko file works with 9.3 as well. So I used that, as a test and seems to work. Your upload should be superior as you must have custom compiled from source code just for Freenas 9.3. Which vmxnet3.ko module should I use for better stability? Anyone with opinion?
 
Status
Not open for further replies.
Top