vmxnet3.ko for FreeNAS 9.X in ESXi 5.5

Status
Not open for further replies.

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
As far has I know the drivers that come with ESXi should work fine with 9.2.1.6. There was just a version or 2 in the 9.2.x series that broke the pre-built drivers.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It won't. The bottom line is that *if* you include a vmxnet driver with FreeNAS that causes a whole set of problems. And since the vmxnet3 driver isn't always compatible across ESXi versions, multiple people use ESXi 4.x, 5.0, 5.1, and 5.5, and the fact that the forums do not endorse or recommend using FreeNAS with ESXi, the answer is pretty obvious.
 

spidernas

Dabbler
Joined
Mar 26, 2014
Messages
10
OT: I still don't get this overblown aversion to virtualizing FreeNAS. People seem to want it - why ignore this user base?

Anyhow, thanks again David E!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It's not about "ignoring" the user base. The problem is that the FreeNAS project has no clue what is wrong and VMWare isn't about to give us access to their code to find out.

For reasons that are outside of our control it just doesn't end well for many users. If you want to do it, great. Good luck to you. For most of us the serious (and often unknown but definitely misunderstood) risk is just not worth the benefit.
 

spidernas

Dabbler
Joined
Mar 26, 2014
Messages
10
The problem is that the FreeNAS project has no clue what is wrong and VMWare isn't about to give us access to their code to find out.

Maybe it's just me then... because I did not realize anything was "wrong". It works great for me on sizable workloads, but maybe I just got lucky on my first try. Or maybe, I have realistic expectations and knew I'd have to pass-through an HBA, give the guest lots of RAM, and even the guest dedicated CPU reservations.

I've read a lot of your posts in regards to your viewpoint on avoiding virtualization of FreeNAS, but I just never saw or experienced any headaches. I can definitely see newbies to both FreeNAS and VMWare making a mess though, so maybe the advice is more for them?

If I'm grossly misunderstanding the issues and I'm sitting on a ticking time-bomb, I'd like to learn why. Just so far, it has sounded more like FUD.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
spidernas,

The problem with virtualization is as I have said many times on the forum and is also in the "please don't virtualize FreeNAS" thread. It works great. It may work great for a day, for a week, or even a year. But, one day your system gets pissed at you and the server Gods smite you and its all over. It's one of those things that works and then suddenly doesn't work. We've had people that did nothing more than reboot their host machine properly and... *poof*.. it was over. Never saw their data again.

The ticking time-bomb analogy is pretty accurate and because we can't fix it it's much easier (and safer for your data) to make it clear it's quite dangerous and you'll get no tears from us than to try to actually engage VMWare to the required extent to fix the problem.

Edit: To be completely honest this is something that has only really become well known in the last year to 18 months, and only because the number of users having unexplained but catastrophic problems became so high we couldn't ignore it. At some point those of us that are mods decide we have to take action because seeing people dump all their data on a virtualized FreeNAS box only to see it fail days later isn't something we enjoy.

This isn't that much farther from the whole ECC RAM debate. If you were around before the ECC RAM is bad thread you probably never heard us mention ECC RAM at all. We recommended server-grade parts, but alot of people didn't go with it and we really didn't hound them. But as we're supposed to be stewards of how to protect your data it became a situation were we had to be "responsible adults" and crack the whip on ECC RAM.
 
Last edited:

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
spidernas,

The problem with virtualization is as I have said many times on the forum and is also in the "please don't virtualize FreeNAS" thread. It works great. It may work great for a day, for a week, or even a year. But, one day your system gets pissed at you and the server Gods smite you and its all over. It's one of those things that works and then suddenly doesn't work. We've had people that did nothing more than reboot their host machine properly and... *poof*.. it was over. Never saw their data again.

And you know what all boxes virtual or non-virtualized can one day get "pissed at you". That's not exactly anything new in the world of IT.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
And you know what all boxes virtual or non-virtualized can one day get "pissed at you". That's not exactly anything new in the world of IT.

You're back huh?

Yes, but when it's significantly more virtualized than non-virtualized you can't ignore the extra risk because 'any box can get pissed at you'.

You wouldn't punch a lion in the face would you? But one might eat you next time you are on a Safari adventure in Africa.

It's all about risk mitigation, something you personally don't seem to understand.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
You're back huh?

Yes, but when it's significantly more virtualized than non-virtualized you can't ignore the extra risk because 'any box can get pissed at you'.

You wouldn't punch a lion in the face would you? But one might eat you next time you are on a Safari adventure in Africa.

It's all about risk mitigation, something you personally don't seem to understand.

Nope I'm not back, I'm just poking the caged tiger....

Actually I deal with risk mitigation all the time, which is why when I poke caged tigers I use a long stick.

Anyways I don't know where you get your info, but their is like 0 risk due to virtualization of FreeNAS when you do it right. And just like anything else in IT when you get fancy things get messy. Think of it has the difference between using the restore disk provided with a stock dell model vs using the retail/OEM windows disk and a custom built home brew system. And yes some folks should just buy the dell. I get that, but you won't ever learn how to make a good home brew system if you always buy from the Dell web page.

Enough now, going back to enjoying my supported virtual SAN.
 

spidernas

Dabbler
Joined
Mar 26, 2014
Messages
10
Please, let's not get this thread locked. It's the only one that has vmxnet3.ko updates. :)
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Please, let's not get this thread locked. It's the only one that has vmxnet3.ko updates. :)

Well I'm guilty of not deploying 9.2.1.6 so I can't answer for it, but the stock drivers provided with ESXi 5.x have worked fine all except for 1 or two of the releases in the 9.2.x series where you had to compile the drivers in order to use them(I'm thinking it was maybe 9.2.1 or something like that). I deployed at some level every build in the 9.2.x series up to 9.2.1.5 and once ix backed out the one kernel patch that changed the APIs in the kernel the stock drivers have worked great. Anyways let's not lock the thread.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
OT: I still don't get this overblown aversion to virtualizing FreeNAS. People seem to want it - why ignore this user base?

It's not an aversion. Feel free to go and do it. I posted a guide to it somewhere. It is even very well thought out in ways that n00bs tend not to understand.

The problem with virtualizing FreeNAS is that, unlike a typical virtual machine, most people are trying to store huge amounts of data. A virtual environment is already an incredibly complex abstraction, and a ZFS system is also an incredibly complex abstraction. If you are not careful and thoughtful with your strategy, you will end up in an M. C. Escher painting that you may become hopelessly trapped in.

See, for example, I give the very good advice to use actual disks by passing them through to the guest. This gives numerous benefits, including - perhaps most importantly - recoverability during a failure scenario, because you should be able to jettison the hypervisor and boot FreeNAS on the bare metal and recover your system.

The underlying problem is that most people in the N00bs forum experimenting with virtualization will try things that make sense to them because it represents the set of options they have readily available. But making four vmdk files on four drives and using RAIDZ on top of that is going to have inherent risk when one of the disks fails and recovery efforts go sideways. And suddenly it turns out that their only copy of their most valuable data was on those disks because, y'know, FreeNAS protects yer data... right? right???

So here's the deal. If you can go and read my "how to virtualize" post and you understand everything in it, then I will declare you competent to virtualize FreeNAS, and will even provide a money-back guarantee... should virtualization fail you, I will return every last penny you paid for my virtualization guide. No refunds for lost bits though.

But until you've sat around here for a few years, watching the failures and tears of users who thought they knew what they were doing and instead lost their data, please don't talk to me about "ignoring this user base." I got frustrated and wrote the guidance both for n00bs not to virtualize, and also the guidance that gives experts a well-thought-out, workable recipe. No part of the user base has been ignored. There is no magic wand that can make the n00bs suddenly make better choices and understand all the complexity sufficiently to allow them to be successful.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Just had to throw in a thanks to you, jgreco. Your guide is amazing. Cyberjock as well. Thank you.

I'd love to see more known good formulas, and optimization shown for running virtualized. Good hardware and vt-d really seems to be flawless. Throw in a usb stick and boot to baremetal instantly if you want. I've been doing my best to break it and I can't. I've pulled drives, cut power, loaded it to the nines.... I keep wondering where the gotchas are.

We really do appreciate the brain trust around here helping those of us that wander from the straight and narrow occasionally.

Mike
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'd love to see more known good formulas, and optimization shown for running virtualized. Good hardware and vt-d really seems to be flawless. Throw in a usb stick and boot to baremetal instantly if you want. I've been doing my best to break it and I can't. I've pulled drives, cut power, loaded it to the nines.... I keep wondering where the gotchas are.

I haven't found any real gotchas except that there's a performance hit, and more danger in performing upgrades. I believe there was some badness upgrading 8->9 due to some MSI issue in FreeBSD 9, but that was a long time ago. In theory if there was a specific issue that created an incompatibility that would lead to corruption, you could lose your pool. I think that's the big thing that gives me nightmares, but it's mostly because I'm paranoid.

If you are careful about backing up your data then fate is likely to go knocking on someone else's door and you may well never need your backups/replications/etc.

If you do not back up your data, you should know that fate is going to wait until the absolute worst time and is then going to teach you about how fast a pool can go byebye.

I'm not really sure that there are many other potential formulas, because when you really look at what I've done, it's "take a quality hardware FreeNAS system and then stick a hypervisor layer in there." Many of the disasters people have run into are simply due to not planning for the INEVITABLE FAILURE OF DRIVES! And how rough that can be to cope with in the virtualized environment.

We really do appreciate the brain trust around here helping those of us that wander from the straight and narrow occasionally.

We like making y'all our guinea pigs. So there. :smile:

Seriously, though, I don't like to suggest things unless I think they'll work well. Wandering from the straight and narrow's great as long as it isn't important data, though!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I did try a 9 to 10 upgrade with a temporary pool and it went badly. I had vt-d and all that and yet the pool somehow munched on itself. Not exactly sure of the cause but it wasn't the same problems we've seen in the past with virtualizing FreeNAS. Pool is detected and mounts but within 30 seconds of mounting the pool the box panics. It's like it can't add the next transaction group without dying. No real clue why and as I no longer virtualize because of the risks I'm not even going to attempt to troubleshoot the problem further. I've said all I can say.. it's bad and not recommended.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Thanks for the info cyberjock. It is hard to find the failures on this forum that weren't user induced. I completely understand your viewpoint.

It seems very odd to me in a world where server consolidation and virtualization is proven and very robust, that we can't make a nas appliance work. I'm paranoid as hell with client data as you are. I've just never seen challenges dismissed out of hand so vehemently. I'd rather see the gurus such as yourself embracing the potential. I can't see the downside to helping those wishing to take a more complicated approach that has significant upsides. It seems those who are willing to test, troubleshoot, and optimize get shot down.

You do great work here. I have learned a ton from your posts, thank you.

Mike
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
mjws00-

Well, considering that virtually no file server out there recommends it be virtualized we are hardly alone. ESXi doesn't even recommend file servers in its training documents.

People keep making the argument about server consolidation but file servers are one of those that has never been recommended to be virtualized, so we're basically "the norm".

In fact, TrueNAS doesn't do jails at all. Commercial entities that do file servers do it in bare metal and never use the OS for anything else. The jails function (and the virtualbox jail) is strictly to make the product much more powerful for your common home user.
 
Last edited:
Status
Not open for further replies.
Top