Will Virtualization of FreeNAS EVER be a good idea?

Status
Not open for further replies.

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Can't put in a bug fix if you can't reproduce the problem, can you? That's always been our problem. When the devs were more active in the forum we look at it, and we couldn't figure it out. Eventually it was a "declare defeat and don't do it with data you don't have backed up". You'll also find that many of the old threads from 2012 and 2013 (before we walked away from ESXi as a forum) just don't exist in here anymore, many won't open even if you have links, and the search engine won't find them even with proper queries. I've got tickets in the bug tracker to fix these problems but nobody has a clue why this is the case, but it all happened when we migrated from vBulletin to XenForo. So blame the forum migration of 2013.

But if someone wants to use freenas with a supported version of esxi then we already know they haven't gone cheap.

Sorry, but you haven't been here long enough to make that statement; and it is very much incorrect. A small number of people don't go cheap when they go ESXi. I didn't go cheap when I virtualized FreeNAS. But by large and far, the vast majority of users that go with ESXi are doing it because they are cheap and trying to do it for cheap. And nobody does ESXi because it's more expensive. That doesn't make sense. I've been here for almost 3 years, and I can say this with experience. I was there. I was trying to help people recover their pools over Teamviewer, quite often for free and on my own time. I wanted to know what was going wrong. I wanted to know because I wanted assurance that the problem wasn't going to be on my server next. (Self-preservation at its finest!) Every time someone loses a pool I always want to know the whys and how it could have been avoided. Most of my advice regarding scrubs, SMART, ECC RAM, things you should do and things you shouldn't, etc are because of learned experience from those that have lost their data. I'd rather people learn from those that lost their data than become a statistic themselves. Only a small number of users on this forum do backups. Unfortunate, but also true.

And that statement I said above about "don't do it with data you don't have backed up".... guess what those cheap people that went with ESXi did? They didn't do a backup because it was too expensive (remember, they are doing it "on the cheap"). Doesn't make what they did the right thing to do, but they still do it, and when we've got people every 2nd or 3rd day starting another thread that begins with "so my pool won't mount and I can't find my data" that doesn't go well for the FreeNAS project, and definitely doesn't go well for someone that's trying to grow a user base that is measured in 4-digits. It makes it look like our product is totally untrustworthy. Doesn't matter how much they do right (or wrong), get 3-4 of those threads on the first page of every section of the forum and potential users decide they don't want to trust our product. After a while you have to cut off your nose despite your face.

And to be honest, compared to how things were 2 years ago, and despite whoever might want to argue otherwise, there's one undeniable truth.


Unfortunately twice this week I've seen threads that have missing data and they virtualized. How many have I seen from bare metal installs that could be related to anything that isn't clearly user error? Zero. There was one user that has a single disk for his pool and can't mount it because he's missing other disks. But that's not a software problem.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I'm curious if FN would be more stable as a VM if people set their raid controller to pass through directly the FN VM. Then FN could have bare metal access to the disk and receive smart info instead of the zpool living in a vmdk.

You can configure adapters on the vmware host to provide directpath i/o to VMs. Seems like this could be a workable solution if you really had to virtualize.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I'm curious if FN would be more stable as a VM if people set their raid controller to pass through directly the FN VM. Then FN could have bare metal access to the disk and receive smart info instead of the zpool living in a vmdk.

You can configure adapters on the vmware host to provide directpath i/o to VMs. Seems like this could be a workable solution if you really had to virtualize.

Misperception. FreeNAS is perfectly stable. The problem is that if you're using a virtualization system where the I/O to a VM stalls due to a failure, because it's going thru the hypervisor's I/O subsystem, that's bad. Problem is that it seems to stall ALL I/O, not just the affected disk.. Not a FreeNAS issue in any way, shape, or form. Naughty hypervisor!

As for passing through a RAID controller? Bad idea. We don't advise the use of RAID controllers because so often you don't get bare metal access and SMART data. But if you look at my virtualization sticky, you'll note that using VT-d to pass in a HBA is part of the recommended strategy; this still assumes a LOT of clue and some other preconditions. The fact of the matter is that virtualization always ends up with a lot more moving parts and moving parts are things that are prone to breakage.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
That is one of the solutions in jgreco's "Absolutely must virtualize" thread. I have yet to see any downside with 90 days or so of pounding and testing. There are great recovery options with that model as well. However, people have run for a long time and then lost data. All the old missing losses make it very hard to assess actual risk. So my bias tends a lot towards jgreco's opinion. There is a significant PEBCAK issue.

For me the question is "When do you declare it safe?" Those with good experiences, can't really speak up. The knowledge base never moves forward because people are fearful. Its' a vicious circle and one that kills advancement of the issue, imho.

Seriously. Onboard cyberjock. Someone donate him a fast e5 with gobs of RAM and esxi and flawless VT-d. Truth is we don't really have the man power on the forum to troubleshoot esxi and freenas at the same time. But I'd sure like to see progress.

Edit: Heh... beat me to enter.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
For me the question is "When do you declare it safe?" Those with good experiences, can't really speak up. The knowledge base never moves forward because people are fearful. Its' a vicious circle and one that kills advancement of the issue, imho.

Seriously. Onboard cyberjock. Someone donate him a fast e5 with gobs of RAM and esxi and flawless VT-d. Truth is we don't really have the man power on the forum to troubleshoot esxi and freenas at the same time. But I'd sure like to see progress.

So you mean like my toy ESXi box, which has been happily running FreeNAS in a VM for well over a year...?

CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.00-MHz K8-class CPU)
real memory = 34359738368 (32768 MB)
avail memory = 33171898368 (31635 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)

I picked this up when I got tired of the limits of 2010-era E3 Sandy gear. The E3 stuff is a LOT cheaper but I wanted to be able to throw RAM at some VM's sometimes. So with 12 cores and 128GB, I can easily assign the VM whatever I want.

It's a gorgeous X9DR7-TF+ in a nice 4U SC846BE26 chassis with about 40 VM's running at the moment. The M1015 is VT-d passthru'd to the FreeNAS VM and is running an array of 12 x 4TB Seagate desktop drives.

I really do eat my own dogfood guys. I generally don't recommend things I haven't actually tried...
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
By raid controller I meant in IT mode or a HBA, sorry for not being more specific. It seems like direct I/O passthrough would solve a lot of the "where is my zpool in the vmdk" issues.

Your host specs are very similar to mine. vmhosts.PNG
 
Last edited:

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
@jgreco Heh. Exactly like that. You don't often mention that box, sir. ;)

LOL. I was just tryin' to keep my head down and not get excommunicated by the powers that be. But admittedly I struggle with the party line.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
By raid controller I meant in IT mode or a HBA, sorry for not being more specific. It seems like direct I/O passthrough would solve a lot of the "where is my zpool in the vmdk" issues.

Ok, but you need to understand something:

RAID controller ≠ HBA

just like

Semi-truck ≠ Car

You can't say "Semi-truck" when you meant "car". ;)

Precision is key. :P
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Ok, but you need to understand something:

RAID controller ≠ HBA

just like

Semi-truck ≠ Car

You can't say "Semi-truck" when you meant "car". ;)

Precision is key. :p
Thanks Jock,

Yes I understand, very well, that they are different.

I really appreciate you semi-truck/car analogy. It's fun having someone talk down to you on a forum.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
u wot m8?!! Your join date is almost as pathetic as mine. No possible way you could know anything. ;)

Besides. Emoji for the win.

Keep piping up. It's nice to have folks around running decent gear and bigger environments. I have yet to misinterpret the intent of one of your posts.
 
Status
Not open for further replies.
Top