FreeNAS in a xen domU

[Hi-Liter]

Hi,
since my approach of running additional services inside jails in FreeNAS didn't really work out, I would like to go the other way around and run FreeNAS inside a VM and have the services I need running on the host or in another VM (preferably on the host).

Since I don't want to use Windows as the host OS (too much resource usage), Debian Linux with Xen would be my first choice, since Linux offers all the services I need and Xen seems to be a top-notch virtualization tool.

My question now is, has anyone ever tried this before? According to Wikipedia, FreeBSD is not fully supported as a guest in Xen.

 

[Hi-Liter]

I found out that FreeBSD 8.2 should run just fine on i386 in HVM and PV mode as domU, and only in HVM (with PV drivers) mode on amd64. I found this tutorial which provides all steps needed to install FreeBSD 8.2 as a xen domU and recompile the kernel to support pv drivers:
http://deadmemes.net/2011/02/26/howto-freebsd-8-2-release-hvm-guest-on-xen-4-debian-squeeze/

Now my question is: Would it even be possible to compile FreeNAS 8.0.2 with KERNCONF=XENHVM?

 

[citrixified]

I have the same issue. :-S Does the FreeNAS disti come with kernel source? It is not at /usr/src if so.

 

[noprobs]

I have a tried FreeNAS 8.3beta1 64bit in Xen and it is working fine; however I believe it is in HVM mode. I would like to have a version which I can load straight into Xen/XCP with full PV drivers and tools - but don't know whether this is possible

At the moment i am just trialling FreeNAS and Nexenta (which has supported Xen image) to decide which to use in production. My original thought was to create a VM (use Xen disk which is in a RAID - I wouldn't want to relay on a single systems disk) and then pass through the SATA/SAS controller to be directly managed by FreeNAS

Comment/advice appreciated

Jon

 

[cyberjock]

The FreeNAS source is not included with the ISO for size reasons. If you want to check out the source for FreeNAS you need to go to the Installation section of the forum. There are 2 threads that explain how to compile it. It is very thorough. My mom could probably compile FreeNAS with the steps in the 2 threads.

 

[noprobs]

I have managed to check out all the source but build has failed as unable to connect to 'cvsup3.freebsd.org' (I have also tried cvsp10) though I can ping the IP. I will keep trying.

a) Am I correct in thinking that after building FreeNAS I will then need to build my own kernel (1. cd /usr/local/freenas/trunk/FreeBSD/src 2. make buildkernel KERNCONF=XENHVM 3. make installkernel KERNCONF=XENHVM ) and replace kernel.gz files in /boot/kernel/?

b) Is there a way to amend the do_build script to build with KERNELCONF=XENHVM in one step?

c) Are there any thoughts/interest in publishing a FreeNAS image with XEN PV drivers already burnt in ?


Thanks for your help

Jon

PS I am starting to wonder whether I would be better of building a standalone FreeNAS server rather than building on top of my my XCP Server. I was aiming for one big server rather than two

 

[cyberjock]

c) Are there any thoughts/interest in publishing a FreeNAS image with XEN PV drivers already burnt in ?
PS I am starting to wonder whether I would be better of building a standalone FreeNAS server rather than building on top of my my XCP Server. I was aiming for one big server rather than two

It's not recommended that you run FreeNAS as a virtual machine except for testing. FreeNAS is designed with the expectation that it has direct access to the hardware. With that in mind, I wouldn't exepct to see a version of FreeNAS published with XEN PV drivers built in.

You would definitely benefit from reconsidering your choice of using FreeNAS in a virtualized environment ;) Quite a few people in this forum have tried using virtual machines in various configurations and some have realized the error of their mistakes when they lost all of their data. One of the features of FreeNAS is the ability to run it on almost any hardware. If your motherboard dies you simply take your hard drives and USB key and install them in another machine. Poof, you're back in business. You can't do anything like that with virtual machines.

 

[jgreco]

You can't do anything like that with virtual machines.

That's not true. That's not even remotely close to true. Our next round of ESXi boxes, in fact, are probably going to be the embodiment of the proof of that. You do need to do some engineering and design though, and that's where the screwups usually happen.

Our main ESXi cluster here has historically consisted of separate SAN, NAS, and host systems. For the most part, we have a fairly consistent VM workload, not particularly heavy, but enough to make ESXi pay off.

Now, the neat thing about ESXi in a SAN environment is that you can do precisely what you suggest: you can take your USB key for ESXi and move it to a different machine, and bam, you're up and running. One host is the same as the next. On top of that, you don't need to use the motherboard SATA ports for anything. So, technically, they're actually available for passthru to a VM... like FreeNAS.

And FreeNAS is notorious for being on 24/7, not needing a lot of resources except when someone's actually doing something, which is kind of a monumental waste-of-watts.

So, the gameplan is to get rid of some of our dedicated NAS hardware. We're upgrading the NAS boxes to be similar to our ESXi cluster hosts, but we're also retaining the local storage. This essentially means we get to run VM's alongside FreeNAS for about the same watts. However, we gain the advantage of easy portability as well. We can take the disks, move them to another box, bring up the FreeNAS instance there instead, and it's all like virtualization magic...

"Can't" might have been a poor choice of words.

 

[cyberjock]

I stand corrected. I should have said "it's difficult without adequate planning during the design phase of your NAS". While I agree that you CAN virtualize FreeNAS and have it run flawlessly(at least, that's what I think you are saying) so many people don't understand how the hardware layer and software layer have to interact to be able to pull of what you are saying that the general consensus from the forum has been that virtualizing isn't recommended. I can probably find 3 threads in the last 60 days of people that claimed to be ESXi wizards(I am definitely not.. never used ESXi personally but I understand the concepts) that have watched their FreeNAS server go up in flames because they didn't design their server adequately for the task.

As an easy thumbrule to follow from others, "if you don't know how to get FreeNAS working in ESXi without outside help you almost certainly don't have the skills/knowledge to do it right".

My VMWare Workstation "test" machine is also my desktop/laptop acronis disk image repository. The odds of my laptop/desktop requiring a restore AND I kill my Windows server containing my virtual machine is pretty remote. Even then, I could reinstall Windows server, VMWare workstation 8, and load the virtual machine in an hour or so. But I haven't needed to do a restore for any of my 3 machines since 2008.

 

[noprobs]

For info my plan was to consolidate an existing Xen (XCP) host and a SAN (Nexentastor) into one server (main driver is electricity costs, second is availability of switch ports and UPS) - Opteron server with lots of memory. I would have a USB drive (or ideally pair) with XCP and FreeNAS but all other storage to Xen served via FreeNAS (probably NFS). I was going to use PCI passthrough for all 12+ drives (LSI SAS and onboard SATA) and an onboard Intel NIC such that FreeNAS has direct access to key hardware. I would then replicate key data to another standalone FreeNAS server each week (this server is not powered on at all times)

Of course all of this was predicated on getting FreeNAS to work well with Xen. I was moving away from Nexenta as the future of its community edition is not clear and it has less community support then FreeNAS

Jon

 

[jgreco]

The question you need to answer is whether or not Xen will be happy to boot with most of its VM's in an inaccessible state, and then properly handle booting them up. (ESXi won't.)

 

[Wabun]

Hi Jon,

I was very interested in your XCP and FreeNAS project, I was going to try to get this going as well for same reasons as you, saving on my AMP usage in the DC.
Would you mind to let me know your progress, I am still waiting for a few hardware parts to go ahead and trial myself.

 

[Andre81]

Hello @all, great project.

My FREENAS works perfectly under Xenserver (in HVM mode) since 2010.
Never an error, always do this dirty job perfectly.

but all developer concentrate his time in VMware hypervisor...

I think in future I'll create my freenas under xenserver in PV mode, recompiling kernel.

A.

 

[noprobs]

Hello @all, great project.

My FREENAS works perfectly under Xenserver (in HVM mode) since 2010.

A.

Curious - how many disks do you have attached are they cirtual disks or passthrough SAS/SATA controller?
I presume none of the Xen Storage Repositories are on FreeNAS

Thx

Jon