ESXI + FreeNAS 8

[Miniwehats]

I have also posted this on Vmware forums:

Hello I am fairly new to setting up Esxi but I have a little test server running in my home and have settup a few VMs on it and played around a bit with it. I am building a new Esxi server in the next few weeks and am having some issues figuring out how exactly I am going to do some things. Please forgive my ignorance I have had a difficult time finding answeres to my questions. And I am also having an issue articulating this into a understandable question.

What I want to do : Building a new ESXI Server - AMD quad 3.6 FX 16gb RAM booting ESXI off USB.
-FreeNAS VM
-Server 2008 VM
-Random other VMs

What I want to do is build all this in one single box but my issues is how FreeNas will load and where since all my VMs will be build from ISCSI volumes from FreeNAS. FreeNAS runs off a USB drive as well.

Loading FreeNAS as a VM off a usb flash drive, where will the data store for the VM go? Can I build it onto the flash drive? Will I need to just have a single local disk on the ESXI server to have this one single data store? All my orther VMs will run from the FreeNAS volumes i create for them. So if my FreeNAS volume goes down ALL my VMs will go down. Granted I can backup the config and reload FreeNAS and it should all come back up. Should I create a Hardware Mirror of 2 drives locally on the ESXI server to just host the data store of FreeNAS so incase one goes down it can use the mirror?

After reviewing this it seems best practice would be to build my FreeNAS box seperate from the ESXI server but I would really like to build it all in one box to save money.

So would you recommend doing a Mirror on the esxi just for the FreeNAS data store? Or building a second box?

Sorry for the confusion here I am just trying to get this all figured out. And thank you for any help you can provide!

 

[louisk]

I'm confused about how you will boot the FreeNAS VM from the storage off the FreeNAS VM. That sounds like a catch 22.

I would suggest you make FreeNAS a physical box, and do NFS to ESXi.

The other alternative would be to have all the disk in the ESXi machine, and have FreeNAS share some of that to other machines.

 

[Miniwehats]

Sorry for the confusion, all the disks are in the same box. I just decided to settup a Hardware Mirror Raid of 2x SSDs to hold the esxi data store and the FreeNAS data store. All the other drives for the raid will be assigned to the FreeNAS VM. This should work out fine right?

 

[survive]

Hi Miniwehats,

I'm with louisk on this.....

Build 2 boxes, one for ESXi , the other for FreeNAS. If that's not an option build one box running ESXi (you'll need a proper RAID card) and use FreeNAS in a VM to share out the storage.

-Will

 

[Miniwehats]

I would like to build a seperate box but the cost just goes way up. I will do a hardware raid on the box with two SSD's in a mirror to hold the Data store for the esxi and the FreeNAS. All the other disks will be assigned to the FreeNAS vm. I have the raid controller built into the motherboard and also a PCI sata 3 card.

 

[hoppel118]

Hallo,

what about this? I want to do something like this. Did you get it to work?

Here is my project in german: http://forums.freenas.org/showthread.php?6631-Projekt-Home-Server-mit-4x3TB-(Raid5-oder-Raid-Z1)

with kind regards

Hoppel

 

[Bever]

Hi Miniwehats,

I'm with louisk on this.....

Build 2 boxes, one for ESXi , the other for FreeNAS. If that's not an option build one box running ESXi (you'll need a proper RAID card) and use FreeNAS in a VM to share out the storage.

-Will

Actually, you don't need a proper RAID card for ESXi. Well, not really anyway. I went down the same route. I virtualized FreeNAS on ESXi5. I'm using a €100 Asus mobo that supports RAID, but it is what is called fake RAID. A software version onboard, not supported by ESX.

So instead of buying an expensive supported SATA RAID card, I searched the internet for a way to use my local disks as RDM's in ESXi. You can then just use those disks within the FreeNAS virtual, and let FreeNAS build a RAIDZ volume with them. Until now I had no issues with that. Of course, the actual FreeNAS OS is installed on a different local disk (the disk on which ESXi is installed).

 

[Letni]

I believe this is possible.. Here is the idea

1. ESXi would boot the physical hardware from a USB stick
- In order to have a "holder" ESX datastore to hold the configuration for the FreeNAS VM, two options are available.
A. Instead of booting from USB, use a very small IDE or SATA drive (even IDE Flash - Cardflash) that will boot ESX AND have space to hold a basic VMFS 3.x datastore
B. Have a secondary USB drive that could be used as the datastore if booting ESX from a first USB flash drive

2. A VM would be configured for FreeNAS that would have the following configuration (VM definition would have to sit on the small datastore on the first USB stick)
- Second Physical USB stick with FreeNAS loaded - configured to pass through to the VM - defined as the boot device in the bios of the VM
- Physical disks all configured as RDM devices to the VM (this is where your ZPOOL would be created)
- This VM would want at minimum 8 GB RAM to fullfill ZFS best practices, so your ESX box probably wants 24-32 GB RAM (gives enough to run other VMs)
- iSCSI target would be defined on FreeNAS to talk to defined ZFS Volumes (created in your ZFS pool sitting on physical disks)

3. ESX would be pointed back to the iSCSI target (internal networking to the box) for actual data stores for OTHER VMs and be formatted as VMFS

Unsure:
1. What type of performance to expect for NAS to FreeNAS to your ZFS datasets
2. What type of performance to expect for iSCSI back from a VM (accessing physical disks) to the physical ESXi
3. Can a ESX datastore reside on a USB flash disk also booting it..

 

[ZFSuser]

working example

Hi,
I have a machine booting ESXi 5, then running Freenas 8.04, then ESXi running guests on iscsi shares from the Freenas vm.
ESXi boots from USB. ESXi 5 boots Freenas from an SSD. The SSD is required for the rdm disk creation. You cannot create rdm devices on USB devices.
FreeNAS runs ZFS on 3 x 2TB (rdm) drives. The spare space on the SSD is used by FreeNAS as a ZFS cache.

This is has been working fine, for 2 months. iSCSI, SMB, NFS all work. Various write speeds 30-100MB/s (using dd random).
*I do seem to get a hang occassionally...which started further investigations and testing. I think it is a ZFS ram usage issue.
*This is a home environment for study. I agree with the other posts, if you want pure performance your NAS should be dedicated h/ware.

Through testing (on my second box with the exact same mobo, ram, cpu) I have found a problem. And would interested in your test results.
Please refer too thread below, which has hardware details and specifics on problem.
FreeNAS-8.0.4-RELEASE-p3-x64 (11703) virtual machine on ESXI 5 with rdm disk problem

 

[jgreco]

I believe this is possible.. Here is the idea

1. ESXi would boot the physical hardware from a USB stick
- In order to have a "holder" ESX datastore to hold the configuration for the FreeNAS VM, two options are available.
A. Instead of booting from USB, use a very small IDE or SATA drive (even IDE Flash - Cardflash) that will boot ESX AND have space to hold a basic VMFS 3.x datastore
B. Have a secondary USB drive that could be used as the datastore if booting ESX from a first USB flash drive

ESXi cannot use a USB flash device for datastore, to the best of my knowledge.

2. A VM would be configured for FreeNAS that would have the following configuration (VM definition would have to sit on the small datastore on the first USB stick)
- Second Physical USB stick with FreeNAS loaded - configured to pass through to the VM - defined as the boot device in the bios of the VM
- Physical disks all configured as RDM devices to the VM (this is where your ZPOOL would be created)
- This VM would want at minimum 8 GB RAM to fullfill ZFS best practices, so your ESX box probably wants 24-32 GB RAM (gives enough to run other VMs)
- iSCSI target would be defined on FreeNAS to talk to defined ZFS Volumes (created in your ZFS pool sitting on physical disks)

3. ESX would be pointed back to the iSCSI target (internal networking to the box) for actual data stores for OTHER VMs and be formatted as VMFS

You'll have a bootstrap problem: the ESXi box wants to mount its iSCSI (or NFS) devices during bootup, before it starts virtual machines. Your FreeNAS instance won't be running, and so when the ESXi box finally gets around to giving up and booting anyways, it'll not start any of your other VM's because the datastores are inaccessible. At least, that's how it works on 4.1.

 

[jclambert1]

What?... I have ESXi running using flash. I have a raid 1 store for certain critical data (such as freenas config, ISO stores). I then map all local drives as separate volumes without raid. I map each volume as a separate drive for the FreeNAS VM. Then Freenas does the rest - i.e. ZFS ;-)

 

[jgreco]

What do you mean, what? I'm pretty sure what I said was clear. If your ESXi cannot mount its SAN/NAS storage during startup, i.e. because the FreeNAS VM is not actually running yet, then you may encounter problems getting those VM's started automatically, because ESXi won't have the datastores available when it wants to do the startup.

 

[jclambert1]

What do you mean, what? I'm pretty sure what I said was clear. If your ESXi cannot mount its SAN/NAS storage during startup, i.e. because the FreeNAS VM is not actually running yet, then you may encounter problems getting those VM's started automatically, because ESXi won't have the datastores available when it wants to do the startup.

My apologies, I had assumptions in my head as I read that comment. Your point is clear. However, it seems to me that most people set the order and time interval (on vSphere at least) of the VMs to be booted.

 

[jgreco]

My apologies, I had assumptions in my head as I read that comment. Your point is clear. However, it seems to me that most people set the order and time interval (on vSphere at least) of the VMs to be booted.

That's irrelevant. The system will get hung up during startup trying to mount its datastores. It will eventually time out and move along, but when it does, the datastores are not mounted, which makes starting any VM's on them just a wee bit tricky.

For iSCSI in particular, pay attention both to what's said and what's not said in here

http://kb.vmware.com/kb/2007108

The missing bits in this KB are that it won't retry the iSCSI mounts later - if they're not there at boot, you actually have to manually go in later and do a "rescan" on the iSCSI initiator.

The problem is similar for NFS, though it seems that it might actually mount the datastores at some later point in some cases that I never quite identified, and never in any consistent manner.

http://kb.vmware.com/kb/1007352

If you cannot get deterministic behaviour out of the storage system, you cannot reliably get your VM's started. Now that might be fine if you don't actually require that your virtual environment be resilient after a restart, but it's a significant hazard to everyone else.

 

[Andre-K]

What I would like to know, if if FreeNAS have drivers needed to get top performance on ESXi ?
then Freenas should support VMXNET 3 network adapter, and VMware vSCSI adapter... does it ?

Wish this kind of info were in VMWare compatibilityDB for FreeNAS: http://partnerweb.vmware.com/comp_g...18,217,168,171,158,196&vcl=true&testConfig=16

BTW: I had no luck finding similar for FreeBSD - nor do I knwo what version FreeBSD FreeNAS is based on

 

[cyberjock]

VMs are great for people learning how FreeNAS works, and the elite of the elite that really know their stuff. Based on your question on if the drivers are installed I'm going to make the leap that you don't know how to check it yourself. In my book, that means you probably shouldn't be virtualizing FreeNAS if your data is important. Virtualizing isn't for everyone despite VMWare trying to pitch it as such.

I could probably do it correctly, but I choose not to because I've seen so many people that come in this forum, talk about their years of experience on VMs, and choose to use them despite people saying you shouldn't. Then weeks/months later they're back in the forum asking what happened and the only answer we can give is "you must have set it up wrong". It's interesting but also disappointing to see self-proclaimed veterans of virtualization get boned over and lose all of their data because they thought they knew better when they didn't. They come in here and talk about their years of experience and how they know and understand, but often they don't. :(

I do know that there are people that HAVE successfully setup FreeNAS in a VM in ESXi so it is capable of working. What kind of performance varies based on your other virtual machines, your configuration, and what you actually use FreeNAS for. I'm sure someone in this forum that has a VM of FreeNAS running is going to probably pop in this thread and argue with me, but I've seen enough people lose their data. As it is, a large number of people lose their data just because they don't understand FreeNAS even with basic hardware. Add in a virtualization layer and there's so many pitfalls you are more likely to do it wrong than right.

 

[Andre-K]

the thing is that I just purchased a T420 server , with PERC 710 adapter, will run HW RAID6 with ESXi.
so virtualization will be one or another, rigt now it seems like if I run the file server on Ubuntu , I'll be fine regarding performance.