Question about ESXI + FreeNAS all in ones

[athos56]

So I'm reading up on running FreeNAS as a VM on ESXI and I get the idea and I'm currently learning and doing tons of reading, so please don't bite my head off if I missed the answer to this somewhere or my terminology is iffy. Thanks in advance. So when I started to research my build I learned that I should put all my drives together in a large pool, say 8x1tb drives in Raidz2 or Raidz3. Goal: Have storage for my virtual machines and NAS storage available to any family member on the network (media: Pictures/Movies/Music) Questions: 1: Do I make 2 pools? Share one to ESXI and one as NAS storage for the network? So that would be something like 3x1tb drives in one pool (ESXI) and 5x1tb in the other? 2: Or, do you share the whole pool to ESXI and then attach a chunk of the datastore to a VM that shares out files? 3: Or is the NFS shared with ESXI already available to the whole network already?
Thanks for the help, especially if this is really obvious.

 

[melloa]

If this is your first go around FreeNAS and ESXi you might end up with a problem is your hands. Setup ESXI is simple, setup FreeNAS is also simple, combine them is easy. You will need to understand that you will be running FreeNAS as a VM and your ESXi should be passing all disks to it, so you can let FreeNAS manage them.

That setup has pros and cons on MHO.

1 - Pros: Less machines sucking power; concentrates all in one, so you can have one "powerful" server doing all; etc.
2 - Cons: Single point of failure

My philosophy, and several here won't agree, is to run FreeNAS as a NAS and my ESXi (Xen, VirtualBox, KVM) on another server with all my applications (Plex, OC, Nagios, Guacamole, etc). That way I can have a decent server for FreeNAS using a smaller CPU (i3)/just enough memory (32GB) and a better server for my ESXi with a better CPU (xeon)/lots of memory (as much you can get as you'd run 1st out of memory for the VMs).

You will get lots of guidance here on how to do it. Even some of the gurus run their systems that way. I did see this in one of the iX newsletter: https://www.youtube.com/watch?v=kToF6iyg5yc

 

[bigphil]

1) There's no need to create two pools for what you're doing. One pool is fine. Your IOPS potential is not going to be very good if you don't use striped mirrors but maybe it doesn't matter to you. If you run more than a couple of vm's, I'd consider doing this vs a raidz level. You'd only have 4TB usable space, but much better disk performance for your vm's.
2) You could really do it either way (consume all space with a ESXi datastore, or split). You could create a dataset or zvol and share that with ESXi for its vm datastore. For media you could then create another dataset and smb share from FreeNAS for pics/movies/music. This would be the way I'd do it. If you went the other way and consumed all disk space for and ESXi datastore, you'd have to have another vm running that would then share out space on a virtual disk for your movies, etc. Keep in mind that no matter what, you'll need a separate local disk and datastore that is purely controlled by ESXi to run your FreeNAS vm from.
3) I don't even know what that means. Nothing is shared out already.

 

[Spearfoot]

There is quite a bit of information here on the forum about All-in-Ones (AIOs), so keep reading and take your time and you'll have a better chance of success.

To answer your question: you can do it either way. You can create two pools, with one dedicated to VM storage; or you can just create a single pool and carve off part of it for VMs. But in either case you set up FreeNAS to provide a datastore for VMs using either NFS or iSCSI. NFS is the easiest to set up and is more than adequate for a home lab situation like yours and mine.

I have two AIOs and I created both with a single pool, using both NFS and iSCSI to share storage to ESXi for VMs. I use NFS mostly, but I've configured iSCSI block storage, too, because I like to tinker. I suggest using a single pool in your case, too. It's just easier, and less wasteful of capacity.

The best single 'How To' article I've found on this subject is @Benjamin Bryan's FreeNAS 9.10 on VMware ESXi 6.0 Guide.

The multiple-server approach @melloa recommends is a good way to implement virtualization and more closely reflects enterprise-level setups, but network connectivity between the ESXi servers and the stand-alone FreeNAS server can be a bottleneck. Most home users don't have high speed networks like the big enterprises use. I get blazingly fast I/O for my virtual machines using a separate virtual switch dedicated to storage (as described in Ben's article above).

Good luck!

 

[athos56]

Awesome, guys (generic non-gender specific :) ) Thanks for the help, let me repeat back what you said in my own words and see if I'm following. Create one pool with all my drives, 6tb assuming 8x1tb in Raidz2 attached to my M1015 in IT mode passed through to FreeNAS. In FreeNAS create 2 zvols out of my pool one with 2tb (esxi) and one 4tb (Media). Make sure to have a separate small? datastore for ESXI to run the FreeNAS VM. Read more about zvols...

 

[Spearfoot]

Awesome, guys (generic non-gender specific :) ) Thanks for the help, let me repeat back what you said in my own words and see if I'm following. Create one pool with all my drives, 6tb assuming 8x1tb in Raidz2 attached to my M1015 in IT mode passed through to Freenas. In Freenas create 2 zvols out of my pool one with 2tb (esxi) and one 4tb (Media). Make sure to have a separate small? datastore for ESXI to run the Freenas VM. Read more about zvols...

Uh, no... you won't need zvols unless you use iSCSI. For NFS you just create a dataset, much as you do for any other kind of share (e.g., media).

You will need separate storage for ESXi and the FreeNAS virtual machine: an SSD attached to the motherboard SATA ports, for example. I install ESXi and the FreeNAS VM on a pair of SSDs set up as a RAID1 (mirror) array on a Dell H200 HBA. ESXi boots from this and then loads the FreeNAS VM. After FreeNAS boots up it can be used to provide NFS (and/or iSCSI) storage to ESXi for additional VMs.

But the point is that you can't boot ESXi and the FreeNAS VM itself from storage provided by FreeNAS.

 

[athos56]

Uh, no... you won't need zvols unless you use iSCSI. For NFS you just create a dataset, much as you do for any other kind of share (e.g., media).

You will need separate storage for ESXi and the FreeNAS virtual machine: an SSD attached to the motherboard SATA ports, for example. I install ESXi and the FreeNAS VM on a pair of SSDs set up as a RAID1 (mirror) array on a Dell H200 HBA. ESXi boots from this and then loads the FreeNAS VM. After FreeNAS boots up it can be used to provide NFS (and/or iSCSI) storage to ESXi for additional VMs.

But the point is that you can't boot ESXi and the FreeNAS VM itself from storage provided by FreeNAS.

Right ok, read up on datasets. I'm getting bogged down in terminology. In my brain I imagine you "partition" the pool, but I know that's the wrong terminology. So I'm creating datasets. Your description of the mirrored SSDs connected to motherboard ports was exactly what I was thinking. Thank you for your patience.

 

[MrVining]

My suggestion is to make sure the CPU you buy supports "Direct Path I/O" aka VT-d. If you simply google the CPU, Intel's archives should be near the top. Scroll down to virtualization VT-x and VT-d.

Once you know your CPU supports that, you can buy something like a LSI 9211-8i or the like. Configure that for JBOD. You'll probably need breakout cables to connect directly to SATA drives (assuming your chassis doesn't have a HDD backplane).

Install ESXi on a thumb drive. No one likes that idea at first, but it's fine AND really easy to back-up if you can afford down time.

Find a couple of the smallest SSD's you can find for ESXi to store your FreeNAS VM on. I say two because once you have it all set up, it's pretty easy to back-up from one to the other. This isn't really all that necessary because it's not going to store your files just minor settings for FreeNAS which can be easily reproduced, so if you want to skimp someplace only buying one here is a good choice. Anyway, once one is formatted you create your FreeNAS VM. In the advanced settings you configure the LSI 9211 to be directly mapped to this VM. That way it has control of the PCIe card directly.

Once you have FreeNAS installed and are able to reach it via the web GUI. Power off your system and install all your HDD's. Since the FreeNAS has direct control over the card, any HDD connected to the card is directly controlled by FreeNAS.

Now you can create your raidz2 pool.

I would actually use both iSCSI and NFS. NFS in ESXi just stinks. It's fine for storing your .iso's so you can access that share from both your workstations and ESXi, but for the actual VM storage, I would use iSCSI all the way.

That's my 2 cents. I did it that way for a long time, before I had vSphere license and it made sense for me to move my storage outside of the chassis.

 

[athos56]

Thanks for the tips, I spent last night filling a Rosewill 4500 case full of electronics, going to power on today, flash my M1015 card to IT mode and get ESXI up and running if my system posts. That's when the real learning begins, it might be weeks before I trust it with real data.

 

[Spearfoot]

Thanks for the tips, I spent last night filling a Rosewill 4500 case full of electronics, going to power on today, flash my M1015 card to IT mode and get ESXI up and running if my system posts. That's when the real learning begins, it might be weeks before I trust it with real data.

Be sure to see the "Building, Burn-In, and Testing your FreeNAS system" and "[How To] Hard Drive Burn-In Testing" threads.

Good luck!

 

[athos56]

Thanks for those links I will perform these burn in procedures. I can also report my build posted, I got my M1015 crossflashed and ESXi 6.0 installed and configured error free. I'm setting up my datastore now and will hopefully get FreeNAS installed tonight. This forum is great, thanks to everyone who replied.

 

[athos56]

Final update, ESXi 6.0 installed, FreeNAS VM installed, Ubuntu + Minecraft Server installed... It all just fell together so nicely over the weekend, no real issues or problems. The only real issue is managing all the consoles and web gui pages, I might have to get a separate monitor.

 

[baummer]

Final update, ESXi 6.0 installed, Freenas VM installed, Ubuntu + Minecraft Server installed... It all just fell together so nicely over the weekend, no real issues or problems. The only real issue is managing all the consoles and web gui pages, I might have to get a separate monitor.

What are your full hardware specs?

 

[athos56]

I got a supermicro x8dtu-f with 2x E5620, 32gb ram and a m1015 flashed to IT mode. 6x 1tb WD drives, ESXi installed to a 16gb thumb drive. A 64gb SSD datastore, a 120gb ssd slog and a 8gb ssd zil. I dumped it all in a Rosewill 4500 case and replaced all the fans with 4 pin pwm fans and the low profile heat sinks with large heatsinks. I have some new processors and 96gb of ram in the mail.

 

[baummer]

I got a supermicro x8dtu-f with 2x E5620, 32gb ram and a m1015 flashed to IT mode. 6x 1tb WD drives, ESXi installed to a 16gb thumb drive. A 64gb SSD datastore, a 120gb ssd slog and a 8gb ssd zil. I dumped it all in a Rosewill 4500 case and replaced all the fans with 4 pin pwm fans and the low profile heat sinks with large heatsinks. I have some new processors and 96gb of ram in the mail.

Thanks! How is it running?

 

[athos56]

I performed a brain transplant tonight, dropped in couple x5675 processors and 96gb ram. It runs noticeably faster. I think one of the sticks of ram and one of the processors it came with were on the way out. I have 3 VMs so far, Freenas, Ubuntu 16.04 running a Minecraft server, and Windows 10 LTSB. I'm figuring out all the stuff I could do with it. I have broken a couple of things, mostly network stuff, but everything seems pretty resilient. My transfer rates vary from 45mbs to 125mbs depending on were things are going. Overall, I'm super satisfied with how things turned out.