BUILD ESXi Home Server + FreeNAS

Status
Not open for further replies.

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
How do you setup the bootorder to make this work? (I read somewhere a grinch comment about boot strap paradoxes)
With FreeNAS shell scripts executed as post-init tasks.

If you enable SSH on the ESXi server and set up 'passwordless' access with public keys, you can use VMware CLI commands to do pretty much anything you need to do on the ESXi server.

I have a set of startup scripts that:

> Force ESXi to re-scan all data stores. This makes the NFS or iSCSI FreeNAS datastores 'wake up' and become available to ESXi when FreeNAS is restarted.
> Start virtual machines

I also have shutdown scripts that gracefully powers down all running VMs when FreeNAS is shut down. This works great -- except with the current stable release of FreeNAS 9.10, though, oddly enough, the scripts work fine with the nightlys. So I anticipate that the next stable 9.10 release will fix the problem. I filed a bug report here (the actual problem is that networking - or name resolution, at least - is stopped before user shutdown tasks are executed) :

https://bugs.freenas.org/issues/15323

The shutdown scripts are available at the bug report, if you're interested in downloading and using them.

Here is the startup script I set up as a post-init task:
Code:
#!/bin/bash

L_LOGFILE="/mnt/tank/sysadmin/log/vmware-startup.log"
L_WAIT_DELAY=30

echo "$(date): Force datastore re-scan on FELIX..." > $L_LOGFILE

ssh root@felix.ncs esxcli storage core adapter rescan --all >> $L_LOGFILE

echo "Pausing $L_WAIT_DELAY seconds before starting virtual machines..." >> $L_LOGFILE
sleep $L_WAIT_DELAY
/mnt/tank/sysadmin/start-virtual-machine.sh root felix adonis  >> $L_LOGFILE
/mnt/tank/sysadmin/start-virtual-machine.sh root felix aphrodite  >> $L_LOGFILE

exit

And here is the script that starts up a given VM ('start-virtual-machine.sh'):
Code:
#!/bin/bash

################################################################################
# Usage: start-virtual-machine.sh user_id esxi_host_name vmx_base_filename
#
# Starts guest virtual machine with vmx file (vmx_base_filename'.vmx') on remote
# VMware ESXi server (esxi_host_name) using given user credentials (user_id)
#
# Sends commands to the remote host using SSH, which must be configured before
# using this script.
#
# Tested with FreeNAS 9.3 (STABLE) running as a VM on VMware ESXi v6.0
################################################################################

# Check for usage errors

if [ $# -ne 3 ]
then
  echo "$0: error! Not enough arguments"
  echo "Usage is: $0 user_id esxi_host_name vmx_filename"
  echo "Only specify the vmx basefilename; leave off the '.vmx' extension"
  exit 1
fi

# Gather command-line arguments for user ID, hostname, and datastore name:

L_USER=$1
L_HOST=$2
L_VMXNAME=$3

# Get server ID for the VM with matching vmx file:

L_GUEST_VMIDS=$(ssh ${L_USER}@${L_HOST} vim-cmd vmsvc/getallvms | grep "/${L_VMXNAME}.vmx" | awk '$1 ~ /^[0-9]+$/ {print $1}')

echo "$0: $L_USER@$L_HOST vmx=$L_VMXNAME.vmx"

for L_VMID in $L_GUEST_VMIDS
do
  ssh ${L_USER}@${L_HOST} vim-cmd vmsvc/power.getstate $L_VMID | grep -i "off\|Suspended" > /dev/null 2<&1
  L_SHUTDOWN_STATUS=$?

  if [ $L_SHUTDOWN_STATUS -ne 0 ]; then
  echo "Guest VM ID $L_VMID already powered up..."
  else
  echo "Powering up guest VM ID $L_VMID..."
  ssh ${L_USER}@${L_HOST} vim-cmd vmsvc/power.on $L_VMID
  fi
done

exit
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Mah, not really ?
The trade-off of my suggested solution would be to accept that FreeNAS is not installed on the same Storage area as the remaining VM's on the same machine.
Comparing to running hardware raid solutions, sticking with FreeNAS, I can see benefits like ease of administration in terms of adding space or IOPS through additional vdevs to the pool below the zvol/iSCSI.
When aiming to run only a single box for all purposes, that seems to be a legit trade.

Cheers / Dice
Exactly so! Both of my all-in-one FreeNAS+VMware servers:

1> Boot ESXi from a 16GB USB stick
2> ESXi boots FreeNAS, which is installed on SSD local storage (mirrored or not)
3> Once booted up, FreeNAS data stores are available to ESXi and other VMs may then be started
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Mah, not really ?
The trade-off of my suggested solution would be to accept that FreeNAS is not installed on the same Storage area as the remaining VM's on the same machine.

That's always a given. Well, I guess unless you do some godawfulhack.

Basically there's no way to make all of this play nice like a professional VMware installation without some godawfulhacks.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
Yea, I would highly recommend that FreeNAS be on your ESXi boot device/datastore since it only consumes a few megabytes (mine is set for 10MB but 4MB is just fine) and it doesn't need to be mirrored, but you do need to backup your configuration file any time you update the system configuration so you can conduct an easy restoration should things go badly.

EDIT: My ESXi + FreeNAS + Sopho(x2) + Ubuntu + Windows VM system has been going strong since Feb 2016. FreeNAS nor any VM has crashed. Windows is the only VM I turn off when not actively in use. Ubuntu is running SETI@Home on BOINC 24/7 using 50% of the resources.
 
Last edited:

Robert Smith

Patron
Joined
May 4, 2014
Messages
270
Let me ask you this, why ESXi for home use?

Licensing costs of ESXi are considerable, unless you are talking about the free ESXi that lacks proper storage fabric.

Why not pick a free open source (FOS) solution such as XenServer or KVM? Even Microsoft has a free fully featured hypervisor (Hyper-V Server); though with Microsoft you never know what kind of licensing curveball they may throw in the future.

ESXi is great, but the choice of the most expensive virtualization software for a budget conscious build is puzzling.
 

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
ESXi is also free license, where u get the basics to run and administer one host at a time. Through 'vSphere Client'.
That is way good enough to me.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Let me ask you this, why ESXi for home use?

Licensing costs of ESXi are considerable, unless you are talking about the free ESXi that lacks proper storage fabric.

Why not pick a free open source (FOS) solution such as XenServer or KVM? Even Microsoft has a free fully featured hypervisor (Hyper-V Server); though with Microsoft you never know what kind of licensing curveball they may throw in the future.

ESXi is great, but the choice of the most expensive virtualization software for a budget conscious build is puzzling.

Because it's the only hypervisor that's been demonstrated to work in a reasonably problem-free manner with FreeNAS.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
ESXi does just work and how many frills does a home system need. I guess if you are a guru on all this stuff then you can go for it and give it a shot but if you know what you are doing with ESXi, that is more than powerful enough for a home system. It's not all point and click but realistically you shouldn't need to do a lot once you set it up.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
ESXi does just work and how many frills does a home system need. I guess if you are a guru on all this stuff then you can go for it and give it a shot but if you know what you are doing with ESXi, that is more than powerful enough for a home system. It's not all point and click but realistically you shouldn't need to do a lot once you set it up.

Well, ESXi probably has more frills than the rest of them combined, but the problem would be that they only exist in the paid editions, which go from merely expensive to insanely crazy.

Five years ago, free ESXi was very compelling because it was head and shoulders better than all the rest of the free hypervisors. That's probably no longer true, but the problem is that VMware has to be very careful about eroding its potential market. I think in the long run they'll be in big trouble as other hypervisors like Xen/KVM/Hyper-V add free capabilities to match vSphere's paid stuff, and VMware's choices become all-bad for a company that tries to profit from things that over time will become available-for-free on other platforms.

However, ESXi is still the only thing that's been demonstrated to work properly with FreeNAS, and has for years, through multiple versions of ESXi. There isn't a compelling reason for most of these guys to go off and be the guinea pig who learns the hard way why some other hypervisor solution "almost" works but still fails in the end. Or maybe doesn't fail in the end. Just as with HBA's, yeah, other HBA's "may" work, but we have a shortlist of them that are known to be rock solid, and people don't tend to stray because at the end of the day people want their storage to be reliable.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
In my current test setup, I have 3 VMs (FreeNas, CentOS and Win 7). Two of which are located on the FreeNas NFS. I have set the Automatic Startup and had to give it like 300 seconds delay. But as it is right now I don't have any issues.

If I tell EXSi to shutdown (and watch each VM) they shutdown properly and in the order desired. As far as the two machines that are on the FreeNas NFS, they power up just fine as well. Sure there is a delay in the process and if you are watching the ESXi Storage you will see at first the two other VMs are "Unavailable" but once FreeNas is online ESXi connects to the storage without issue and the other VMs come right online.
 

Gunndy

Dabbler
Joined
May 17, 2016
Messages
18
I'm moving forward with the Hardware for the ESXi 16 Core/32 Thread Super Server. I'm also going to redo my ZFS Pool and move from 6 2TB drives to 8 2TB drives. I'm torn on whether I should do 8x2TB RaidZ2 or 4 x 2TB Mirrors. My FreeNAS storage is primarily going to be used for the following:

Plex Media Streaming
Windows Backup Repository for my Laptop and Main Desktop system
Home Drive storage for standard Windows 10 Documents, Pictures, etc Libraries
And possibly as additional storage space for ESXi VMs

As I understand it 8 Drive RaidZ2 will cap a 1Gb network connection for reads and writes, so I'm not sure if 4 Mirrors will gain me any performance; however I also like the idea of being able to add space by simply replacing 2 drives at a time.
 

Gunndy

Dabbler
Joined
May 17, 2016
Messages
18
Let me ask you this, why ESXi for home use?

Licensing costs of ESXi are considerable, unless you are talking about the free ESXi that lacks proper storage fabric.

Why not pick a free open source (FOS) solution such as XenServer or KVM? Even Microsoft has a free fully featured hypervisor (Hyper-V Server); though with Microsoft you never know what kind of licensing curveball they may throw in the future.

ESXi is great, but the choice of the most expensive virtualization software for a budget conscious build is puzzling.

I'm choosing ESXi because it's free. To my knowledge the only thing you have to pay for with ESXi is if you want vcenter. I am perfectly fine with using the legacy vSphere client to manage my VMs. Also, I am choosing ESXi because I work in IT and gaining knowledge of that platform helps improve my work knowledge due to it's large install base in the Enterprise.

If I choose to consider trying some of the vcenter stuff, you can get access to those things with relatively cheap annual membership.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
Thanks Mirfster. I will try the web interface tomorrow. I didn't know they added it to the free version. Like yourself, I am so used to the old client, that I never even thought to look.

Even when I had my 3 node cluster with vCenter, I always used the fat client.


Sent from my iPhone using Tapatalk
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
It is not the best thing and is a more along the lines of "beta" if you ask me, but is nice if you wanted to allow certain users to be able to check VMs or even start/stop them if needed. At least that is what I will mainly use it for. :)
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
Web interface in the 6U2? Hum... maybe I should upgrade to U2. I'm always looking for alternative ways to manage my system. If I don't need to install vSphere on a computer then I'm much happier.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Web interface in the 6U2? Hum... maybe I should upgrade to U2. I'm always looking for alternative ways to manage my system. If I don't need to install vSphere on a computer then I'm much happier.
The web interface feels very limited compared to the vSphere client. I gave it a run-through once; was not impressed; and haven't used it since.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
I'll upgrade this weekend and see how things go.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
I gave it a spin this morning. It's putty, but like Spearfoot, I will use the fat client for now.


Sent from my iPhone using Tapatalk
 
Status
Not open for further replies.
Top