FN9.10: Bhyve / Iohyve hacking

[freenas4n00b]

So I'm trying to run iohyve setup pool=mydataset, but I'm running into an issue. I keep getting

Code:

iohyve setup pool=io_volun
Setting up iohyve pool...                                                                                                       
cannot create 'io_volun/iohyve': no such pool 'io_volun'                                                                        
cannot open 'io_volun/iohyve': dataset does not exist                                                                           
cannot create 'io_volun/iohyve/ISO': no such pool 'io_volun'                                                                    
cannot create 'io_volun/iohyve/Firmware': no such pool 'io_volun'

Running as sudo root returns "Sorry, user root is not allowed to execute '/usr/local/sbin/iohyve setup pool=io_volun' as root on freenas.local.", so I tried running sudo as my main user account, only to get the same errors as before. Anyone else run into this before and know the solution?

@Spa
Can you show zfs list command?

 

[freenas4n00b]

Well I think I'm close, first of all ~. exits the console not ~~. as the command tells you. I did a ifconfig tap0 create to create the interface and when I do iohyve install <guest> <iso> it starts, tap0 comes up, and I can connect and install, but after the install is done and I exit the console and do a destroy and a start it won't boot. It's an ubuntu server guest using the same version as freenas4noob used. iohyve list tells me vmm Yes but running No. Is there a log somewhere to see what's going on with it? The destroy after the install returns errno 37 if that means something to anyone.

@Codo ~~. works for me everytime.
Lets troubleshoot this together. Can you give exact steps you used to deploy this VM? Hard for me to follow what commands you ran

Can you also give me output of iohyve getall ##VMNAME## ?

 

[Codo]

@Codo ~~. works for me everytime.
Lets troubleshoot this together. Can you give exact steps you used to deploy this VM? Hard for me to follow what commands you ran

Can you also give me output of iohyve getall ##VMNAME## ?

Headed out of town for a few days, I'll post the information later in the week when I return.

 

[Spa]

@Spa
Can you show zfs list command?

Code:

/% zfs list
NAME  USED  AVAIL  REFER  MOUNTPOINT
SSD_Storage  648K  106G  96K  /mnt/SSD_Storage
SSD_Storage/IO_VOLUN  96K  106G  96K  /mnt/SSD_Storage/IO_VOLUN

I trimmed it to just show the volume in question where I'm installing things. With the new version of Iohyve (0.7.5), running the setup pool command returns some new information along with the earlier error:

Code:

sudo iohyve setup pool=IO_VOLUN
Password:
Setting up iohyve pool...
cannot create 'IO_VOLUN/iohyve': no such pool 'IO_VOLUN'
cannot open 'IO_VOLUN/iohyve': dataset does not exist
cannot create 'IO_VOLUN/iohyve/ISO': no such pool 'IO_VOLUN'
cannot create 'IO_VOLUN/iohyve/Firmware': no such pool 'IO_VOLUN'
On FreeNAS installation.
Checking for symbolic link to /iohyve from /mnt/iohyve...
iohyve does not seem to be setup.

Trying ln -s /mnt/iohyve /iohyve returns this:

Code:

sudo ln -s /mnt/iohyve /iohyve
ln: /iohyve: File exists

Which I believe means the link does already exist, correct?

Looking at the iohyve source code, it seems the "iohyve does not seem to be setup" is somewhat of a catch-all for when everything else doesn't work. (As of 4-19-16, line 226 here: https://github.com/pr1ntf/iohyve/blob/master/iohyve)

I appreciate how helpful you've been @freenas4noob, this thread has been a goldmine of information.

 

[freenas4n00b]

@Spa i need to check my notes but i believe i had problems getting iohyve to accept a pool location that is nested. So i just installed the iohyve data in the root.
If i am reading your zfs list command properly, using pool=SSD_Storage should work.
I personally did not care where in the zfs structure the iohyve data was. If you do care, i suggest you open a github request with the developer. He has been really responsive.

You also may want to try pool=SSD_Storage/IO_VOLUN and see if that works

 

[Spa]

Ah, I didn't realize it couldn't be in a nested pool! I've tried again on SSD_Storage, and thus far everything is working, with one exception; my Ubuntu server VM cannot auto-configure the network, and I think I know why, but not how to fix it.

In your walkthrough, you do this:

Code:

ifconfig bridge0 addm tap0

# Then bring up the tap interface in question
# I have read there is a tunable to make sure all tap interfaces are auto enabled.
#  I havent tested that yet but will look into implementing that soon
ifconfig tap0 up

This didn't work for me (bridge0 does not exist, tap0 does not exist), so I read more and saw the below:

Code:

2a. bridge0 is "added" by default by iohyve. The reason for the quotes around added is because you would have a bridge0 interface already if you used freenas jails. If a person never used jails at all, it would have just worked.

I'm in the category of never having used jails at all, although it appears not to "just work." After the install completed, I got a message in console saying "ifconfig: interface bridge0 does not exist". Should I be manually setting up bridge0 somehow for this to work? Sorry if this is an obvious question....

Also, as an aside, the FreeNas GUI Network->Network Summary does show a tap0, and I have run sudo ifconfig tap0 up.

I also realize this may be more of an Ubuntu install question than an iohyve one, but honestly I'm too green at this stuff to tell.

 

[freenas4n00b]

@Spa can you provide the following:

1. Results of iohyve getall ###VMNAME##
2. Results of iohyve version

 

[Spa]

Code:

iohyve getall volunteersearch
Getting volunteersearch iohyve properties...
bargs  -A_-H_-P
boot  0
con  nmdm0
cpu  1
description  Tue Apr 19 23:59:43 PDT 2016
install  yes
loader  grub-bhyve
name  volunteersearch
net  igb2
os  debian
persist  1
ram  1024M
size  8G
tap  tap0

~% iohyve version
iohyve v0.7.5 2016/04/14 Tennessee Cherry Moonshine Edition

 

[freenas4n00b]

Very weird. To my eyes the config looks right. If i were you, i would just create the bridge interface to get things working and open an issue for the iohyve developer to investigate why it doesn't just work.

 

[Spa]

I'll make sure to do that - thanks for all your help! I also noticed you updated your guide - I feel like that should be in the current 9.10 documentation, it's much more robust than what's there right now. I really appreciate all your help. Here's a couple things I noticed when going through the install process:

My install hung, so I opened a new session simultaneously and did sudo iohyve console <name>, which brought me to I think the grub menu? Anyway, first time around, I didn't know what happened so I deleted my VM and re-created it, same thing. I explored the commands, and it turns out all I had to do was type "boot" and it went into the install menu.

Another thing was during the Ubuntu install process, I set it up to use the entire disk with LVM first time around, and the grub bootloader was super buggy when I attempted to start up and get into the server, so I deleted that one again and selected no LVM the next time around.
I also had to stop, then destroy the VM after the Ubuntu installer finished. I tried to destroy first, but it threw an error (37). Attempting to stop, then destroy worked, and then I could start and console into the actual Ubuntu OS for the first time. Hopefully this helps someone else facing similar issues.

 

[icsy7867]

Really stupid question...
When you say "~~." exits the console, what does that mean? I have tried all sorts of combinations of pressing "~" and "." with no such luck. I feel really dumb for not knowing/understanding but I just can't figure it out!

Second dumb question, does BHYVE/IOHYVE support local storage? I really like the storage mounting features of the jails, but I realize that BHYVE might not be there yet.

Thanks!

 

[Hartmut Horch]

Hi to all,
I've upgraded freenas to 9.3. I was able to migrate my centos6 virtualbox image to bhyve. It's running realy good and stable. But I'm no able to start my migrated Win10 virtualbox image with bhyve. I'm also not able to install a new Windows (10,7,2008r2,2012) with bhyve. The question now for me is, does freenas 9.3 supports Windows guests with bhyve or not? Has someone successfully run a Windows machine?

 

[icsy7867]

Hi to all,
I've upgraded freenas to 9.3. I was able to migrate my centos6 virtualbox image to bhyve. It's running realy good and stable. But I'm no able to start my migrated Win10 virtualbox image with bhyve. I'm also not able to install a new Windows (10,7,2008r2,2012) with bhyve. The question now for me is, does freenas 9.3 supports Windows guests with bhyve or not? Has someone successfully run a Windows machine?

Do you mean Freenas 9.10?

Also for Windows VM's I know you need to inject a certain driver, and update an xml file but I have not done this myself yet:
http://pr1ntf.xyz/windowsunderbhyve.html

Let us know if it works! I am interested in this as well, but I just dont have the time yet.

 

[Hartmut Horch]

Do you mean Freenas 9.10?

Also for Windows VM's I know you need to inject a certain driver, and update an xml file but I have not done this myself yet:
http://pr1ntf.xyz/windowsunderbhyve.html

Let us know if it works! I am interested in this as well, but I just dont have the time yet.

Hi icsy7867,
thanks for the quick answer.
You are right, we speak about FreeNAS 9.10.
The link you've posted to me I know.
They speak about FreeBSD 11-Current.
What I understood is that FreeNAS 9.10 is based on FreeBSD 10.3 and my hope was to find another "trick" to get it work.
So I have to wait.
Thanks again

 

[Spa]

Really stupid question...
When you say "~~." exits the console, what does that mean? I have tried all sorts of combinations of pressing "~" and "." with no such luck. I feel really dumb for not knowing/understanding but I just can't figure it out!

Second dumb question, does BHYVE/IOHYVE support local storage? I really like the storage mounting features of the jails, but I realize that BHYVE might not be there yet.

Thanks!

The actual command is ~. I'm not sure why all the documentation says ~~.

 

[Spa]

So I think I stumbled upon another bug, and a warning for everyone - stop/shutdown your VMs prior to rebooting your FreeNAS box itself. I failed to do so before restarting, and now I get this error when I try to start up my VM.

Code:

[@freenas] ~% sudo iohyve start vsearch
Starting vsearch... (Takes 15 seconds for FreeBSD guests)
[@freenas] ~% Could not create VM ioh-vsearch
Error in initializing VM
vm_create: No such file or directory
VM:ioh-vsearch is not created.

I didn't have this error before when updating FreeNAS (meaning FreeNAS rebooted), however I had shut down my VMs prior to upgrading. I did not this time, which is what leads me to believe that's the cause.
I'll come back if I find a fix.

EDIT: Nevermind, I'm dumb, I forgot to reload the two .kos - kldload nmm & nmdm fixes the issue.

 

[icsy7867]

Has anyone played with this yet?

https://github.com/moogle19/iohyve-windows

I still have some other things I am trying to get figured out with linux and bhyve, but I would love to have a windows machine running at some point down the road

 

[phier]

Here are my instructions. Lines with # are comments meant to add some context to the commands.
Hopefully it will help people to start off. Definitely should do some reading on the iohyve github page (https://github.com/pr1ntf/iohyve) for further knowledge and tweaking.

Code:

# This documentation was built on iohyve version "iohyve v0.7 2015/10/27 Im Here for the Party Edition"
# Commands might have changed in newer iohyve version

# See if the kernel modules are loaded. Need to look for vmm.ko and nmdm.ko
kldstat

# load the missing module
# I have read that people should be adding a tunables for these modules to be loaded by default at boot.
#  The disclaimer on the tunables doc says it is for advanced usages and can break things so i hesitate to do that.
#  I still manually load all these at boot.
kldload vmm
kldload nmdm

# use iohyve to setup storage pool
# Your pool name might be different so please make sure to change that
# I was not able to get the iohyve script to recognize nested pools so i just used the root pool.
# ie if my zfs structure looks like DATASTORE/VirtualMachines, pool=DATASTORE would work
# but pool=DATASTORE/VirtualMachines would not work.
# I did not care where iohyve puts it's content so i did not open ticket on it.
iohyve setup pool=DATASTORE

# confirm iohyve created zfs pool
# My structure looked like this
# DATASTORE/iohyve                                            1.06T  4.39T   192K  /mnt/iohyve
# DATASTORE/iohyve/Firmware                                    192K  4.39T   192K  /mnt/iohyve/Firmware
# DATASTORE/iohyve/ISO                                         580M  4.39T   192K  /mnt/iohyve/ISO
# DATASTORE/iohyve/ISO/ubuntu-14.04.4-server-amd64.iso         580M  4.39T   580M  /mnt/iohyve/ISO/ubuntu-14.04.4-server-amd64.iso
# DATASTORE/iohyve/zoneminder001                               567G  4.39T   200K  /mnt/iohyve/zoneminder001
# DATASTORE/iohyve/zoneminder001/disk0                        51.6G  4.43T  6.26G  -
# DATASTORE/iohyve/zoneminder001/disk1                         516G  4.88T  16.0G  -
# DATASTORE/iohyve/zoneminderdb001                             516G  4.39T   200K  /mnt/iohyve/zoneminderdb001
# DATASTORE/iohyve/zoneminderdb001/disk0                       516G  4.87T  20.4G  -
zfs list

# In it's current iteration, iohyve script only works with storage mounted in /iohyve.
# So the workaround is to create a symbolic link:
ln -s /mnt/iohyve /iohyve

# Create the volume for the VM
iohyve create zoneminder001 50G

# Set various VM variables
# Your settings might be different so do not just copy and paste.
iohyve set zoneminder001 loader=grub-bhyve
iohyve set zoneminder001 os=debian
iohyve set zoneminder001 ram=4096M
iohyve set zoneminder001 cpu=4
iohyve set zoneminder001 net=igb1

# If you want to add another volume. I usually separate out my data into its own volume
# Totally optional.
# You will need to follow the ubuntu documentation on how to mount this volume into filesystem
iohyve add zoneminder001 500G DATASTORE

# Check if there is a bridge0 interface
ifconfig

# If you do not have the bridge0 interface then create it
ifconfig bridge0 create

# Add your main network interface to the bridge0
# Replace ##PRIMARYINTERFACE## with your interface name (ie igb1 etc)
ifconfig bridge0 addm ##PRIMARYINTERFACE##

# Each VM will create a tap interface.
# You will need to manually add the tap interface to the bridge
ifconfig bridge0 addm tap0

# If you had to manually create the bridge interface now is the time to bring it up
ifconfig bridge0 up

# Then bring up the tap interface in question
# I have read there is a tunable to make sure all tap interfaces are auto enabled.
#  I havent tested that yet but will look into implementing that soon
ifconfig tap0 up

# You will need to put the guest iso into local filesystem
# In my case i scped the iso from my laptop to the server and ran cpiso
# iohyve also has documentation for the iohyve fetch command which can pull down the iso directly onto the server
iohyve cpiso ./ubuntu-14.04.4-server-amd64.iso

# Install command
iohyve install zoneminder001 ubuntu-14.04.4-server-amd64.iso

# connect into console and run through the installation steps
iohyve console zoneminder001

# I still do not know the difference between destroy and stop a VM
iohyve destroy zoneminder001

# Start the VM
iohyve start zoneminder001

# Confirm VM is running
# Guest            VMM?  Running?  rcboot?  Description
# zoneminder001    YES   YES       NO       Fri_Apr__8_11:43:29_PDT_2016
# zoneminder001    YES   YES       NO       Fri_Apr__8_11:43:29_PDT_2016
# zoneminderdb001  YES   YES       NO       Fri_Apr__8_20:34:51_PDT_2016
iohyve list

# Log into the VM
iohyve console zoneminder001

Hi,
i went step by step, but instead of debian im trying to install ubuntu-server ubuntu-16.04-server-amd64.iso.

Installation went fine, i installed it with full drive encryption (lvm).

After reboot, i tried to destroy and start machine... but its not starting anymore

[root@freenas ~]# iohyve start ubuntu_universe
Starting ubuntu_universe... (Takes 15 seconds for FreeBSD guests)
[root@freenas ~]# iohyve list
Guest VMM? Running rcboot? Description
ubuntu-server YES NO NO Thu May 12 01:26:08 CEST 2016
ubuntu_universe YES NO NO Thu May 12 01:46:21 CEST 2016


So i connected to the console and all what is there is grub prompt : grub >
grub> ls
(hd0) (hd0,msdos5) (hd0,msdos1) (cd0) (cd0,msdos5) (cd0,msdos1) (host)
grub>


no idea what should i do?


Also im wondering where are stored the virtual drives inside my pool?
du -hs /iohyve/ubuntu_universe/
7.5K

but i created drive as 20G image.


Im running freenas 10.3
iohyve v0.7.3 2016/01/08 Bear in a Datacenter Edition


Thanks!

 

[icsy7867]

Hi,
i went step by step, but instead of debian im trying to install ubuntu-server ubuntu-16.04-server-amd64.iso.

Installation went fine, i installed it with full drive encryption (lvm).

After reboot, i tried to destroy and start machine... but its not starting anymore

[root@freenas ~]# iohyve start ubuntu_universe
Starting ubuntu_universe... (Takes 15 seconds for FreeBSD guests)
[root@freenas ~]# iohyve list
Guest VMM? Running rcboot? Description
ubuntu-server YES NO NO Thu May 12 01:26:08 CEST 2016
ubuntu_universe YES NO NO Thu May 12 01:46:21 CEST 2016


So i connected to the console and all what is there is grub prompt : grub >
grub> ls
(hd0) (hd0,msdos5) (hd0,msdos1) (cd0) (cd0,msdos5) (cd0,msdos1) (host)
grub>


no idea what should i do?


Also im wondering where are stored the virtual drives inside my pool?
du -hs /iohyve/ubuntu_universe/
7.5K

but i created drive as 20G image.


Im running freenas 10.3
iohyve v0.7.3 2016/01/08 Bear in a Datacenter Edition


Thanks!

Does it work without encryption/LVM? I could have sworn I read about LVM not working with BHYVE... But I could be wrong!

Poking around the googles...

https://forums.freebsd.org/threads/49731/

The Ubuntu text installer will start.

The installer asks some basic questions. Watch out -- select "Guided - use entire disk".

Note: Select Guided - use entire disk
The default -- Guided -use entire disk and set up LVM
Caused Ubuntu to crash later on during boot. YMMV.

I have an ubuntu guest up and running without LVM, and I use X11RDP to RDP into it, and havent had any issues yet!

 

[phier]

.