FN9.10: Bhyve / Iohyve hacking

[G3rYy]

@freenas4n00b I had the same network problems, maybe it'll work for you too. I basically just added tap0 as a member of bridge0.

Code:

ifconfig bridge0 addm tap0

I tested it muliple times and it always worked for me.
I also manually updated to the latest version of iohyve but that didn't fix it for me.

 

[freenas4n00b]

@freenas4n00b I had the same network problems, maybe it'll work for you too. I basically just added tap0 as a member of bridge0.

Code:

ifconfig bridge0 addm tap0

I tested it muliple times and it always worked for me.
I also manually updated to the latest version of iohyve but that didn't fix it for me.

G3rYy, can you share how did you manually updated to latest version of iohyve?
I have now two working bhyve ubuntu 14.04 VMs. I had to add the tap interfaces to the bridge0 and also ifconfig up the tap interfaces.

Once i have some breathing time i will post the entire VM deployment steps here for everyone.

 

[fracai]

In addition to your write-up (which I eagerly await) can you include any observations regarding performance? Comparison to jails? Virtual box?

 

[G3rYy]

Glad you got it working too. Here's what I did to manually update iohyve since its just a script.

Code:

freenas# git clone https://github.com/pr1ntf/iohyve.git
Cloning into 'iohyve'...
remote: Counting objects: 1154, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 1154 (delta 13), reused 0 (delta 0), pack-reused 1128
Receiving objects: 100% (1154/1154), 316.87 KiB | 421.00 KiB/s, done.
Resolving deltas: 100% (661/661), done.
Checking connectivity... done.
freenas# cd iohyve/
freenas# which iohyve
/usr/local/sbin/iohyve
freenas# mv /usr/local/sbin/iohyve iohyve.bak
freenas# mv iohyve /usr/local/sbin/iohyve
freenas# gzip iohyve.8
freenas# mv iohyve.8.gz /usr/local/man/man8/iohyve.8.gz
freenas# mv rc.d/iohyve /etc/local/rc.d/iohyve

But in order to get it to work you really don't need to update.

 

[freenas4n00b]

Glad you got it working too. Here's what I did to manually update iohyve since its just a script.

Code:

freenas# git clone https://github.com/pr1ntf/iohyve.git
Cloning into 'iohyve'...
remote: Counting objects: 1154, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 1154 (delta 13), reused 0 (delta 0), pack-reused 1128
Receiving objects: 100% (1154/1154), 316.87 KiB | 421.00 KiB/s, done.
Resolving deltas: 100% (661/661), done.
Checking connectivity... done.
freenas# cd iohyve/
freenas# which iohyve
/usr/local/sbin/iohyve
freenas# mv /usr/local/sbin/iohyve iohyve.bak
freenas# mv iohyve /usr/local/sbin/iohyve
freenas# gzip iohyve.8
freenas# mv iohyve.8.gz /usr/local/man/man8/iohyve.8.gz
freenas# mv rc.d/iohyve /etc/local/rc.d/iohyve

But in order to get it to work you really don't need to update.

Thanks for that G3rYy. I agree updating is not absolutely required to get it to work. However they are constantly adding bug fixes that will help make it easier to use.
Updating it myself would be better than having the Freenas guys do it for every little iteration

 

[freenas4n00b]

Here are my instructions. Lines with # are comments meant to add some context to the commands.
Hopefully it will help people to start off. Definitely should do some reading on the iohyve github page (https://github.com/pr1ntf/iohyve) for further knowledge and tweaking.

Code:

# This documentation was built on iohyve version "iohyve v0.7 2015/10/27 Im Here for the Party Edition"
# Commands might have changed in newer iohyve version

# See if the kernel modules are loaded. Need to look for vmm.ko and nmdm.ko
kldstat

# load the missing module
# I have read that people should be adding a tunables for these modules to be loaded by default at boot.
#  The disclaimer on the tunables doc says it is for advanced usages and can break things so i hesitate to do that.
#  I still manually load all these at boot.
kldload vmm
kldload nmdm

# use iohyve to setup storage pool
# Your pool name might be different so please make sure to change that
# I was not able to get the iohyve script to recognize nested pools so i just used the root pool.
# ie if my zfs structure looks like DATASTORE/VirtualMachines, pool=DATASTORE would work
# but pool=DATASTORE/VirtualMachines would not work.
# I did not care where iohyve puts it's content so i did not open ticket on it.
iohyve setup pool=DATASTORE

# confirm iohyve created zfs pool
# My structure looked like this
# DATASTORE/iohyve                                            1.06T  4.39T   192K  /mnt/iohyve
# DATASTORE/iohyve/Firmware                                    192K  4.39T   192K  /mnt/iohyve/Firmware
# DATASTORE/iohyve/ISO                                         580M  4.39T   192K  /mnt/iohyve/ISO
# DATASTORE/iohyve/ISO/ubuntu-14.04.4-server-amd64.iso         580M  4.39T   580M  /mnt/iohyve/ISO/ubuntu-14.04.4-server-amd64.iso
# DATASTORE/iohyve/zoneminder001                               567G  4.39T   200K  /mnt/iohyve/zoneminder001
# DATASTORE/iohyve/zoneminder001/disk0                        51.6G  4.43T  6.26G  -
# DATASTORE/iohyve/zoneminder001/disk1                         516G  4.88T  16.0G  -
# DATASTORE/iohyve/zoneminderdb001                             516G  4.39T   200K  /mnt/iohyve/zoneminderdb001
# DATASTORE/iohyve/zoneminderdb001/disk0                       516G  4.87T  20.4G  -
zfs list

# In it's current iteration, iohyve script only works with storage mounted in /iohyve.
# So the workaround is to create a symbolic link:
ln -s /mnt/iohyve /iohyve

# Create the volume for the VM
iohyve create zoneminder001 50G

# Set various VM variables
# Your settings might be different so do not just copy and paste.
iohyve set zoneminder001 loader=grub-bhyve
iohyve set zoneminder001 os=debian
iohyve set zoneminder001 ram=4096M
iohyve set zoneminder001 cpu=4
iohyve set zoneminder001 net=igb1

# If you want to add another volume. I usually separate out my data into its own volume
# Totally optional.
# You will need to follow the ubuntu documentation on how to mount this volume into filesystem
iohyve add zoneminder001 500G DATASTORE

# Check if there is a bridge0 interface
ifconfig

# If you do not have the bridge0 interface then create it
ifconfig bridge0 create

# Add your main network interface to the bridge0
# Replace ##PRIMARYINTERFACE## with your interface name (ie igb1 etc)
ifconfig bridge0 addm ##PRIMARYINTERFACE##

# Each VM will create a tap interface.
# You will need to manually add the tap interface to the bridge
ifconfig bridge0 addm tap0

# If you had to manually create the bridge interface now is the time to bring it up
ifconfig bridge0 up

# Then bring up the tap interface in question
# I have read there is a tunable to make sure all tap interfaces are auto enabled.
#  I havent tested that yet but will look into implementing that soon
ifconfig tap0 up

# You will need to put the guest iso into local filesystem
# In my case i scped the iso from my laptop to the server and ran cpiso
# iohyve also has documentation for the iohyve fetch command which can pull down the iso directly onto the server
iohyve cpiso ./ubuntu-14.04.4-server-amd64.iso

# Install command
iohyve install zoneminder001 ubuntu-14.04.4-server-amd64.iso

# connect into console and run through the installation steps
iohyve console zoneminder001

# I still do not know the difference between destroy and stop a VM
iohyve destroy zoneminder001

# Start the VM
iohyve start zoneminder001

# Confirm VM is running
# Guest            VMM?  Running?  rcboot?  Description
# zoneminder001    YES   YES       NO       Fri_Apr__8_11:43:29_PDT_2016
# zoneminder001    YES   YES       NO       Fri_Apr__8_11:43:29_PDT_2016
# zoneminderdb001  YES   YES       NO       Fri_Apr__8_20:34:51_PDT_2016
iohyve list

# Log into the VM
iohyve console zoneminder001

 

[freenas4n00b]

@fracai
In terms of performance, i built my server with virtualization in mind. So far two VMs is barely breaking a sweat on my hardware.
I felt that jails never really supported linux that well. So far, everything i have done on these VMs felt like native linux experience.
I didn't use the freenas implementation of virtualbox.
In terms of ease of use, I can deploy an ubuntu VM in 10 mins so it is quite easy.

I will report back as I use it longer with performance. Once i start adding more cameras to my zoneminder VM, that will generate true load.

 

[fracai]

These are great. Thanks very much for taking the time to put them together.

Two questions about how resources are assigned.
- I see that you've allocated 4GB of RAM. Is that reserved outright? Or just the maximum that can be acquired by the VM?
- I see where you're enabling the network interface, but I don't see bridge0 or tap0 specifically connected to the VM. It sounds like it's created for you by iohyve and can be identified with:

Code:

iohyve get zoneminder001 tap

And thanks for the performance info. It sounds a bit like you're getting better performance than under a jail?

 

[fracai]

I forgot one, do you know how to mount other paths so they're available inside the VM? Similar to the "Add Storage" option with Jails.

 

[freenas4n00b]

These are great. Thanks very much for taking the time to put them together.

Two questions about how resources are assigned.
- I see that you've allocated 4GB of RAM. Is that reserved outright? Or just the maximum that can be acquired by the VM?
- I see where you're enabling the network interface, but I don't see bridge0 or tap0 specifically connected to the VM. It sounds like it's created for you by iohyve and can be identified with:

Code:

iohyve get zoneminder001 tap

And thanks for the performance info. It sounds a bit like you're getting better performance than under a jail?

1. Looking at top command on the freenas command line, i see bhyve is not using all of the memory allocated to the VMs. So that brings me to conclude it is not reserved outright. It should be like other hypervisors and just specify a limit.
2a. bridge0 is "added" by default by iohyve. The reason for the quotes around added is because you would have a bridge0 interface already if you used freenas jails. If a person never used jails at all, it would have just worked.
2b. Tap interfaces are created by default at creation of the VM. For the 2nd VM, i manually set tap=tap1 which iohyve will create for you. Then tap=tap2 for the third VM etc etc
3. I used a freenas regular jail for plex only. Never needed a lot of performance so it is really hard to compare for me
4. The equivalent of "add storage" in jails is something i am looking into next. I don't have an answer for that just yet but definitely need it to migrate my plex jail to ubuntu VM.

 

[fracai]

Great. Thanks for replying, you are saving me a ton of time in the future!

 

[freenas4n00b]

The more people that adopts this technology would lead to us learning from each other

 

[kjp4756]

The version of iohyve that comes with the latest freenas is a few versions behind now. Does manually updating iohyve to the latest version cause any issues with freenas? Any problems with freenas updates or anything?

Thanks.

 

[Pheran]

I forgot one, do you know how to mount other paths so they're available inside the VM? Similar to the "Add Storage" option with Jails.

Unlike jails, bhyve VMs are true virtual machines, so you can't directly add paths from the FreeNAS filesystem to them. You have to share out the path from FreeNAS via NFS or CIFS and then mount that share inside the VM.

 

[fracai]

Hmm, virtual box allows creating a "shared folder". I would have thought the same was possible with bhyve.

 

[mav@]

Passing inside file system requires special drivers from both host and quest. There was some work going to support in bhyve 9p filesystem, supported at least by Linux guests, but I am not sure about its present state.

 

[Spa]

So I'm trying to run iohyve setup pool=mydataset, but I'm running into an issue. I keep getting

Code:

iohyve setup pool=io_volun
Setting up iohyve pool...                                                                                                        
cannot create 'io_volun/iohyve': no such pool 'io_volun'                                                                         
cannot open 'io_volun/iohyve': dataset does not exist                                                                            
cannot create 'io_volun/iohyve/ISO': no such pool 'io_volun'                                                                     
cannot create 'io_volun/iohyve/Firmware': no such pool 'io_volun'

Running as sudo root returns "Sorry, user root is not allowed to execute '/usr/local/sbin/iohyve setup pool=io_volun' as root on freenas.local.", so I tried running sudo as my main user account, only to get the same errors as before. Anyone else run into this before and know the solution?

 

[Codo]

Code:

iohyve add zoneminder001 500G DATASTORE
ifconfig bridge0 addm tap0
ifconfig tap0 up

@freenas4noob, what is the first command supposed to do? Or is it a typo?
Regarding tap0 does that need to be manually created and if so what are the commands for that? Running those 2 lines I get:
# ifconfig bridge0 addm tap0
ifconfig: BRDGADD tap0: No such file or directory
# ifconfig tap0 up
ifconfig: interface tap0 does not exist

And how on earth do you get out of the console once you run the console command? It mentions ~~. but I've tried every combination of ~'s and .'s I can think of and the only way I can find to get out is to kill putty and log in again.

 

[Codo]

Well I think I'm close, first of all ~. exits the console not ~~. as the command tells you. I did a ifconfig tap0 create to create the interface and when I do iohyve install <guest> <iso> it starts, tap0 comes up, and I can connect and install, but after the install is done and I exit the console and do a destroy and a start it won't boot. It's an ubuntu server guest using the same version as freenas4noob used. iohyve list tells me vmm Yes but running No. Is there a log somewhere to see what's going on with it? The destroy after the install returns errno 37 if that means something to anyone.

 

[freenas4n00b]

@freenas4noob, what is the first command supposed to do? Or is it a typo?
Regarding tap0 does that need to be manually created and if so what are the commands for that? Running those 2 lines I get:
# ifconfig bridge0 addm tap0
ifconfig: BRDGADD tap0: No such file or directory
# ifconfig tap0 up
ifconfig: interface tap0 does not exist

And how on earth do you get out of the console once you run the console command? It mentions ~~. but I've tried every combination of ~'s and .'s I can think of and the only way I can find to get out is to kill putty and log in again.

@Codo My apologies for the confusion about that first command.
I have fixed my documentation. It is to add a 2nd volume.