FN9.10: Bhyve / Iohyve hacking

[freenas4n00b]

I am trying to following the guides on the GH https://github.com/pr1ntf/iohyve
Running into a bunch of stalls

# my zfs pool is called DATASTORE yours will probably be different

I ran setup and the utility created all the sub folders
iohyve setup pool=DATASTORE

Next i created the VM
iohyve create ubuntuguest 20G DATASTORE

At this point you should see
[root@freenas] ~# zfs list | grep -i iohyve
DATASTORE/iohyve 20.6G 5.43T 192K /mnt/iohyve
DATASTORE/iohyve/Firmware 192K 5.43T 192K /mnt/iohyve/Firmware
DATASTORE/iohyve/ISO 192K 5.43T 192K /mnt/iohyve/ISO
DATASTORE/iohyve/ubuntuguest 20.6G 5.43T 192K /mnt/iohyve/ubuntuguest
DATASTORE/iohyve/ubuntuguest/disk0 20.6G 5.45T 128K -

Set some variables and config options
[root@freenas] ~# iohyve set ubuntuguest loader=grub-bhyve
Setting ubuntuguest prop loader=grub-bhyve...

[root@freenas] ~# iohyve set ubuntuguest os=debian
Setting ubuntuguest prop os=debian...

[root@freenas] ~# iohyve set ubuntuguest ram=512M
Setting ubuntuguest prop ram=512M...

[root@freenas] ~# iohyve set ubuntuguest cpu=2
Setting ubuntuguest prop cpu=2...

[root@freenas] ~# iohyve set ubuntuguest con=nmdm1
Setting ubuntuguest prop con=nmdm1...

At this point documentation is telling me to install so i moved the ubuntu ISO to DATASTORE/iohyve/ISO
cp ubuntu-14.04.4-server-amd64.iso /mnt/iohyve/ISO/

Then run the install command
iohyve install ubuntuguest ubuntu-14.04.3-server-amd64.iso

But it will not install
[root@freenas] ~# iohyve install ubuntuguest ubuntu-14.04.4-server-amd64.iso
Not a valid guest name

I am looking at the install script at https://github.com/pr1ntf/iohyve/blob/master/iohyve
It looks quite buggy. It assume /iohyve dir which no where is that installed
"if [ -d /iohyve/$pool/$name ] || [ -d /iohyve/$name ]; then"

So i am stuck right here and pondering the next steps to get this working
Just wanted to share and maybe brainstorm with those who are interested in testing this

 

[freenas4n00b]

Looks like FN9.10 has an old version of iohyve.
[root@freenas] ~# iohyve version
iohyve v0.7 2015/10/27 Im Here for the Party Edition


Current is
iohyve v0.7.3 2016/01/08 Bear in a Datacenter Edition"

Maybe some of these bugs are fixed in latest version. i will submit bug for Freenas team to upgrade to latest version of iohyve

 

[fracai]

Looks like there's a workaround in this ticket: https://bugs.freenas.org/issues/14212

 

[dlavigne]

That's as far as I got as well when I was writing up the docs to appear in the next SU. I was told it was because I was testing in vbox. If that's not your case, it is possible that the newer version of iohyve will fix it.

 

[freenas4n00b]

I am definitely not on vbox.

 

[freenas4n00b]

Freenas bug (14290) said that the latest version of iohyve is in nightlies. I dont use nightlies so i need to wait for next SU.
I also filed a github issue on iohyve repo. Lets see if he can help me sort this out

 

[freenas4n00b]

So i did make some progress.
I do have a running ubuntu VM however with no networking so that is the next problem i need to solve.

[root@freenas] ~# iohyve list
Guest VMM? Running? rcboot? Description
ubuntuguest YES YES NO Fri_Mar_25_17:32:58_PDT_2016

What happened was i noticed that the vmm.ko and nmdm.ko were not loaded (using kldstat).
So after reading the docs, i found that i did not follow the instructions properly.

This is the first command which i messed up and probably where i need to troubleshoot the network issues.

iohyve setup pool=DATASTORE kmod=1 net=igb1

 

[freenas4n00b]

Couple of things:

1. Can someone guide me on how to load vmm and nmdm kernal modules at boot?
I found that they dont and it is causing iohyve issues
[root@freenas] ~# kldstat
Id Refs Address Size Name
1 80 0xffffffff80200000 17dd000 kernel
2 1 0xffffffff81cae000 fdd54 ispfw.ko
3 1 0xffffffff81e11000 f946 geom_mirror.ko
4 1 0xffffffff81e21000 46a1 geom_stripe.ko
5 1 0xffffffff81e26000 ffca geom_raid3.ko
6 1 0xffffffff81e36000 ec6a geom_raid5.ko
7 1 0xffffffff81e45000 574f geom_gate.ko
8 1 0xffffffff81e4b000 4a32 geom_multipath.ko
9 1 0xffffffff81e50000 5717 fdescfs.ko
10 1 0xffffffff81e56000 89c dtraceall.ko
11 10 0xffffffff81e57000 3ad17 dtrace.ko
12 1 0xffffffff81e92000 4636 dtmalloc.ko
13 1 0xffffffff81e97000 225b dtnfscl.ko
14 1 0xffffffff81e9a000 63d7 fbt.ko
15 1 0xffffffff81ea1000 58109 fasttrap.ko
16 1 0xffffffff81efa000 49ca lockstat.ko
17 1 0xffffffff81eff000 15f8 sdt.ko
18 1 0xffffffff81f01000 d8d7 systrace.ko
19 1 0xffffffff81f0f000 d491 systrace_freebsd32.ko
20 1 0xffffffff81f1d000 4da2 profile.ko
21 1 0xffffffff81f22000 7fde ipmi.ko
22 1 0xffffffff81f2a000 1a63a hwpmc.ko
23 2 0xffffffff81f45000 2b32 vboxnetflt.ko
24 2 0xffffffff81f48000 45370 vboxdrv.ko
25 1 0xffffffff81f8e000 41ca ng_ether.ko
26 1 0xffffffff81f93000 3fd4 vboxnetadp.ko
27 1 0xffffffff81f97000 9fc5 linprocfs.ko
28 1 0xffffffff81fa1000 66f9 linux_common.ko

iohyve only does it at pool creation time. Unless you plan to create pools after every reboot, it wouldn't work.

2. Does anyone know when nightlies fixes are merged into STABLE SUs?
https://bugs.freenas.org/issues/142900
My bug is resolved but the version in STABLE is still old version even after updating to the latest SU.

Thanks

 

[mav@]

Can someone guide me on how to load vmm and nmdm kernal modules at boot?

Set "vmm_load" and "nmdm_load" loader tunables to "YES".

Does anyone know when nightlies fixes are merged into STABLE SUs?

There is no separate 9.10 stable branch yet, so each SU automatically includes all previous nightly changes.

 

[freenas4n00b]

Thanks for answering so fast mav

1. So is it supposed to look like this?
So under tunables UI:
Variable: vmm_load
Value: YES
type: loader

1a. In the documentation there is this warning "Warning adding a sysctl, loader, or rc.conf option is an advanced feature. A sysctl immediately affects the kernel running the FreeNAS® system and a loader could adversely affect the ability of the FreeNAS® system to successfully boot. Do not create a tunable on a production system unless you understand and have tested the ramifications of that change."

Does anyone know if there are any problems loading these two kernel modules?

2. Weird i still do not have the fix in the latest SU. I guess i will wait until next one before i open a ticket.

 

[freenas4n00b]

Can someone please answer?

 

[mav@]

1. This is new experimental feature. There are no guaranties other then expectations that it can be useful.

2. Questions about ticket is better to send to ticket.

 

[Spa]

I'm having the same issue - in addition, I updated to the current nightly build and the iohyve version is still 0.7.0, not 0.7.4 as has been reported in the bug tracker. Side note, does the bug tracker link posted in here for $149200 404 for everyone else too?

 

[dlavigne]

The bug # is 14290 and it should be fixed for the next SU.

 

[freenas4n00b]

thanks dlavigne for pushing this through.

 

[Spa]

Thanks for the upcoming fix.

 

[dlavigne]

I'm starting to wonder if this is the issue as I'm definitely on the latest iohyve, both modules are loaded, and I get the same error:

bhyve is a hypervisor/virtual machine manager for FreeBSD that was introduced in FreeBSD 10.0 and supports most Intel and AMD processors that report the "POPCNT" (POPulation Count) processor feature in dmesg(8).

I'm definitely not seeing anything POP related in my dmesg on an Intel i5-4200M.

 

[freenas4n00b]

@dlavigne
What error are you referring to? I have no problems deploying VMs right now.

My current nightmare is getting the networking to function. iohyve's script expects bridge0 to not be there and stops if it is there. I have a bridge0 already because i use freenas jails
So i have been spending hours and hours hacking at the network settings to get my VMs to network properly.

 

[dlavigne]

Following my own instructions at http://doc.freenas.org/9.10/freenas_jails.html#using-iohyve, the very last command (iohyve install) fails with:

vm_create: Device not configured

NOTE: if anyone does have it working, if they could send me the output of a successful create plus any other useful list commands, I can at least get the docs updated to what it looks like when it works.

 

[freenas4n00b]

Based on this mailing list discussion i would say your assessement about CPU features as a root cause is correct.

https://lists.freebsd.org/pipermail/freebsd-virtualization/2013-April/001194.html


If i ever get my networking issue resolved. i would post it here for everyone to consume.