FN9.10: Bhyve / Iohyve hacking

[Spa]

Yes, I was able to fix grub.
But the rest of the problems remained. It's a iohyve problem, caused by the very old Linux version, Meteohub is using.

Interesting, my issue must be different then, since I'm using the same version of Ubuntu I used in prior VMs. Back to troubleshooting.

 

[Spa]

Yes. As in my post, I have two VMs running perfectly, one Crashplan/CentOS and one Ubuntu Svr 16.04. They just don't show up in any of the iohyve lists except 'info.'

Ah I see, I misinterpreted your post to mean you had set up the basic iohyve steps, but not the VMs. My bad.

 

[Spa]

So with the new update, I've tried setting up a new ubuntu vm again - this time I got a different error message on the console:

Code:

Connected
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(root@gauntlet, Mon Nov  7 17:20:52 UTC 2016)
\
can't load 'kernel'

lsdev and show return this:

Code:

OK lsmod
OK lsdev
host devices:
  host0:  Host filesystem
disk devices:
  disk0:  Guest drive image
  disk0s2: Unknown
zfs devices:
OK show
LINES=24
boot_serial=1
console=userboot
currdev=disk0:
interpret=OK
loaddev=disk0:
prompt=${interpret}
smbios.bios.vendor=BHYVE
twiddle_divisor=1

I'm not quite up to speed enough with *unix to really figure out how this information is helpful, but if anyone else is encountering similar/the same problems and wants to see what someone else's stuff looks like, hopefully this helps.


Also, my properties for the config:

Code:

bargs  -A_-H_-P
boot  0
con  nmdm3
cpu  6
description  Tue Nov  8 15:18:06 PST 2016
install  yes
loader  grub-byhve
name  plex
net  igb0
os  debian
persist  1
ram  4096M
size  20G
tap  tap3

Edit: Fixed with 0.7.7

 

[RoboKaren]

Can people tell me if they have more than 2GB ram (ram=2048M+) if they are actually seeing that in their vm? You can do this by running the

Code:

lshw

command. You can do it from userspace and it'll still tell you the ram amount (and some more details if you sudo it). I'm only seeing 2GB available ram despite what I set in iohyve.

 

[Spa]

Can people tell me if they have more than 2GB ram (ram=2048M+) if they are actually seeing that in their vm? You can do this by running the

Code:

lshw

command. You can do it from userspace and it'll still tell you the ram amount (and some more details if you sudo it). I'm only seeing 2GB available ram despite what I set in iohyve.

This is late, but now that the latest update also updated iohyve, I see the full 8GB of ram I have assigned to my server when I run lshw. I do recall reading something about there being an issue with bhyve instances of Windows not showing more than 2GB, but that was a while ago.

 

[RoboKaren]

This is late, but now that the latest update also updated iohyve, I see the full 8GB of ram I have assigned to my server when I run lshw. I do recall reading something about there being an issue with bhyve instances of Windows not showing more than 2GB, but that was a while ago.

When I upgraded, my bhyve VM wouldn't restart. I got it to restart by switching from UEFI to grub-bhyve and it restarted.

After I did that, my memory mysteriously jumped from 2 to 8 GB. I'm not sure if it was the upgrade to bhyve or the switch from UEFI to grub that did it, but I'm now a happier camper. The new version also appears to be much more stable.

 

[tslw]

Earlier in thread I saw other people having networking issues if they both have jails and iohyve.
I have the same issue. It appears that my network interface is linked to the bridge of the jails.
It seems I cannot also link to the bridge of iohyve.
How do you solve this issue when you want to use both iohyve and jails and networking in both?

 

[RoboKaren]

Earlier in thread I saw other people having networking issues if they both have jails and iohyve.
I have the same issue. It appears that my network interface is linked to the bridge of the jails.
It seems I cannot also link to the bridge of iohyve.
How do you solve this issue when you want to use both iohyve and jails and networking in both?

Interesting. I haven't encountered this. I have both a Plex jail as well as a bhyve ubuntu VM that runs CrashPlan. They're sharing the same folders (I use SMB to mount the network shares in the Crashplan VM).

 

[tslw]

Can you post your ifconfig of freenas?

 

[tslw]

I just figured it our myself. Somehow I got two bridges. If I remove the iohyve interface from the iohyve bridge and add it to the existing jail bridge is works.

 

[zuzuzzzip]

Hello,

I am trying to get a centos vm running with iohyve.

I saw that iohyve got updated in 9.10.2 and decided to try it out again.

But I always get the error:

Code:

vm_create: No such file or directory

I am using a lagg with LACP, and vlan interfaces on top.
So I setup iohyve to use vlan100.
Config below:

Code:

freenas# iohyve create centos-freeipa 20G  
Creating centos-freeipa...
freenas# iohyve set centos-freeipa loader=grub-bhyve os=centos7 ram=2048m cpu=2
Setting centos-freeipa loader=grub-bhyve...
Setting centos-freeipa os=centos7...
Setting centos-freeipa ram=2048m...
Setting centos-freeipa cpu=2...
freenas# iohyve install centos-freeipa CentOS-7-x86_64-Everything-1611.iso  
Installing centos-freeipa...
GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console centos-freeipa in second terminal to complete GRUB process...
Could not create VM ioh-centos-freeipa
Error in initializing VM
freenas# vm_create: No such file or directory

freenas#
freenas# iohyve getall centos-freeipa
Getting centos-freeipa iohyve properties...
bargs  -A_-H_-P
boot  0
con  nmdm0
cpu  2
description  Wed Jan 11 11:40:39 CET 2017
install  yes
loader  grub-bhyve
name  centos-freeipa
os  centos7
persist  1
ram  2048m
size  20G
tap  tap0
template  NO
vnc  NO
vnc_h  600
vnc_ip  127.0.0.1
vnc_tablet  NO
vnc_w  800
vnc_wait  NO
freenas#

Here is my ifconfig, as you can see I i also have jails running, but directly on the vlan100 interface.

Code:

freenas# ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: connected to core01 (gi21)
   options=a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM>
   ether 9c:5c:8e:4e:d5:7d
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: connected to core01 (gi22)
   options=a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM>
   ether 9c:5c:8e:4e:d5:7d
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 9c:5c:8e:4e:d5:7f
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: no carrier
igb3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 9c:5c:8e:4e:d5:80
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM>
   ether 9c:5c:8e:4e:d5:7d
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: active
   laggproto lacp lagghash l2,l3,l4
   laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
   laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 9c:5c:8e:4e:d5:7d
   inet 192.168.109.11 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.15 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.17 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.18 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.158 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.19 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.16 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.13 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.12 netmask 0xffffff00 broadcast 192.168.109.255
   inet 192.168.109.14 netmask 0xffffff00 broadcast 192.168.109.255
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: active
   vlan: 100 parent interface: lagg0
vlan101: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 9c:5c:8e:4e:d5:7d
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: active
   vlan: 101 parent interface: lagg0
vlan102: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 9c:5c:8e:4e:d5:7d
   inet 10.32.2.2 netmask 0xffffff00 broadcast 10.32.2.255
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: active
   vlan: 102 parent interface: lagg0
vlan103: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 9c:5c:8e:4e:d5:7d
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: active
   vlan: 103 parent interface: lagg0
tap1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: iohyve-centos7
   options=80000<LINKSTATE>
   ether 00:bd:b8:55:6d:01
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: no carrier
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: iohyve-bridge
   ether 02:2a:c5:08:de:00
   nd6 options=1<PERFORMNUD>
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 13 priority 128 path cost 2000000
   member: vlan100 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 7 priority 128 path cost 2000000
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: iohyve-centos-freeipa
   options=80000<LINKSTATE>
   ether 00:bd:47:3d:6a:00
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect
   status: no carrier
freenas# iohyve version
iohyve v0.7.7 2016/11/10 I Think I'll Go for a Walk Edition

Maybe the bridging is not supported on a vlan interface?

EDIT: running iohyve setup with the vlan100 interface solved this.

 

[random003]

Since setting up iohyve, virtualbox vms fail to start with the following error. Anyone know how to fix this? As far as I can tell bhyve is not running anymore.


VT-x is being used by another hypervisor. (VERR_VMX_IN_VMX_ROOT_MODE). VirtualBox can't operate in VMX root mode. Please close all other virtualization programs. (VERR_VMX_IN_VMX_ROOT_MODE)

EDIT: A reboot fixed this issue.

 

[Juppers]

I discovered if you replicate to another pool on the same machine, iohyve will find the duplicate VMs and try to run them after a reboot. This doesn't work out very well.

 

[seltz]

I just figured it our myself. Somehow I got two bridges. If I remove the iohyve interface from the iohyve bridge and add it to the existing jail bridge is works.

Thanks, this solved my problem. Had a similar issue where iohyve was creating bridge0 when I already had a functioning bridge1.

 

[Wasurerarenai]

I discovered if you replicate to another pool on the same machine, iohyve will find the duplicate VMs and try to run them after a reboot. This doesn't work out very well.

Fixed this in my github fork. I've entered a pull request to have this merged upstream.

https://github.com/pr1ntf/iohyve/pull/257

Another pull entered, added the same path filtering to the rest of the iohyve functions.

https://github.com/pr1ntf/iohyve/pull/272