FN9.10: Bhyve / Iohyve hacking

[andyclimb]

Hello all,

I'm looking into upgrading my home server to 9.10 from 9.2. I'm going to do a fresh install and not try to upgrade it, as i've hacked a lot behind the scenes. I've had 9.2 working for a long long time, without issues. but I've got some free time now, and i need to take the plunge to get onto package based updates... i digress...

my questions.. mainly directed at iohyve.

I want to virtualise pfsense in byhyve, and pass through the LAN on one ethernet socket, and also passthrough the other LAN socket which has the WAN on it / to make things a bit more complicated, it will potentially be VLAN tagged. my plan is buy just one managed switch... plug the WAN into it configured as a VLAN1, then pass that to bhyve as the WAN. does any of this sound undoable?

I guess the question is how to pas through two network interfaces using iohyve, more than anything else? And is it possible to pass through the whole interface if i don't want to share it. Does it have to be a tap? I'm very new to this end of things.

I would also want it to be secure? so if this is a bad idea then feel free to pitch in?

Cheers

Andrew

 

[millst]

Probably not the most secure setup...

https://forums.freenas.org/index.php?threads/pfsense-within-virtualbox-on-freenas.39747/

-tm

 

[andyclimb]

this is true.

Alternatively a second NIC card could be added, and passed through to the bhyve pfsense guest. This would mean there would be no 'traffic' being passed by the host os.

 

[brando56894]

A lot of us have asked this, including myself, and it's always recommended against. A more secure route would probably be to use ESXi to virtualized both pfSense and FreeNAS so they're completely separate.

Sent from my Pixel C using Tapatalk

 

[millst]

Yeah, bhyve and virtualbox are both type 2 hypervisors. Even by adding another NIC, you're not truly isolating FreeNAS from the WAN.

Another option, just buy an EdgeRouter Lite. They are less than $100 and you have a dedicated piece of hardware that is designed as a router/firewall.

-tm

 

[andyclimb]

Fair point.

So I'm still interested in this for experimentation purposes in a non-live environment, even though it is not recommended. What is the crack with ESXi anyway, since i've been around with freenas its like a thermonuclear no..... warnings allover the place, and lots of people looking data. has this changed...

anyway...

had anyone managed to get a PCI pass through working.. I can't seem to get it to work..

device:
```
igb1@pci0:6:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
```

this is added to /etc/loader.conf

```
vmm_load="YES"
pptdevs="6/0/0"
```

same things added to a tuneable for loader (-/+ the above entered in loader.con)

What ever I do thought the device is not captured..

any ideas..

 

[brando56894]

From what I've seen it's perfectly fine to virtualize FreeNAS as long as you do it correctly. I'm interested in going this route also just for the fun of it, but I need another 32 GB worth of RAM first. I bought a barebones box to run pfSense on but it didn't come with a CPU fan (it's an integrated Celeron D) so the fan I had on it was hacky and loud, it's also another thing to add to the mess. I replaced it with a revived TP-Link Archer C7 running OpenWRT, although it would be nice to run pfSense on my server in a VM and just use the C7 for wifi :D

 

[sharky]

This thread helps me to get autostart working . https://github.com/pr1ntf/iohyve/issues/150

 

[cyky]

I'm looking into experimenting with iohyve. However I'm wondering is it recommended to have different zfs pool for backups and iohyve or will iohyve just create own volume inside given pool?

 

[sharky]

I used my existing pool and created a new dataset

 

[sharky]

Hmm, did a reboot and now this.
Password:
freenas# iohyve tvh getall
Getting iohyve properties...
cannot open 'Storage/iohyve/': invalid dataset name

freenas# df -h
Filesystem Size Used Avail Capacity Mounted on
....
Storage/iohyve 4.1T 128K 4.1T 0% /mnt/iohyve
Storage/iohyve/Firmware 4.1T 128K 4.1T 0% /mnt/iohyve/Firmware
Storage/iohyve/ISO 4.1T 128K 4.1T 0% /mnt/iohyve/ISO
Storage/iohyve/ISO/debian-8.2.0-amd64-i386-netinst.iso 4.1T 545M 4.1T 0% /mnt/iohyve/ISO/debian-8.2.0-amd64-i386-netinst.iso
Storage/iohyve/ISO/debian-8.4.0-amd64-CD-1.iso 4.1T 128K 4.1T 0% /mnt/iohyve/ISO/debian-8.4.0-amd64-CD-1.iso
Storage/iohyve/tvh 4.1T 128K 4.1T 0% /mnt/iohyve/tvh

freenas# zfs list
NAME USED AVAIL REFER MOUNTPOINT
20.9M 4.15T 20.5M /mnt/Storage/XBMC
Storage/iohyve 10.8G 4.15T 128K /mnt/iohyve
Storage/iohyve/Firmware 128K 4.15T 128K /mnt/iohyve/Firmware
Storage/iohyve/ISO 545M 4.15T 128K /mnt/iohyve/ISO
Storage/iohyve/ISO/debian-8.2.0-amd64-i386-netinst.iso 545M 4.15T 545M /mnt/iohyve/ISO/debian-8.2.0-amd64-i386-netinst.iso
Storage/iohyve/ISO/debian-8.4.0-amd64-CD-1.iso 128K 4.15T 128K /mnt/iohyve/ISO/debian-8.4.0-amd64-CD-1.iso
Storage/iohyve/tvh 10.3G 4.15T 128K /mnt/iohyve/tvh
Storage/iohyve/tvh/disk0 10.3G 4.16T 3.68G -


What's wrong ?

 

[ryanhunt]

Hi All,

Just wanted to post something which others may find useful - how I managed to get a Linux guest (Ubuntu 16.04-LTS) running inside of FreeNAS 9.10 under iohyve/bhyve.

First up, follow the instructions on doc.freenas.org

Except when creating the Linux guest, create as (all iohyve commands from this point on assume you're running as root or via sudo):

iohyve create ubuntu 1G (change 'ubuntu' to whatever you call your VM, and set the appropriate amount of RAM assigned)

Then set parameters:

iohyve set ubuntu loader=grub-bhyve os=d8lvm ram=1G cpu=1

Tthe OS part is important, as Ubuntu defaults to LVM and otherwise barfs on boot, landing you at a grub> loading screen and not booting. Your other alternative is to set the partition within Ubuntu to non-LVM and set os=debian, but I prefer the standard Ubuntu setup so I'd advise setting os=d8lvm.

Download and add the ISO to bhyve's ISO list. To do so, see instructions at doc.freenas.org, but if you're like me and keep a local copy of all OS ISOs I'd recommend using the command 'cpiso' which allows you to add existing ISOs to the bhyve repository, do this by:

iohyve cpiso /mnt/ZFSPOOL/path/to/isos/ubuntu-16.0.etc.etc.iso to save you re-downloading another ISO.

Once this is done, you want to run the install, but this is where I came unstuck. I assumed it would perform a magic silent install - but it doesn't. You'll need to connect a console to it and install as if you were setting up any other VM. Now the console may look a bit messed up, I found the best success by connecting the console first and THEN running the install, so run:

iohyve console ubuntu (suggested from a secondary terminal)

and then run

iohyve install ubuntu (suggested from your main terminal)

Run through the setup via the console, and then when it asks to reboot - it may or may not perform this properly. (In order to disconnect from the console properly, press "~" THEN "~" THEN ".") If it doesn't reboot properly, in your original terminal, run:

iohyve destroy ubuntu
iohyve start ubuntu

Now when my bhyve VM boots up, it gets to the login screen (in the console, see above) and then strangely dumps me back to a grub loader. I don't know what's going on here - but I'm unable to login via the console. Thankfully I installed the OpenSSH server in the options, and by checking out my router I could see the VM's assigned IP address via DHCP and then was able to SSH in.

I'm now up and running, but it's not super straight forward. Hope this helps some people until FreeNAS 10 comes out and it's all done via the GUI :)

 

[rosabox]

I'm trying to run a meteohub VM under iohyve because I have problems with USB ports in VirtualBox: https://forums.freenas.org/index.php?threads/virtualbox-usb-support.44895/

I have converted the vdi image to raw using qemu-img.
I've setup iohyve:

Code:

iohyve setup pool=tank kmod=1 net=igb0

created the guest:

Code:

iohyve create meteohub 32G

set the parameters:

Code:

iohyve set meteohub loader=grub-bhyve
iohyve set meteohub os=debian
iohyve set meteohub ram=512M
iohyve set meteohub cpu=1
iohyve set meteohub net=igb0

checked:

Code:

 iohyve getall meteohub
Getting meteohub iohyve properties...
bargs          -A_-H_-P
boot           0
con            nmdm1
cpu            1
description    Sun Jul 24 10:07:45 CEST 2016
install        no
loader         grub-bhyve
name           meteohub
net            igb0
os             debian
persist        1
ram            512M
size           32G
tap            tap1

copied the raw image to the guest disk:

Code:

dd if=/iohyve/meteohub.img of=/dev/zvol/tank/iohyve/meteohub/disk0

opened a console:

Code:

iohyve console meteohub

tried to start the vm:

Code:

iohyve start  meteohub

and I'm stuck in Grub. I assume I need custom grub.cfg and device.map files but here I'm a bit lost.
I assume I have to create a device.list file in /iohyve/meteohub/ with this content

Code:

(hd0) /dev/zvol/tank/iohyve/meteohub/disk0

but I have no idea what to put in the grub.cfg

Any help please?

Some info about the meteohub vm:

Code:

Linux 2.6.24-etchnhalf.1-486 #1 Thu Feb 25 05:19:29 UTC 2010 i686 GNU/Linux

device.map
(hd0)   /dev/hda

menu.lst
title           Debian GNU/Linux, kernel 2.6.24-etchnhalf.1-486
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.24-etchnhalf.1-486 root=/dev/hda1 ro quiet
initrd          /boot/initrd.img-2.6.24-etchnhalf.1-486
savedefault

title           Debian GNU/Linux, kernel 2.6.24-etchnhalf.1-486 (single-user mode)
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.24-etchnhalf.1-486 root=/dev/hda1 ro single
initrd          /boot/initrd.img-2.6.24-etchnhalf.1-486
savedefault

 

[rosabox]

I'm a step further.
By issuing this commands in grub:

Code:

linux /vmlinuz
initrd /initrd.img
boot

I was able to boot the meteohub vm.

But in the console I got this:

Code:

Failed to emulate instruction [0x87 0x94 0x01 0x00 0xc0 0xff 0xff 0xc3 0xa1 0x28 0x03 0x3d 0xc0 0x85 0xc0] at 0xc0115091
Could not reinit VM ioh-meteohub
Error in initializing VM
bhyve: could not activate CPU 0: Device busy
Could not reinit VM ioh-meteohub
Error in initializing VM
bhyve: could not activate CPU 0: Device busy

and so on ...

in the meteohub console:

Code:

Linux version 2.6.24-etchnhalf.1-486 (Debian 2.6.24-6~etchnhalf.9etch3) (dannf@debian.org) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 Thu Feb 25 05:19:29 UTC 2010
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009ffff (usable)
BIOS-e820: 0000000000100000 - 0000000020000000 (usable)
512MB LOWMEM available.
found SMP MP-table at 000f0000
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   131072
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->   131072
DMI 2.4 present.
ACPI: RSDP 000F2400, 0024 (r2 BHYVE )
ACPI: XSDT 000F2480, 0044 (r1 BHYVE  BVXSDT          1 INTL 20150515)
ACPI: APIC 000F2500, 005A (r1 BHYVE  BVMADT          1 INTL 20150515)
ACPI: FACP 000F2600, 010C (r5 BHYVE  BVFACP          1 INTL 20150515)
ACPI Warning (tbfadt-0230): FADT (revision 5) is longer than ACPI 2.0 version, truncating length 0x10C to 0xF4 [20070126]
ACPI: DSDT 000F2800, 08DE (r2 BHYVE  BVDSDT          1 INTL 20150515)
ACPI: FACS 000F27C0, 0040
ACPI: HPET 000F2740, 0038 (r1 BHYVE  BVHPET          1 INTL 20150515)
ACPI: MCFG 000F2780, 003C (r1 BHYVE  BVMCFG          1 INTL 20150515)
ACPI: PM-Timer IO Port: 0x408
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 6:13 APIC version 16
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0 already used, trying 1
IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Enabling APIC mode:  Flat.  Using 1 I/O APICs
ACPI: HPET id: 0x80860701 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 30000000 (gap: 20000000:e0000000)
swsusp: Registered nosave memory region: 000000000009f000 - 0000000000100000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
Kernel command line: console=ttyS0 BOOT_IMAGE=/vmlinuz
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 8192 bytes)
Detected 2399.768 MHz processor.
Console: colour EGA 80x25
console [ttyS0] enabled
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 511524k/524288k available (1754k kernel code, 12272k reserved, 721k data, 324k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb3000 - 0xfffff000   ( 304 kB)
    vmalloc : 0xe0800000 - 0xfffb1000   ( 503 MB)
    lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
      .init : 0xc036e000 - 0xc03bf000   ( 324 kB)
      .data : 0xc02b68a7 - 0xc036ade4   ( 721 kB)
      .text : 0xc0100000 - 0xc02b68a7   (1754 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 4810.92 BogoMIPS (lpj=9621842)
Security Framework initialized
SELinux:  Disabled at boot.
Capability LSM initialized
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
CPU: L1 I cache: 32K, L1 D cache: 24K
CPU: L2 cache: 1024K
Compat vDSO mapped to ffffe000.
CPU: Intel(R) Atom(TM) CPU  C2550  @ 2.40GHz stepping 08
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126

Any ideas?

 

[rm-r]

hi guys - made this to make it easy - its rough and ready but give you an idea for an ubuntu VM

Code:

#!/bin/bash

#simple script to create a machine in iohyve (ubuntu 16.04 by default) by rm-rf - all care and love - no responsibility

#edit these settings as needed

# download an ISO (if you havent already)
isourl=http://mirror.pnl.gov/releases/16.04/ubuntu-16.04-server-amd64.iso

# Name your machine
name=$(date +"%-H%-M%-S")

#set the disk size
cpu=4

#set the disk size
size=50gb

# set the machine ram
ram=4000mb

# set the OS
os=debian

#set the ISO to use for install
iso=ubuntu-14.04.4-server-amd64.iso

# set auto boot (1=yes 2 = no)
boot=1

#set the bootloader
loader=grub-bhyve


# set the comnsole id (best to not be shared with a running machine)
con=5

# uncomment this if you want to download an ISO to use at the same time

# iohyve fetch $isourl




##############################################################
##### Don't edit below here unless you really want to ########
##############################################################

iohyve list
iohyve isolist

iohyve create $name $size
iohyve set $name cpu=$cpu
iohyve set $name ram=$ram
iohyve set $name os=$os
iohyve set $name boot=$boot
iohyve set $name con=nmdm$con
iohyve set $name loader=$loader

iohyve list
iohyve isolist

echo -e '\e[47;30mnow enter this command in a new ssh window to FreeNAS\e0'
echo iohyve console $name

iohyve install $name $iso

https://github.com/J-C-B/freenas_and_iohyve/blob/master/iohyve_ubuntu.sh

 

[Crolya]

Hi All. Forgive me if this is in the wrong place. I'm not finding much on this topic. I am a semi-noob at this, nursing FreeNAS home boxes since 8.1. Limited Linux experience, but making my way, thanks to support like this forum.

I brought up a new FN9.10 box (Dell Precision T5500, 24GB ECC, USB boot, 500GB SSD disk0, Intel GB NIC, etc.,.) and am very pleased with it so far. I setup bhyve / iohyve following this link for CrashPlan VM. It works much better than plugin as I can be on latest versions of svr and win client.
https://github.com/jadzdotcom/freenas-crashplan-bhyve-centos-
vm
I added one other step, forming iohyve link in /mnt

I'd like to lean on byhve more, to migrate my 9.3 plugin and VM dependencies to byhve from VirtualBox, in preparation for 10, but I seem to be missing some byhve config. The machines seem to be working well, but when I do 'iohyve list', I get nothing but column headings.. same with 'conlist'. However, 'isolist' and 'taplist' do show results, as does 'info'. Any ideas on the config I'm missing?

 

[Spa]

I'm a step further.
By issuing this commands in grub:
....
....

Any ideas?

Did you ever figure it out? I'm making a new iohyve VM for a plex server, and I can't get it to actually install, likely the same issue as you. It keeps getting stuck at grub. I wonder if something in the newer versions of FreeNAS broke something in Iohyve, because Iohyve worked flawlessly for me before when I made new VMs. At least all my old VMs are still up and running without a hitch.

 

[Spa]

Hi All. Forgive me if this is in the wrong place. I'm not finding much on this topic. I am a semi-noob at this, nursing FreeNAS home boxes since 8.1. Limited Linux experience, but making my way, thanks to support like this forum.

I brought up a new FN9.10 box (Dell Precision T5500, 24GB ECC, USB boot, 500GB SSD disk0, Intel GB NIC, etc.,.) and am very pleased with it so far. I setup bhyve / iohyve following this link for CrashPlan VM. It works much better than plugin as I can be on latest versions of svr and win client.
https://github.com/jadzdotcom/freenas-crashplan-bhyve-centos-
vm
I added one other step, forming iohyve link in /mnt

I'd like to lean on byhve more, to migrate my 9.3 plugin and VM dependencies to byhve from VirtualBox, in preparation for 10, but I seem to be missing some byhve config. The machines seem to be working well, but when I do 'iohyve list', I get nothing but column headings.. same with 'conlist'. However, 'isolist' and 'taplist' do show results, as does 'info'. Any ideas on the config I'm missing?

Have you actually made a VM yet? I don't think you'd get a result unless you created a VM.

 

[rosabox]

Did you ever figure it out? I'm making a new iohyve VM for a plex server, and I can't get it to actually install, likely the same issue as you. It keeps getting stuck at grub. I wonder if something in the newer versions of FreeNAS broke something in Iohyve, because Iohyve worked flawlessly for me before when I made new VMs. At least all my old VMs are still up and running without a hitch.

Yes, I was able to fix grub.
But the rest of the problems remained. It's a iohyve problem, caused by the very old Linux version, Meteohub is using.

 

[Crolya]

Have you actually made a VM yet? I don't think you'd get a result unless you created a VM.

Yes. As in my post, I have two VMs running perfectly, one Crashplan/CentOS and one Ubuntu Svr 16.04. They just don't show up in any of the iohyve lists except 'info.'