Virtualbox jail not running after virtualbox-ose-kmod updated

[Lee Spangler]

If anyone has a step by step method of doing this, I would appreciate it. I can't get anything to properly install in bhyve (it just kicks me to the shell). I miss phpvirtualbox. I checked the package and it looks like the system can still support 5.1.26.

 

[rdmitry0911]

Hi,
is it a good idea to install latest kmod drivers into freenas host? latest virtualbos-ose-kmod version was compiled with the latest depended packages. and also latest kernel, to which it is integrated as it runs as a kernel module.

I was working on other solution - to compile the same version of virtualbox-ose in a jail, as it is now installed in Freenas 11.0-RC4 system. Currently it is virtualbox-ose-kmod 5.1.6, which matches revision 423390 of 5.1.6_1 version of virtualbox-ose.

Here are the steps and shell script you can use for automated build in a jail. Just create or copy vboxcreate.sh script into new 11.0 jail and execute it. Take a coffe, complilation takes more hours..

Steps to create new virtualbox jail
===================================
Jails -> Add Jail -> "virtualbox"

jexec virtualbox

vi vboxcreate.sh

#!/bin/sh
echo Installing virtualbox
rm -rf /usr/ports
# workaround for svn problem without ipv6 connectivity to use ipv4 svn.freebsd.org address
host svn.freebsd.org | awk '/has address/ { print $4 " svn.freebsd.org"; exit }' >> /etc/hosts
svn checkout --non-interactive --trust-server-cert -r 423390 https://svn.freebsd.org/ports/head /usr/ports
sed -i -e '/svn.freebsd.org/d' /etc/hosts
export BATCH=yes
# compilation of virtualbox-ose fails without bulding python3 port first
cd /usr/ports/lang/python3 && make install clean
cd /usr/ports/emulators/virtualbox-ose && make install clean
pkg -o ASSUME_ALWAYS_YES=YES install phpvirtualbox apache24 mod_php56
pkg -o ASSUME_ALWAYS_YES=YES clean
pw groupadd -n vbox -g 1001
echo "VBox123" | pw useradd -n vbox -c vbox -u 1001 -s /bin/sh -m -g vbox -G vboxusers -d /home/vbox -h -
sed -i -e 's/'\''pass'\'';/'\''VBox123'\'';/' /usr/local/www/phpvirtualbox/config.php
sed -i -e 's/apache24\/data/phpvirtualbox/g' /usr/local/etc/apache24/httpd.conf
sed -i -e '/AddType application\/x-gzip \.gz \.tgz/a\
\
AddType application\/x-httpd-php .php\
AddType application\/x-httpd-php-sources .phps\
' /usr/local/etc/apache24/httpd.conf
sed -i -e 's/#ServerName www.example.com:80/ServerName 127.0.0.1:80/' /usr/local/etc/apache24/httpd.conf
sed -i -e 's/@constant('\''PHPVBOX_VER'\'')/"5.1-0"/' /usr/local/www/phpvirtualbox/endpoints/api.php
sysrc apache24_enable="YES"
sysrc vboxwebsrv_user="vbox"
sysrc vboxwebsrv_enable="YES"
service vboxwebsrv start
service apache24 start
echo Done. Connect to phpvirtualbox at http://`ifconfig -a | grep inet | grep -v inet6 | grep -v 127.0.0.1 | awk '{print $2}'` with user "admin" password "admin"

sh vboxcreate.sh

In FreeNAS web interface
========================
Add Tunable, type "Loader":
vboxdrv_load="YES"
Add Tunable, type "rc.conf":
vboxnet_enable="YES"

and reboot to load vbox drivers.

I haven't tested it fully yet, as I don't have Freenas 11 on physical machine. Only running under other virtualbox and as it does not allow nested virtualisation, I cannot check it if it really works.

Please report back if you try it if it works or you suggest some changes.

Enjoy!

Hi Kam, thank you for the great job. I used your approach in freenas 11.0 a while ago and everything is still working fine. Now I'm trying to make the same on freenas 11.2, but can't make it run. What I do is this:

• I check the version of vbox driver in freenas:

Code:

root@freenas[~]# pkg info | grep -i virtualbox														
virtualbox-ose-kmod-5.2.8_1	VirtualBox kernel module for FreeBSD
root@freenas[~]#

• I make a jail with 11.2-release template and as the /sys in jail is empty I have to fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/src.txz and install it otherwise I have a VirtualBox compile error: No rule to make target ‘/sys/kern/bus_if.m’

Code:

tar -C / -xvf src.txz

• I install subversion

Code:

pkg install subversion

• Here I find which virtualbox revision corresponds to the freenas drivers version. In my case driver version is 5.2.8_1 which corresponds to the revision 464084 so, I fill ports directory with this command:

Code:

svn checkout --non-interactive --trust-server-cert -r 464084 https://svn.freebsd.org/ports/head /usr/ports

• Then I do everything according to your vboxcreate.sh file. The only thing is that 464084 revision of virtualbox uses php 7.1 and I need mod_php71 for that.

With these corrections the installation goes without any errors and I can see phpvirtualbox interface and can create a virtual machine. However, when I try to run it, I have an error:

Code:

Exception Object
(
	[message:protected] => The virtual machine 'test' has terminated unexpectedly during startup with exit code 1 (0x1)
	[string:Exception:private] =>
	[code:protected] => 0
	[file:protected] => /usr/local/www/phpvirtualbox/endpoints/lib/vboxconnector.php
	[line:protected] => 2429
	[trace:Exception:private] => Array

in /var/log/messages I see nothing strange.
but when I try to start vm from console, I have this:

Code:

vbox@Freenas-11_2:/ % /usr/local/lib/virtualbox/VBoxHeadless --comment test --startvm 192d1bc1-47d5-45c4-a70c-a69484286c46 --vrde config
VBoxHeadless: Error -1908 in suplibOsInit!
VBoxHeadless: Kernel driver not installed
VBoxHeadless: Tip! Make sure the kernel module is loaded. It may also help to reinstall VirtualBox.
vbox@Freenas-11_2:/ % 

However, the driver is loaded:

Code:

root@freenas[~]# kldstat|grep vb 
 6	3 0xffffffff8283c000 89508	vboxdrv.ko
26	2 0xffffffff82e2e000 2ae0	 vboxnetflt.ko
29	1 0xffffffff82e3d000 3f70	 vboxnetadp.ko
root@freenas[~]# 

What's wrong with my setup?

Thank you, Dmitry

 

[Kam]

Hi,
are you using iocage or warden for 11.2 jails?

I think iocage is hiding /dev devices, which is on one side good but can make problems here.
On my test 11.2-RC2 system I see vbox devices:

root@freenas3:~ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

But in jail there are not:
root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
ls: No match.

I think we need to find out how to unhide /dev/vb* devices in a virtualbox jail.
Kam

 

[Kam]

Hi,
Check this:

root@freenas3:~ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
ls: No match.
root@test:/ # exit

root@freenas3:~ # devfs -m /mnt/tpool/iocage/jails/test/root/dev rule apply path "vbox*" unhide

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

But I don't know how to automate unhiding of vbox drivers automatically in iocage on jail startup.
I haven't tested if it will work after unhide, as I have 11.2 only running under virtualbox, which till now does not support nested virtualisation to test running nested vm.

Kam

 

[rdmitry0911]

Hi,
Check this:

root@freenas3:~ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
ls: No match.
root@test:/ # exit

root@freenas3:~ # devfs -m /mnt/tpool/iocage/jails/test/root/dev rule apply path "vbox*" unhide

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

But I don't know how to automate unhiding of vbox drivers automatically in iocage on jail startup.
I haven't tested if it will work after unhide, as I have 11.2 only running under virtualbox, which till now does not support nested virtualisation to test running nested vm.

Kam

You're right, I'm using iocage and making dev tree visible inside the jail allowed me to start the machine, but just after the start it crashes the freenas host. In fact, for these tests I use my lab config, where I run freenas inside proxmox 5.2 with nested virtualization enabled. To check if this configuration work, I first installed ubuntu guest in proxmox host and then installed virtualbox inside this ubuntu. I can create and run vms under this nested virtualbox without problems. So, this freenas crash is not related to the configuration. I think, it is a kind of incompatibility between virtualbox and freenas.
As for automation. there is a parameters in jail properties: mount.devfs which "mount a devfs filesystem on the chrooted /dev directory and apply the ruleset in the devfs_ruleset parameter to restrict the devices visible inside the jail." I think with the help of this parameter it is possible to make visible all needed devices, after we find a way how to run virtual machines without crashing the host.
I managed to find some debug information (attached) which says it is vboxdrv <--> vmm problem:

Code:

vboxdrv: XXXXXXXXXXXXXXXX VMMR0.r0
vboxdrv: XXXXXXXXXXXXXXXX VBoxDDR0.r0
kernel trap 9 with interrupts disabled

Fatal trap 9: general protection fault while in kernel mode
cpuid = 6; apic id = 06
instruction pointer	 = 0x20:0xfffffe04698668b3
stack pointer		   = 0x28:0xfffffe046978f678
frame pointer		   = 0x28:0xfffffe046978f700
code segment			= base 0x0, limit 0xfffff, type 0x1b
						= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags		= resume, IOPL = 0
current process		 = 4908 (VBoxHeadless)

 

[rdmitry0911]

Hi,
Check this:

root@freenas3:~ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
ls: No match.
root@test:/ # exit

root@freenas3:~ # devfs -m /mnt/tpool/iocage/jails/test/root/dev rule apply path "vbox*" unhide

root@freenas3:~ # iocage exec test
root@test:/ # ls -la /dev/vb*
crw------- 1 root wheel 0x38 Nov 22 19:44 /dev/vboxdrv
crw-rw-rw- 1 root wheel 0x39 Nov 22 19:44 /dev/vboxdrvu
crw------- 1 root wheel 0x66 Nov 22 21:46 /dev/vboxnetctl

But I don't know how to automate unhiding of vbox drivers automatically in iocage on jail startup.
I haven't tested if it will work after unhide, as I have 11.2 only running under virtualbox, which till now does not support nested virtualisation to test running nested vm.

Kam

I checked again the configuration. I installed freenas 11.0 under proxmox and built phpvirtualbox in a jail. Exactly the same version I have running on my fileserver at the moment. It also crashes when I start vm. So, virtualbox can run under linux under proxmox, but can't run under freenas under proxmox. Next step I have to find a spare host to check freenas 11.2+jail+phpvirtualbox on a bare metal.

 

[rdmitry0911]

I checked again the configuration. I installed freenas 11.0 under proxmox and built phpvirtualbox in a jail. Exactly the same version I have running on my fileserver at the moment. It also crashes when I start vm. So, virtualbox can run under linux under proxmox, but can't run under freenas under proxmox. Next step I have to find a spare host to check freenas 11.2+jail+phpvirtualbox on a bare metal.

Today I did another test. This time in my proxmox host I rolled out a guest with freenas 9.3 release and configured a jail inside with virtualbox 4.3.12 template. Inside this vbox I successfully started ubuntu 18.04.1 x64 desktop. This result screwed me up a little bit. Can anybody explain me why the same virtual machine can run inside virtualbox 4.3.12 inside freenas 9.3 inside proxmox 5.2 and crashes the freenas host when it is running inside virtualbox 5.1.6 inside freenas 11.0 inside the same proxmox 5.2? On top of that, the same configuration of virtualbox 5.1.6 inside freenas 11.0 rolled out on a bare metal runs this vm without any problems.

 

[Tsaukpaetra]

run inside virtualbox 4.3.12 inside freenas 9.3 inside proxmox 5.2 and crashes the freenas host

My guess is some incompatibility with nested virtualization. But other than that, not sure.

 

[rdmitry0911]

Today I did another test. This time in my proxmox host I rolled out a guest with freenas 9.3 release and configured a jail inside with virtualbox 4.3.12 template. Inside this vbox I successfully started ubuntu 18.04.1 x64 desktop. This result screwed me up a little bit. Can anybody explain me why the same virtual machine can run inside virtualbox 4.3.12 inside freenas 9.3 inside proxmox 5.2 and crashes the freenas host when it is running inside virtualbox 5.1.6 inside freenas 11.0 inside the same proxmox 5.2? On top of that, the same configuration of virtualbox 5.1.6 inside freenas 11.0 rolled out on a bare metal runs this vm without any problems.

I've just found the mistake. I forgot to unload nmdm and vmm before starting virtualbox jail. In my production server it is done automatically in jail-pre-start file and I was pretty sure I have transferred the configuration to my testing platform correctly. It turned out it was not. After unloading these modules everything is OK. I have ubuntu x64 running in virtualbox 5.2.8_1 inside freenas 11.2 inside proxmox 5.2 The case is closed.
Thanks to all, Dmitry