FreeNAS 9.2.0 Ubuntu 13.04 Jail Issues

[Tyler]

So, I was extremely excited when I heard FreeNAS had gained the support for Linux Jails. But, after I had created the jail, I was unable to do anything. Inside of the jail, I am without bash and I am unable to resolve hostnames to retrieve software from the repo, and to make matters worse, utilities like ping and ifconfig throw up errors. From outside of the jail, I am able to ping the jails IP Address from another machine on the LAN. All while my FreeNAS network configuration is fine.

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-backports/un iverse/binary-i386/Packages Something wicked happened resolving 'us.archive.ubu ntu.com:http' (-11 - System error) W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-backports/mu ltiverse/binary-i386/Packages Something wicked happened resolving 'us.archive.u buntu.com:http' (-11 - System error) E: Some index files failed to download. They have been ignored, or old ones used instead. # ping 8.8.8.8 ping: icmp open socket: Operation not permitted # sudo chmod u+s 'which ping' sudo: unable to resolve host rTorrent chmod: cannot access ‘which ping’: No such file or directory # which ping /bin/ping # sudo chmod u+s /bin/ping sudo: unable to resolve host rTorrent # ifconfig bridge: error fetching interface information: Invalid argument #


"rTorrent" is the name of the Ubuntu 13.04 jail.

 

[dlavigne]

This looks related to https://bugs.freenas.org/issues/3759.

 

[Scharbag]

I read through issue 3759, and while not having ping etc. is a PITA, it does not mention on how to fix the resolving issues. I am in the same boat that I cannot add packages/programs as the Ubuntu jail cannot seem to resolve the outside world. The jail itself can be pinged from another computer so it does have an interface that works. I tried editing the /etc/network/interfaces but nothing will resolve.

Any help would be awesome.

Cheers,

 

[dlavigne]

As described in the Guide, networking must be configured from the Jail's Edit screen in the GUI. For Linux jails, command line networking utils from within the jail itself will not work as these tools assume the Linux networking stack while the jail really uses the FreeBSD networking stack.

If the jail is correctly configured from the GUI and the FreeNAS system itself has no problem accessing the Internet, it sounds like we need to wait until the dev figures out what the underlying bug is and fixes it in that ticket.

Has anyone tried to see if this issue is the same in the other Linux jails or does it appear to be isolated to just Ubuntu?

 

[freenas4n00b]

I spinned up a centos 6.4 jail. Networking tools doesnt work (expected based on feedback), i was able to use yum to update the jails packages (which proves the networking stack is kind of working).

 

[MaIakai]

I spinned up a centos 6.4 jail. Networking tools doesnt work (expected based on feedback), i was able to use yum to update the jails packages (which proves the networking stack is kind of working).

Dude How

I just spun up a CentOS jail, My Jail/Prison is on my ZFS pool with plenty of space.

Yum update fails due to hard-drive space
Yum install anything (regardless of how small in size) fails due to hard-drive space on /

df -h shows I have 3.7T Free

Transaction Check Error:
installing package apr-1.3.9-5.el6_2.i686 needs 319KB on the / filesystem

[root@AEMonitor usr]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1.7T 714M 3.7T 1% /
sysfs 4.0K 4.0K 0 100% /sys
devpts 1.0K 1.0K 0 100% /dev/pts
df: `/dev/shm': No such file or directory
/dev/sda1 3.7T 714M 3.7T 1% /boot
df: `/proc/sys/fs/binfmt_misc': No such file or directory

 

[warri]

Dude How
Yum update fails due to hard-drive space
Yum install anything (regardless of how small in size) fails due to hard-drive space on /

df -h shows I have 3.7T Free

Here is the bug report with a workaround: https://bugs.freenas.org/issues/3668

 

[MaIakai]

Here is the bug report with a workaround: https://bugs.freenas.org/issues/3668

Thank you! No ammount of googlefu helped me find this issue. Didn't think to check the bugs page.

Added diskspacecheck=0 to /etc/yum.conf and all is well now.
looks like I had diskspace=0


Keywords to help other people searching for this issue :
CentOS freespace / yum fails linux jail diskspace

 

[freenas4n00b]

I would still like to figure out how to fix the ubuntu 13.04 issue

 

[perlguy9]

Yes, Ubuntu 13.04 jail seems broken. Networking may be working, but you can't start any services due to this error with upstart:

Code:

root@ubuntu:/etc# service ssh start
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: No such file or directory

and DNS resolution seems broken?

 

[david kennedy]

So, I was extremely excited when I heard FreeNAS had gained the support for Linux Jails. But, after I had created the jail, I was unable to do anything. Inside of the jail, I am without bash and I am unable to resolve hostnames to retrieve software from the repo, and to make matters worse, utilities like ping and ifconfig throw up errors. From outside of the jail, I am able to ping the jails IP Address from another machine on the LAN. All while my FreeNAS network configuration is fine.

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-backports/un iverse/binary-i386/Packages Something wicked happened resolving 'us.archive.ubu ntu.com:http' (-11 - System error) W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-backports/mu ltiverse/binary-i386/Packages Something wicked happened resolving 'us.archive.u buntu.com:http' (-11 - System error) E: Some index files failed to download. They have been ignored, or old ones used instead. # ping 8.8.8.8 ping: icmp open socket: Operation not permitted # sudo chmod u+s 'which ping' sudo: unable to resolve host rTorrent chmod: cannot access ‘which ping’: No such file or directory # which ping /bin/ping # sudo chmod u+s /bin/ping sudo: unable to resolve host rTorrent # ifconfig bridge: error fetching interface information: Invalid argument #


"rTorrent" is the name of the Ubuntu 13.04 jail.

I was also pretty interested to see linux jails in 9.2. I then tried both Ubuntu and CentOS and hit the issues you found.

This got me wondering if i could get the software i needed running in a native freebsd jail instead. It took a bit more time but i do have it running (its a web app based on php/mysql/apache).

There are a number of ports and packages available to help and might be a way to avoid the linux jails?

 

[MaIakai]

confirmed on Ubuntu, can't do anything.
Nameservers/dns does not work, networking utilities are broken, can't install, can't change anything.

As for CentOS, managed to install nagios and apache, can't start them because of Epoll errors.

 

[MaIakai]

OpenSUSE same deal

Jan 1 11:58:16 freenas kernel: linux: pid 96635 (gpg2): syscall prlimit64 not implemented
Errors thrown about when doing anything install/update related with YAST

Inside the jail you get
warning: Failed to read auxiliary vector, /proc not mounted?

------------------------
Debian same thing as ubuntu(No real surprise), but it seems to work somewhat better

Jan 1 12:29:06 freenas last message repeated 356 times
Jan 1 12:29:06 freenas kernel: linux: pid 7322 (dpkg): syscall sync_file_range not implemented
Jan 1 12:29:06 freenas kernel: linux: pid 7322 (dpkg): syscall utimensat not implemented

Apt-get upgrade works for 80% of the items, i do see a lot of errors in the console, especially towards the end. looking at version numbers though some things did update.

Apt-get install -y php5 (after apt-get upgrade) works, somewhat, still too many errors to trust the installation.

--------------------

since I'm seeing so many errors in freenas console itself I'm going to guess that no ammount of tinkering with the jail will fix these problems. I have been looking at recompiling things in my CentOS jail but think I'll just give up for now.

Maybe Freenas 9.4

 

[mattyman]

Not sure if the OP has worked out what they need to do to get internet connectivity going in Ubuntu so I thought I'd post here.
If I get anything wrong be gentle ... I'm a Windows guy that has kind of been playing around with FreeNAS over the last few weeks. I don't know much ... but I'm pretty good at Google. :)

From what I can tell the issue that the opt is having is that DNS that isn't resolving in the Ubuntu jail. I was having exactly the same issue.

First I double checked that adding a static interface to the interfaces file wouldn't work (I know it's not supposed to in Jails).

I had tried apt-get update earlier and it failed as it did with the OP.
There are two addresses that apt-get tries to download from:

security.ubuntu.com
us.archive.ubuntu.com

I pinged these from a computer which has working DNS and received replies from:

us.archive.ubuntu.com 91.189.91.14
security.ubuntu.com 91.189.92.201

I just added these to the /etc/hosts file (and renamed the hostname so that was correct too).

Code:

127.0.0.1 localhost
 
# changed hostname here
127.0.1.1 ubuntu_1
 
#Added these two lines for testing.
91.189.91.14 us.archive.ubuntu.com
91.189.92.201 security.ubuntu.com
 
# The following lines are desirable for IPv6 capable hosts
::1    ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allroutersuntu.com

I then ran apt-get update again and it worked properly which means that it's definitely a DNS issue. Network connectivity is there, but Ubuntu just doesn't know where anything is. I had no idea where the DNS settings are stored in Linux so I Googled until I found out that they are stored in /etc/resolv.conf.

After some more digging ... this looks like it's an Ubuntu bug https://bugs.launchpad.net/ubuntu/ source/resolvconf/ bug/1000244.
EDIT: Actually re-reading that bug description during during the day (and not a 1am in the morning) it seems that if it is related it's not conclusive. I think I read this part and got overly excited, "Based on https://help.ubuntu.com/12.04/serverguide/network-configuration.html#name-resolution, I tried establishing this symlink myself, but the file /run/resolvconf/resolv.conf does not exist on my system."

The issue.
/etc contains a file called resolv.conf which normally lists the domain and DNS server addresses. In the Jail version of Ubuntu the file is Symlinked to ../run/resolvconf/resolv.conf which doesn't exist.

I could not find a run directory but for all I know this could be some sort of weird "executable" Linux thing that I haven't come across before. I do have a /etc/resolveconf directory but that doesn't contain the resolv.conf file that the symlink is pointing to (it does contain a resolv.conf.d file though - I haven't looked at that either).

Anyway I ended renaming the symlink file to resolv.conf.old and copied the resolv.conf file over from the FreeNAS /etc folder.
I rolled back the changes to the hosts file and rebooted the server. apt-get update now works fine - still can't ping though.

Solution.
Open a terminal session logged in as root in FreeNAS then run the following commands.

Code:

[root@freenas] /# cd /mnt/drivename/jails/ubuntu_1/etc
[root@freenas] /mnt/drivename/jails/ubuntu_1/etc# mv resolv.conf resolv.conf.old
[root@freenas] /mnt/drivename/jails/ubuntu_1/etc# cp /etc/resolv.conf ./resolv.conf

Feel free to pick apart anything I've stated above. If I'm wrong about anything, I'd like to know.

 

[cyberjock]

You are welcome to link to anything you want. You have nothing to fear with links unless you link to crap ads, adult material, etc. Then we'll delete it(obviously).

 

[mattyman]

No worries.

I'll edit my post above.

 

[mattyman]

Fix DNS and other apt-get installation problems.
So I was playing around a bit more with the Ubuntu server tonight and tried installing some packages using apt-get. The packages would install but the config wouldn't automatically configure due to the fact that /var/run didn't exist. I had a look at where this folder should've been and wouldn't you know it, it was symlinked back to the same non-existent /run directory that resolv.conf was pointing to.
In this post I've basically revised what I posted earlier and fixed the apt-get issues I was having.

Apologies if I flick between the terms "folder" and "directories". It took me ages to get out of the MS-DOS habit of calling directories "folders" for Windows and now I find myself trying to change back.


APT-GET
Open a terminal session and login to your Ubuntu jail as root and then run these commands.
# cd /
# mkdir run
# chmod 777 run

Note: I’m not sure what permissions the run folder is supposed to have – if anyone knows let me know and I’ll update this.

Now if you navigate to /var you will notice that there is a symbolic link there to run which points back to /run. That link is needed for apt-get and was broken until we just created that /run folder.

Some packages also require logon information to be stored in /var/run/utmp.
# touch /var/run/utmp
# chmod 664 /var/run/utmp


DNS
For those that followed my earlier post. Run these commands and then follow along below skipping the command where I have marked the # in RED. For those that didn't action my previous post please skip these two commands.
# mv /etc/resolv.conf /etc/resolvconf
# mv /etc/resolv.conf.old /etc/resolv.conf

You may or may not have noticed that your computer currently cannot access the internet. This is because DNS hasn’t been setup correctly. Run the two commands below.

# cd /etc
# ls –l ./resolv*

This should bring up a symlinked fileand a folder.

resolv.conf -> ../run/resolvconf/resolv.conf

resolvconf:
total 31K
-rw-r--r-- 1 root root 172 Dec 13 2012 interface-order
drwxr-xr-x 2 root root 4 Aug 21 12:21 resolv.conf.d
drwxr-xr-x 2 root root 4 Aug 21 12:21 update.d

The symlinked file will look a little something like this:
resolv.conf -> ../run/resolvconf/resolv.conf

Now, if we look at the file that the symlink is pointing to we can see that it’s pointing back to the /runfolder that we created in the earlier steps. You will also know that the /resolveconf folder doesn’t exist in the /rundirectory.

Rather than copy or move the directories from one to the other we’ll symlink the /etc/resolveconf so that it appears in the /run directory.

# cd /run
# ln -s /etc/resolvconf /run/resolvconf

The folders will now be symlinked together. If you run the following command you should get the output below.

# ls -l
total 1K
lrwxr-xr-x 1 root root 15 Jan 6 08:13 resolvconf -> /etc/resolvconf

The last thing that we need to do is use terminal to login as root to FreeNAS and then copy the DNS settings from /etc/resolv.conf on FreeNAS to the Ubuntu jail under /etc/resolvconf.

# cp /etc/resolv.conf /mnt/<path to jail folder>/ubuntu_1/etc/resolvconf

Where <path to jail folder> is the directory structure to your jail folder and ubuntu_1 is the name of your Ubuntu jail.

Lastly restart the Ubuntu jail in FreeNAS.

If I come across anymore issues I'll try to fix them and detail that here too.

It really does look like the folder structure in the Ubuntu jail was borked.

 

[mattyman]

Yes, Ubuntu 13.04 jail seems broken. Networking may be working, but you can't start any services due to this error with upstart:

Code:

root@ubuntu:/etc# service ssh start
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: No such file or directory

and DNS resolution seems broken?

With regards to the Upstart issue there's a nice little solution here - http://mrzard.github.io/blog/2011/02/01/failed-to-connect-to-socket-comubuntuupstart/
I've read conflicting posts about whether it actually stays between reboots or not (some even say that it will cause Ubuntu not to start) ... I guess I'll find out when I reboot. I can say that it does bypass the Upstart issue and packages will install without the problem.

 

[mattyman]

Also if you install packages you may get fopen: Permission denied messages. This is to do with the package trying to update the man when it doesn't have access. Can be fixed with the following line.

# chown -R man:root /var/cache/man​

​

Source: http://blog.bytetouch.com/linux/apt-get-fopen-permission-denied/

 

[david kennedy]

Curious, what are you trying to get running on a Ubuntu jail, and why not focus on getting it running on a native FreeBSD jail instead?