After Upgrade to FreeNAS 11: Jails stop working

[svbito]

Hi all,

after upgrading FreeNAS to 11 a few days ago, I can't create new jails. It gives the following error:

Code:

x ./ x ./dev/ x ./boot/ x ./var/ x ./.profile x ./tmp/ x ./mnt/ x ./sys x ./proc/ x ./COPYRIGHT x ./root/ x ./rescue/ x ./lib/ x ./libexec/ x ./sbin/ x ./media/ x ./etc/ x ./.cshrc x ./bin/ x ./usr/ x ./usr/tests/ x ./usr/ports/ x ./usr/libdata/ x ./usr/libexec/ x ./usr/bin/ x ./usr/share/ x ./usr/src/ x ./usr/obj/ x ./usr/include/ x ./usr/lib32/ x ./usr/local/ x ./usr/lib/ x ./usr/sbin/ x ./usr/sbin/devinfo x ./usr/sbin/ypserv x ./usr/sbin/sysrc x ./usr/sbin/snapinfo x ./usr/sbin/apm x ./usr/sbin/zzz x ./usr/sbin/hcseriald x ./usr/sbin/uathload x ./usr/sbin/rrenumd x ./usr/sbin/callbootd x ./usr/sbin/ndis_events x ./usr/sbin/rmextattr x ./usr/sbin/sdpd x ./usr/sbin/rmt x ./usr/sbin/boot0cfg x ./usr/sbin/pwd_mkdb x ./usr/sbin/etcupdate x ./usr/sbin/smbmsg x ./usr/sbin/mlxcontrol x ./usr/sbin/pstat x ./usr/sbin/chkgrp x ./usr/sbin/acpidump x ./usr/sbin/yppoll x ./usr/sbin/fwcontrol x ./usr/sbin/mixer x ./usr/sbin/syslogd x ./usr/sbin/chown x ./usr/sbin/ntpd x ./usr/sbin/l2ping x ./usr/sbin/spkrtest x ./usr/sbin/chroot x ./usr/sbin/mprutil x ./usr/sbin/cron x ./usr/sbin/crashinfo x ./usr/sbin/tcpdrop x ./usr/sbin/unbound-checkconf x ./usr/sbin/cdcontrol x ./usr/sbin/flowctl x ./usr/sbin/setfib x ./usr/sbin/vigr x ./usr/sbin/sicontrol x ./usr/sbin/pkg x ./usr/sbin/getextattr x ./usr/sbin/blacklistctl x ./usr/sbin/quotaon x ./usr/sbin/ntptime x ./usr/sbin/ftp-proxy x ./usr/sbin/bhyveload x ./usr/sbin/tcpdump x ./usr/sbin/moused x ./usr/sbin/unbound-anchor x ./usr/sbin/ctm_smail x ./usr/sbin/hoststat x ./usr/sbin/hostapd_cli x ./usr/sbin/nfscbd x ./usr/sbin/ctm_dequeue x ./usr/sbin/dconschat x ./usr/sbin/watch: truncated bzip2 input tar: Error exit delayed from previous errors. ERROR: missing files cannot open 'harbour/jails/.warden-template-Test': dataset does not exist find: /mnt/harbour/jails/.warden-template-Test: No such file or directory ERROR: Failed to create ZFS base dataset

This is with the custom template "Test", but the same message appears if I don't use a template, too. Interesting thing: Creating the jails, the warden creates a dataset "harbour/jails/.warden-template-*templatename", then it takes a few minutes, then it gets deleted (?!) and then the jail creation fails.

Any ideas on how to debug this?

 

[svbito]

I already tried a few things: I moved the jails to another pool, that had no effect. I recreated the dataset - also no effect. I added the -standard template by hand - that got removed when I tried to create a jail. I'm getting confused. Are Jails in FN11 not working correctly?

 

[dlavigne]

Anything in /var/log/messages after you try to create a jail?

 

[svbito]

no, absolutly nothing. Only the Error on the Interface. If that matters: I tried both the new and the old interface. Same behaviour.

In the meantime, I got some jails setup on the console using iocage. But I would really like the ease of use of the interface - espacially with the mounts inside the jails.

 

[SweetAndLow]

Create a new dataset then point your jail root at that dataset. This will force the redownload of the correct template. If this doesn't work you probably have a networking problem.

You should also provide all your hardware specs for your system.

Sent from my Nexus 5X using Tapatalk

 

[svbito]

I already tried using another dataset. I used another one, removed the old one, created a new one - all to no avail. I redownloaded the template several times - even created a template by hand. All of this worked.

iocage had no problems downloading the .tgz's it needs. DNS works, manual fetch of the tgz's work, too. Somehow the warden just does not seem to work.

I'm using a HP MicroServer N54l - probably an often used hardware.

for reference:

Code:

root@fidelity:~ # fetch -o /tmp/pluginjail.tgz https://download.freenas.org/jails/11/x64/freenas-pluginjail-11.0-RELEASE.tgz
/tmp/pluginjail.tgz						   100% of  299 MB   31 MBps 00m10s

altough I notice that the initial connections seems to take quite a bit of time:

Code:

root@fidelity:~ # time fetch -o /tmp/pluginjail.tgz https://download.freenas.org/jails/11/x64/freenas-pluginjail-11.0-RELEASE.tgz
/tmp/pluginjail.tgz						   100% of  299 MB   29 MBps 00m11s
6.065u 0.599s 1:25.55 7.7%	30+167k 0+0io 0pf+0w

 

[SweetAndLow]

I already tried using another dataset. I used another one, removed the old one, created a new one - all to no avail. I redownloaded the template several times - even created a template by hand. All of this worked.

iocage had no problems downloading the .tgz's it needs. DNS works, manual fetch of the tgz's work, too. Somehow the warden just does not seem to work.

I'm using a HP MicroServer N54l - probably an often used hardware.

for reference:

Code:

root@fidelity:~ # fetch -o /tmp/pluginjail.tgz https://download.freenas.org/jails/11/x64/freenas-pluginjail-11.0-RELEASE.tgz
/tmp/pluginjail.tgz						   100% of  299 MB   31 MBps 00m10s

altough I notice that the initial connections seems to take quite a bit of time:

Code:

root@fidelity:~ # time fetch -o /tmp/pluginjail.tgz https://download.freenas.org/jails/11/x64/freenas-pluginjail-11.0-RELEASE.tgz
/tmp/pluginjail.tgz						   100% of  299 MB   29 MBps 00m11s
6.065u 0.599s 1:25.55 7.7%	30+167k 0+0io 0pf+0w

What are the permissions of your root dataset and jail dataset? Can you screen shot them?

Sent from my Nexus 5X using Tapatalk

 

[svbito]

Sorry for the late reply. I was out of the office for a few days. Here are the screenshots:

And the console output:
root:

Code:

drwxr-xr-x   5 root  wheel	5 Jul 14 15:21 harbour

jails:

Code:

drwxr-xr-x  5 root  wheel	6 Jul 17 17:31 jails

If I understand it correctly, If I'm creating a new dataset for jails, the permissions should be set correctly?

 

[svbito]

After a lot of fiddeling this caught my eye again:

altough I notice that the initial connections seems to take quite a bit of time:

Code:

root@fidelity:~ # time fetch -o /tmp/pluginjail.tgz https://download.freenas.org/jails/11/x64/freenas-pluginjail-11.0-RELEASE.tgz
/tmp/pluginjail.tgz						   100% of  299 MB   29 MBps 00m11s
6.065u 0.599s 1:25.55 7.7%	30+167k 0+0io 0pf+0w

Turns out: Something went wrong with IPv6 autoconfigure and Python does not seem to handle that correctly. fetch for example just falls back to IPv4 and works. Python does not. Disabling IPv6 makes the jails work again :).

Thanks for your help everyone!

 

[dlavigne]

Thanks for the update. Looks like that is on track for being fixed in 11.1: https://bugs.freenas.org/issues/24607.