Jail inaccessible after upgrade

[Glorious1]

Upgrading a jail to 13.1 seemed to go well, but lately I found it is actually broken. Nothing works in the jail. The WebGUI shows it runs and all seems well. But when I try to SSH into it or use the GUI shell, I get the following output. The shell then gives no prompt and the screen is unresponsive. SSH fails to enter the jail. Here is SSH:

Code:

Tabernacle:~$ sudo iocage console transmission
Password:
Last login: Thu Aug 11 08:07:49 on pts/1
FreeBSD 13.1-RELEASE n245376-eba770b30ff TRUENAS

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

To change this login announcement, see motd(5).
ld-elf.so.1: Shared object "libncurses.so.8" not found, required by "bash"
Tabernacle:~$

This jail is very complex to set up and I'd like to avoid starting from scratch. Any thoughts how to fix it? Seems the libncurses.so.8 file might be key.

 

[homer27081990]

Try to find the freebsd folder containing shared stuff and place a copy of libncurses.so (version 8) inside. There is now a version 9, probably got replaced by the upgrade. This is just intuitive, though, I really know nothing about jails.

 

[Glorious1]

Try to find the freebsd folder containing shared stuff and place a copy of libncurses.so (version 8) inside. There is now a version 9, probably got replaced by the upgrade. This is just intuitive, though, I really know nothing about jails.

Thanks, that makes sense (assuming I could find such a file). Based on something I found googling, I put a symbolic link in one of the folders that house such files, linking from libncurses.so.8 to another so.9 version, but then trying to enter the jail led to another error that another so file was missing. So I think it won't be that simple.

 

[Samuel Tai]

You can always "upgrade" the jail back to the previous version.

 

[Jailer]

What is "Tabernacle"? Is that your username? Are you trying to log in via a SSH client or the GUI?

The tilde dollar sign is not anything I see in my console when trying to log in via SSH either by username or root. The tilde denotes that you are logged in to your home directory so you need to provide a bit more information on what you are trying to do and start from the beginning please.

ETA: So I did some more research. The dollar sign ($) indicates a bash shell. So yeah, provide some more information on your login procedure please because it looks like you are doing something non standard and clarification would be helpful here.

 

[homer27081990]

Thanks, that makes sense (assuming I could find such a file). Based on something I found googling, I put a symbolic link in one of the folders that house such files, linking from libncurses.so.8 to another so.9 version, but then trying to enter the jail led to another error that another so file was missing. So I think it won't be that simple.

maybe you can link (and then copy, and then upgrade the base from inside) the entire repository of what is missing? Maybe someone a lot (a looot) more knowledgeable than me knows if and what it is?

 

[homer27081990]

Try downloading the version of freebsd your freenas had.

 

[homer27081990]

So, basically, what @Samuel Tai said. I just got it.

 

[Glorious1]

What is "Tabernacle"? Is that your username? Are you trying to log in via a SSH client or the GUI?

The tilde dollar sign is not anything I see in my console when trying to log in via SSH either by username or root. The tilde denotes that you are logged in to your home directory so you need to provide a bit more information on what you are trying to do and start from the beginning please.

ETA: So I did some more research. The dollar sign ($) indicates a bash shell. So yeah, provide some more information on your login procedure please because it looks like you are doing something non standard and clarification would be helpful here.

Tabernacle is the host name of the TrueNAS server. I use bash, both in the main server and in jails.

As mentioned, I have tried to enter the jail via an SSH client AND in GUI jail shell. To explain completely
via SSH
1. First log into the server from my laptop terminal window; I enter an alias of ssh -p <ssh port number> jim@192.168.0.102. Authentication is taken care of by public key.
2. After logging into the server, I enter sudo iocage console transmission. Then ensues the output shown in my first post. Afterward I am still in the main server account, not in the jail as I should be.
via GUI's jail shell
1. First log into my TrueNAS web GUI
2. Go to Jails > Transmission (jail name) > Shell
3. The same output appears in the shell, which is then non-responsive. Can't type anything or see a prompt.

I think there is something screwed up in FreeBSD 13 with not properly linking files like the one showing in the error message.

 

[homer27081990]

Just an instinct, try reinstalling transmissions libraries from pkg.

 

[Glorious1]

Unable to find a fix, I "upgraded" to 12.3, as Samuel Tai suggested. The jail works again. Thanks Samuel.

Stupid side-effect: the GUI and login message both say the jail is running 13.1. Other commands show a difference of opinion.

Code:

[root@tran ~]# freebsd-version
12.3-RELEASE-p6

[root@tran ~]# uname -a
FreeBSD tran 13.1-RELEASE FreeBSD 13.1-RELEASE n245376-eba770b30ff TRUENAS  amd64

 

[troonas]

I ran into this exact issue too... not sure why nobody suggested the most obvious solution to the libraries being out of sync with the base system. Just upgrade the packages directly from iocage without going through the jail's broken shell.

Code:

sudo iocage exec <jail-name> pkg upgrade

You may have to run that a couple times to get pkg upgraded first. Now everything works perfectly. Hopefully that saves someone from needlessly rebuilding or downgrading a jail.

 

[Jailer]

Unable to find a fix, I "upgraded" to 12.3, as Samuel Tai suggested. The jail works again. Thanks Samuel.

Stupid side-effect: the GUI and login message both say the jail is running 13.1. Other commands show a difference of opinion.

Code:

[root@tran ~]# freebsd-version
12.3-RELEASE-p6

[root@tran ~]# uname -a
FreeBSD tran 13.1-RELEASE FreeBSD 13.1-RELEASE n245376-eba770b30ff TRUENAS  amd64

uname -a will give you the kernel version. Since the host and jails share the same kernel it will display the same version in a jail as it does on the host. freebsd-version is the correct command to show what version of FreeBSD you are on.

 

[Glorious1]

I ran into this exact issue too... not sure why nobody suggested the most obvious solution to the libraries being out of sync with the base system. Just upgrade the packages directly from iocage without going through the jail's broken shell.

Code:

sudo iocage exec <jail-name> pkg upgrade

You may have to run that a couple times to get pkg upgraded first. Now everything works perfectly. Hopefully that saves someone from needlessly rebuilding or downgrading a jail.

This happened again after I replicated the pool to the new larger pool. I tried the trick above and got an error I don't understand.

Code:

Tabernacle:/$ sudo iocage exec ffmpeg pkg upgrade
Password:
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating FreeBSD repository catalogue...
[ffmpeg] Fetching packagesite.pkg: 100%    6 MiB   6.7MB/s    00:01   
Processing entries: 100%
FreeBSD repository update completed. 32329 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
    pkg: 1.18.3 -> 1.18.4

Number of packages to be upgraded: 1

The operation will free 5 MiB.
7 MiB to be downloaded.

Proceed with this action? [y/N]: y
[ffmpeg] [1/1] Fetching pkg-1.18.4.pkg: 100%    7 MiB   7.7MB/s    00:01   
Checking integrity... done (0 conflicting)
[ffmpeg] [1/1] Upgrading pkg from 1.18.3 to 1.18.4...
ld-elf.so.1: /usr/local/sbin/pkg: Undefined symbol "regcomp@FBSD_1.6"
Command: pkg upgrade failed!

Running it with pkg bootstrap -f didn't help.

 

[Glorious1]

I was eventually able to upgrade FreeBSD in the jail; then sudo iocage exec [I]jailname[/I] pkg update and sudo iocage exec [I]jailname[/I] pkg upgrade worked fine. I was then able to enter the jail normally. Thanks @troonas .