SOLVED Cannot upgrade jails to 13.1-RELEASE (spaces in pool name)

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
I have two jails I created manually using the web GUI. They're currently sitting at 12.3-RELEASE. I just upgraded to the newest TrueNAS-13.0-U3.1 release from version 12, so I wanted to upgrade the jails as well. As far as I can tell, there are two main commands required to achieve this: iocage fetch and iocage upgrade. When I try to run iocage fetch, it downloaded some archives the first time I ran it then decompresses what it downloaded, but then fails, apparently trying to run what I presume is freebsd-update (based on the usage printed) with incorrect arguments:

Code:
root@truenas[~]# iocage fetch
[0] 12.3-RELEASE
[1] 12.4-RELEASE
[2] 13.0-RELEASE
[3] 13.1-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (13.1-RELEASE)
Type EXIT to quit:
Fetching: 13.1-RELEASE

Extracting: base.txz...
Extracting: lib32.txz...
Extracting: src.txz...

* Updating 13.1-RELEASE to the latest patch level...
usage: tmpi_ru6kbm [options] command ... [path]

Options:
  -b basedir   -- Operate on a system mounted at basedir
                  (default: /)
  -d workdir   -- Store working files in workdir
                  (default: /var/db/freebsd-update/)
  -f conffile  -- Read configuration options from conffile
                  (default: /etc/freebsd-update.conf)
  -F           -- Force a fetch operation to proceed in the
                  case of an unfinished upgrade
  -k KEY       -- Trust an RSA key with SHA256 hash of KEY
  -r release   -- Target for upgrade (e.g., 11.1-RELEASE)
  -s server    -- Server from which to fetch updates
                  (default: update.FreeBSD.org)
  -t address   -- Mail output of cron command, if any, to address
                  (default: root)
  --not-running-from-cron
               -- Run without a tty, for use by automated tools
  --currently-running release
               -- Update as if currently running this release
Commands:
  fetch        -- Fetch updates from server
  cron         -- Sleep rand(3600) seconds, fetch updates, and send an
                  email if updates were found
  upgrade      -- Fetch upgrades to FreeBSD version specified via -r option
  updatesready -- Check if there are fetched updates ready to install
  install      -- Install downloaded updates or upgrades
  rollback     -- Uninstall most recently installed updates
  IDS          -- Compare the system against an index of "known good" files
  showconfig   -- Show configuration
usage: tmpi_ru6kbm [options] command ... [path]

Options:
  -b basedir   -- Operate on a system mounted at basedir
                  (default: /)
  -d workdir   -- Store working files in workdir
                  (default: /var/db/freebsd-update/)
  -f conffile  -- Read configuration options from conffile
                  (default: /etc/freebsd-update.conf)
  -F           -- Force a fetch operation to proceed in the
                  case of an unfinished upgrade
  -k KEY       -- Trust an RSA key with SHA256 hash of KEY
  -r release   -- Target for upgrade (e.g., 11.1-RELEASE)
  -s server    -- Server from which to fetch updates
                  (default: update.FreeBSD.org)
  -t address   -- Mail output of cron command, if any, to address
                  (default: root)
  --not-running-from-cron
               -- Run without a tty, for use by automated tools
  --currently-running release
               -- Update as if currently running this release
Commands:
  fetch        -- Fetch updates from server
  cron         -- Sleep rand(3600) seconds, fetch updates, and send an
                  email if updates were found
  upgrade      -- Fetch upgrades to FreeBSD version specified via -r option
  updatesready -- Check if there are fetched updates ready to install
  install      -- Install downloaded updates or upgrades
  rollback     -- Uninstall most recently installed updates
  IDS          -- Compare the system against an index of "known good" files
  showconfig   -- Show configuration


Regardless it appears to have downloaded something into iocage/download and iocage/releases. If I try to upgrade my jails, it again appears to try to run freebsd-update with invalid syntax and fails. The jail is changed to "up" state but otherwise nothing changes.

Code:
root@truenas[~]# iocage upgrade -r 13.1-RELEASE OpenVPN
usage: tmpl9bndxps [options] command ...

Options:
  -b basedir   -- Operate on a system mounted at basedir
                  (default: /)
  -d workdir   -- Store working files in workdir
                  (default: /var/db/freebsd-update/)
  -f conffile  -- Read configuration options from conffile
                  (default: /etc/freebsd-update.conf)
  -F           -- Force a fetch operation to proceed in the
                  case of an unfinished upgrade
  -j jail      -- Operate on the given jail specified by jid or name
  -k KEY       -- Trust an RSA key with SHA256 hash of KEY
  -r release   -- Target for upgrade (e.g., 11.1-RELEASE)
  -s server    -- Server from which to fetch updates
                  (default: update.FreeBSD.org)
  -t address   -- Mail output of cron command, if any, to address
                  (default: root)
  --not-running-from-cron
               -- Run without a tty, for use by automated tools
  --currently-running release
               -- Update as if currently running this release
Commands:
  fetch        -- Fetch updates from server
  cron         -- Sleep rand(3600) seconds, fetch updates, and send an
                  email if updates were found
  upgrade      -- Fetch upgrades to FreeBSD version specified via -r option
  updatesready -- Check if there are fetched updates ready to install
  install      -- Install downloaded updates or upgrades
  rollback     -- Uninstall most recently installed updates
  IDS          -- Compare the system against an index of "known good" files
  showconfig   -- Show configuration
Upgrade failed, nothing to install after fetch!


The only other reference I found to this "Upgrade failed, nothing to install after fetch!" error was in the FreeBSD bug tracker and this post, but they aren't getting the "usage: ..." part and that doesn't seem to be the root problem here. I'm not sure where to go from here since it seems like the iocage command itself is broken due to a change in freebsd-update, even though iocage is the latest released version (1.2).
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
700
just for clarity .. are you running the command from the main shell and not from within the jail itself?
 

Viandoriel

Dabbler
Joined
Jul 17, 2021
Messages
20
I've had the same problem since release of Version 13, but I tried to live with it until today.
Just for your Information, how it was working with me, to upgrade all jails from 12 to 13:

1. Connect via SSH Client to TrueNAS (the web version hungs up after a while, putty is more stable in my case)
2. Using command below, works fine for me:

iocate upgrade PLEX -r 13.1-RELEASE-p2
-> Where "PLEX" is my Jail...

Today I successfully upgraded all my jails with this procedure :)
 

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
I've had the same problem since release of Version 13, but I tried to live with it until today.
Just for your Information, how it was working with me, to upgrade all jails from 12 to 13:

1. Connect via SSH Client to TrueNAS (the web version hungs up after a while, putty is more stable in my case)
2. Using command below, works fine for me:

iocate upgrade PLEX -r 13.1-RELEASE-p2
-> Where "PLEX" is my Jail...

Today I successfully upgraded all my jails with this procedure :)
Unfortunately specifying the patch number (p2) as well doesn't change the behavior. Still seems to be running freebsd-update with invalid arguments internally. Also tried doing the same over SSH but it still does the same thing.

Seems like I might have some kind of corrupted installation? How could I verify the install? I can't install a manual update to the latest version (complains that I'm already using 13.0-U3.1).
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Have you done anything outside of the norm to the base system such as installing a different shell on TrueNAS?
 

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
Have you done anything outside of the norm to the base system such as installing a different shell on TrueNAS?
I think I messed with PF for a while (but reboots reverted any changes), then made a startup script that enables PF inside a jail by making a new devfs ruleset. I wouldn't think either of these things would cause this.

The root user's shell is currently set to /usr/local/bin/zsh, which is not a symlink to anything else.
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
700
I've had the same problem since release of Version 13, but I tried to live with it until today.
Just for your Information, how it was working with me, to upgrade all jails from 12 to 13:

1. Connect via SSH Client to TrueNAS (the web version hungs up after a while, putty is more stable in my case)
2. Using command below, works fine for me:

iocate upgrade PLEX -r 13.1-RELEASE-p2
-> Where "PLEX" is my Jail...

Today I successfully upgraded all my jails with this procedure :)
just an FYI .. 13.1-RELEASE-p5 is latest
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Sorry. iocage upgrade works here every single time so far.
 
Joined
Oct 22, 2019
Messages
3,641
Can you post the output of:
iocage list -l

As well as:
zfs get -t filesystem -r exec poolname/iocage
 

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
Can you post the output of:
iocage list -l

As well as:
zfs get -t filesystem -r exec poolname/iocage
Code:
root@truenas[~]# iocage list -l
+-----+-------------+------+-------+------+--------------+---------------------+-----+----------+----------+
| JID |    NAME     | BOOT | STATE | TYPE |   RELEASE    |         IP4         | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+==============+=====================+=====+==========+==========+
| 1   | Discord-Bot | on   | up    | jail | 12.3-RELEASE | vnet0|172.16.0.2/30 | -   | -        | no       |
+-----+-------------+------+-------+------+--------------+---------------------+-----+----------+----------+
| 2   | OpenVPN     | on   | up    | jail | 12.3-RELEASE | vnet0|172.16.0.6/30 | -   | -        | no       |
+-----+-------------+------+-------+------+--------------+---------------------+-----+----------+----------+
root@truenas[~]# zfs get -t filesystem -r exec BIG\ DATA/iocage
NAME                                        PROPERTY  VALUE  SOURCE
BIG DATA/iocage                             exec      on     default
BIG DATA/iocage/download                    exec      on     default
BIG DATA/iocage/download/12.2-RELEASE       exec      on     default
BIG DATA/iocage/download/12.3-RELEASE       exec      on     default
BIG DATA/iocage/download/13.1-RELEASE       exec      on     default
BIG DATA/iocage/images                      exec      on     default
BIG DATA/iocage/jails                       exec      on     default
BIG DATA/iocage/jails/Discord-Bot           exec      on     default
BIG DATA/iocage/jails/Discord-Bot/root      exec      on     default
BIG DATA/iocage/jails/OpenVPN               exec      on     default
BIG DATA/iocage/jails/OpenVPN/root          exec      on     default
BIG DATA/iocage/log                         exec      on     default
BIG DATA/iocage/releases                    exec      on     default
BIG DATA/iocage/releases/12.2-RELEASE       exec      on     default
BIG DATA/iocage/releases/12.2-RELEASE/root  exec      on     default
BIG DATA/iocage/releases/12.3-RELEASE       exec      on     default
BIG DATA/iocage/releases/12.3-RELEASE/root  exec      on     default
BIG DATA/iocage/releases/13.1-RELEASE       exec      on     default
BIG DATA/iocage/releases/13.1-RELEASE/root  exec      on     default
BIG DATA/iocage/templates                   exec      on     default


Conceivably the spaces in the pool name could be causing problems if not properly escaped, but I never had any problems creating these jails through the web UI...
 
Joined
Oct 22, 2019
Messages
3,641
Conceivably the spaces in the pool name could be causing problems if not properly escaped

That's actually an interesting note. Perhaps the white spaces in the pool's name is causing an issue for the iocage fetch and/or upgrade commands.

Everything else looks fine.

If it means anything, I'm using Basejails (not Clone jails).

What I also find odd is that "iocage" is being referenced as some temporary garbled name when you run it:
usage: tmpl9bndxps [options] command
 

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
If it means anything, I'm using Basejails (not Clone jails).
I should probably change to basejails if I ever recreate these, but at the time I didn't know the difference and just left the default. The OpenVPN one was a bit of a PITA to get NAT working in, so was hoping to not have to do that right now.

What I also find odd is that "iocage" is being referenced as some temporary garbled name when you run it:
That's freebsd-update, not iocage complaining about its usage. That garbled name is generated here for upgrade or here for fetch. The arguments there are constructed using an array element for each one so there shouldn't be a problem with escaping, unless it's a problem within freebsd-update (it is a shell script, so possible).

What's also interesting is that although iocage reports itself as version 1.2
Code:
root@truenas[~]# iocage --version
Version 1.2

... the files in /usr/local/lib/python3.9/site-packages/iocage_lib don't match those at the 1.2 tag on GitHub. However, all the parameters that are supplied to freebsd-update seem correct.

Also worthy of note is that each FreeBSD version has its own freebsd-update script that is fetched from e.g. https://raw.githubusercontent.com/freebsd/freebsd/release/13.1.0/usr.sbin/freebsd-update/freebsd-update.sh for version 13.1. Of the four versions available to be fetched...
Code:
root@truenas[~]# iocage fetch
[0] 12.3-RELEASE
[1] 12.4-RELEASE
[2] 13.0-RELEASE
[3] 13.1-RELEASE
...

Version 13.0 has a different usage...
Code:
$ diff 13.0.0 13.1.0
39c39
< usage: `basename $0` [options] command ... [path]
---
> usage: `basename $0` [options] command ...
49a50
>   -j jail      -- Operate on the given jail specified by jid or name

... but every other version has the -j flag.

However, notice that the two usage printouts I posted above from my two iocage commands are subtly different: one has the -j flag and the other doesn't. This is because iocage upgrade pulls its freebsd-update from the URL scheme above, while iocage fetch pulls its version from https://raw.githubusercontent.com/freebsd/freebsd/master/usr.sbin/freebsd-update/freebsd-update.sh. This is a different branch that is now obsolete and broken.

That doesn't explain why iocage upgrade is also broken, but the fact that iocage fetch is pulling scripts from a broken, unstable, and obsolete branch to be run on an unmatched version of FreeBSD seems bad.
 

mJh78B

Dabbler
Joined
Apr 2, 2022
Messages
20
there shouldn't be a problem with escaping, unless it's a problem within freebsd-update (it is a shell script, so possible).
Yep, freebsd-update doesn't correctly parse its command line arguments.
(nor does it produce a nonzero exit code on this failure...)

There are many places where this script cannot handle spaces, including passing raw file lists into xargs. It also calls certctl rehash which, guess what, is another shell script that incorrectly handles spaces. I basically added a bunch of quotes to this version of freebsd-update, commented out the certctl rehash step (because I couldn't be bothered to fix it — just run it from within the jail), and modified the xargs call to fix all this (at least fetch, upgrade, and install seem to work). My hacked freebsd-update is attached.

There is also a bug in iocage where it doesn't construct the arguments for freebsd-update correctly here. --currently-running and the release name should be separate arguments separated by commas in the array, instead of in the same argument separated by a space. This only worked before because freebsd-update was broken.

After pointing iocage to my version of freebsd-update instead of the version it downloads into a temporary location, I can now upgrade my jails.

Code:
root@truenas[/tmp]# iocage list -l                                 
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+
| JID |    NAME     | BOOT | STATE | TYPE |     RELEASE     |         IP4         | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+=================+=====================+=====+==========+==========+
| 1   | Discord-Bot | on   | up    | jail | 13.1-RELEASE-p5 | vnet0|172.16.0.2/30 | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+
| 2   | OpenVPN     | on   | up    | jail | 13.1-RELEASE-p5 | vnet0|172.16.0.6/30 | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+


Perhaps there should be a warning to not put spaces in your pool name.
 

Attachments

  • freebsd-update-hacked.sh.txt
    93.5 KB · Views: 141
  • ioc_upgrade.py.txt
    14.5 KB · Views: 134
Joined
Oct 22, 2019
Messages
3,641
Perhaps there should be a warning to not put spaces in your pool name.
I never use spaces in pool names or dataset names (or snapshot names, or bookmark names, or jail names...) :wink:

Good to know you got it "working" now. Still a shame that white spaces break the ability to upgrade your jails. :oops:
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Glad you got it figured out. You should submit a bug report to iX so they can work on a fix for this so others don't run into the same issue. Might not also hurt to submit a bug report to the FreeBSD project as well since this issue is with freebsd-update as well.
 
Top