SOLVED Cannot upgrade jails to 13.1-RELEASE (spaces in pool name)

[mJh78B]

Bug reports and pull requests:

Correctly construct arguments for freebsd-update by aWZHY0yQH81uOYvH · Pull Request #1291 · iocage/iocage

When calling freebsd-update, --currently-running and the jail's current release should be separate arguments, instead of in the same argument separated by a space. This only works now because freeb...

github.com

Pull `freebsd-update` from active branch of freebsd/freebsd-src by aWZHY0yQH81uOYvH · Pull Request #1292 · iocage/iocage

The master branch of freebsd-src is obsolete and is no longer receiving commits, so iocage fetch is using an old version of freebsd-update. They switched to the main branch for active development. ...

github.com

268686 – freebsd-update cannot handle spaces in file paths

bugs.freebsd.org

 

[SHREDDER007]

Yep, freebsd-update doesn't correctly parse its command line arguments.
(nor does it produce a nonzero exit code on this failure...)

There are many places where this script cannot handle spaces, including passing raw file lists into xargs. It also calls certctl rehash which, guess what, is another shell script that incorrectly handles spaces. I basically added a bunch of quotes to this version of freebsd-update, commented out the certctl rehash step (because I couldn't be bothered to fix it — just run it from within the jail), and modified the xargs call to fix all this (at least fetch, upgrade, and install seem to work). My hacked freebsd-update is attached.

There is also a bug in iocage where it doesn't construct the arguments for freebsd-update correctly here. --currently-running and the release name should be separate arguments separated by commas in the array, instead of in the same argument separated by a space. This only worked before because freebsd-update was broken.

After pointing iocage to my version of freebsd-update instead of the version it downloads into a temporary location, I can now upgrade my jails.

Code:

root@truenas[/tmp]# iocage list -l                                
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+
| JID |    NAME     | BOOT | STATE | TYPE |     RELEASE     |         IP4         | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+=================+=====================+=====+==========+==========+
| 1   | Discord-Bot | on   | up    | jail | 13.1-RELEASE-p5 | vnet0|172.16.0.2/30 | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+
| 2   | OpenVPN     | on   | up    | jail | 13.1-RELEASE-p5 | vnet0|172.16.0.6/30 | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+---------------------+-----+----------+----------+

Perhaps there should be a warning to not put spaces in your pool name.

Thank you for this. I had given up trying to figure out this issue a few months back, thought I'd try again today and came across your post. I never would have figured this out. Got my upgrade working now though thanks to your files!

 

[mJh78B]

Thank you for this. I had given up trying to figure out this issue a few months back, thought I'd try again today and came across your post. I never would have figured this out. Got my upgrade working now though thanks to your files!

Glad to hear it was helpful!

Also, if you've got an automatic scrub set up for your pool with spaces, it might not be running properly. I just came across this bug (missing escapes in the cron file) and found that they'd fixed it 9 days after the release tag for 13-3.1.

Fix cron task for scrubbing a pool containing whitespace in its name · truenas/middleware@e4ce3c2

TrueNAS CORE/Enterprise/SCALE Middleware Git Repository - Fix cron task for scrubbing a pool containing whitespace in its name · truenas/middleware@e4ce3c2

github.com

You might want to apply this commit patch to /usr/local/lib/python3.9/site-packages/middlewared/etc_files/cron.d/middlewared.mako and reboot to fix that issue as well.

 

[SHREDDER007]

Glad to hear it was helpful!

Also, if you've got an automatic scrub set up for your pool with spaces, it might not be running properly. I just came across this bug (missing escapes in the cron file) and found that they'd fixed it 9 days after the release tag for 13-3.1.

Fix cron task for scrubbing a pool containing whitespace in its name · truenas/middleware@e4ce3c2

TrueNAS CORE/Enterprise/SCALE Middleware Git Repository - Fix cron task for scrubbing a pool containing whitespace in its name · truenas/middleware@e4ce3c2

github.com

You might want to apply this commit patch to /usr/local/lib/python3.9/site-packages/middlewared/etc_files/cron.d/middlewared.mako and reboot to fix that issue as well.

I'll definitely have to have a look at this too, but first it looks like I may have spoken too soon, unfortunately. It looks like for some reason the new .sh script isn't able to access some files during the upgrade process.

Looking up update.FreeBSD.org mirrors... 2 mirrors found. Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done. Fetching metadata index... done. Fetching 1 metadata patches. done. Applying metadata patches... done. Fetching 1 metadata files... done. Inspecting system... done. The following components of FreeBSD seem to be installed: src/src world/base world/doc world/lib32 The following components of FreeBSD do not seem to be installed: world/base-dbg world/lib32-dbg Does this look reasonable (y/n)? y Fetching metadata signature for 13.1-RELEASE from update1.freebsd.org... done. Fetching metadata index... done. Fetching 1 metadata patches. done. Applying metadata patches... done. Fetching 1 metadata files... done. Inspecting system... done. Fetching files from 12.2-RELEASE for merging... done. Preparing to download files... done. /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory /tmp/freebsd-update-hacked.sh: cannot open files/.gz: No such file or directory

...and that continues for several more lines. I'm then getting asked about a TON of files

The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /boot/device.hints Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/auto_master Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/include_ldap Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/include_nis Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/include_nis_nullfs Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/special_hosts Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/special_media Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/special_noauto Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/autofs/special_null Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/blacklistd.conf Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/bluetooth/hcsecd.conf Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/bluetooth/hosts Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/bluetooth/protocols Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/cron.d/at Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/crontab Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/csh.cshrc Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/csh.login Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/csh.logout Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/ddb.conf Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/defaults/bluetooth.device.conf Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/defaults/devfs.rules Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/defaults/periodic.conf Does this look reasonable (y/n)? y The following file will be removed, as it no longer exists in FreeBSD 13.1-RELEASE: /etc/defaults/rc.conf Does this look reasonable (y/n)? y ..... .....

And that just doesn't seem right to me, so I'm going to have to figure out what's going on. If you've got any ideas, I'm all ears!

 

[SHREDDER007]

I'm going to try clearing out the /var/db/freebsd-update folder in my jail. I'm wondering if that's what's causing the issue... maybe a previously failed upgrade is getting in the way.

 

[mJh78B]

Those .gz files it's trying to decompress might be part of the stuff it downloaded. I think they should be in iocage/download/13.1-RELEASE. Are you running the script on its own manually or through the iocage utility? I was able to run iocage fetch to get the thing downloaded and it was able to unzip files at that step where it's failing.

 

[SHREDDER007]

Those .gz files it's trying to decompress might be part of the stuff it downloaded. I think they should be in iocage/download/13.1-RELEASE. Are you running the script on its own manually or through the iocage utility? I was able to run iocage fetch to get the thing downloaded and it was able to unzip files at that step where it's failing.

So I'm running it via iocage upgrade. The fetch still throws that weird tmp error for me, so maybe I missed a step somewhere when adding your fixes?

 

[SHREDDER007]

Oh now that I'm more awake than I was last night, I'm realizing I probably just need to edit the ioc_fetch.py file similar to what you did for ioc_upgrade.py. Will report back.

 

[SHREDDER007]

Ok yeah, got fetch working now. I'm going to try another upgrade and see what happens!

 

[grahamperrin]

… I'm going to try another upgrade and see what happens!

Can you recall what happened?

Previously:

I'm going to try clearing out the /var/db/freebsd-update folder in my jail.…

Can you recall whether you did try that, and whether there was an effect?

TIA