Mount error when trying to update

[Rana]

I am unable to update from TrueNAS-SCALE-22.02.4 to Bluefin. Independently on whether I let the GUI download the update or whether I provide it manually, I get the following Error:

[EFAULT] Invalid update image file. Please, re-download update. Error: mount: /tmp/tmp72zlj55e: mount failed: Operation not permitted.​

The file is certainly OK, it seems to be a permission issue.

I have root access to a CLI.

Can someone please tell me how to handle this?

 

[Rana]

I found a stackstrace of the error:

Fehler: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/utils_linux.py", line 18, in mount_update
subprocess.run(["mount", "-t", "squashfs", "-o", "loop", path, mounted], **run_kw)
File "/usr/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mount', '-t', 'squashfs', '-o', 'loop', '/var/tmp/firmware/manualupdate.tar', '/tmp/tmplaesqzt4']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 412, in file
await self.middleware.call('update.install_manual_impl', job, destfile, dest_extracted)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1345, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1305, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1206, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 23, in install_manual_impl
self._install(
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 34, in _install
with mount_update(path) as mounted:
File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
return next(self.gen)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/utils_linux.py", line 20, in mount_update
raise CallError(f"Invalid update image file. Please, re-download update. Error: {e.stdout}")
middlewared.service_exception.CallError: [EFAULT] Invalid update image file. Please, re-download update. Error: mount: /tmp/tmplaesqzt4: mount failed: Operation not permitted.

The issue seems to be Command '['mount', '-t', 'squashfs', '-o', 'loop', '/var/tmp/firmware/manualupdate.tar', '/tmp/tmplaesqzt4']' returned non-zero exit status 1. The update file (TrueNAS-SCALE-22.12.3.1.update) is not a tar archive. I would like to know: which of the files listed in https://download.truenas.com/TrueNAS-SCALE-Bluefin/22.12.3.1/ are supposed to be supplemented manually for the update? Is it save to submit an iso file?

 

[Rana]

So my TrueNAS Scale Anglefish is expecting a TAR update file. I have obtained an old TAR update of the expected format and found it to contain three TGZ files (1 base OS archive and 2 FreeBSD PKG archives) plus some auxiliary files. (This is just for inspection, I will certainly not try to up/downgrade a Debian based OS with FreeBSD)

The current Bluefin iso file seems to contain a complete live system. So I can certainly not simply provide the iso instead.

My last hope is to understand the TrueNAS-SCALE-22.12.3.1.update file, the format of which can hopefully be digested by TrueNAS-SCALE-22.02.4 in some way. So the question is: What is its file format? It is not a TAR. Is it a binary?

 

[Rana]

Sooo, its a squashfs containing yet another squashfs containing a Debian system.

My impression is that the update mechanism that Anglefish provides differs substantially from the update format in which Bluefin is provided. Thank you so much!

 

[sretalla]

Just backup your config, install Bluefin fresh, restore the config... why waste all this time with a broken system?

 

[Rana]

Well, it is a mission critical system, thats why I have waited so long with the update. And it's not broken, except for the update mechanism, it works flawlessly. But it sounds easy enough to update as you described, I will try this.

 

[Rana]

I believe a more save update procedure is to not jump directly from 22.02.04 to 22.12.3.1 but to go through the minor update steps. Luckily, the older versions are still available.

 

[sretalla]

I believe a more save update procedure is to not jump directly from 22.02.04 to 22.12.3.1 but to go through the minor update steps. Luckily, the older versions are still available.

I don't think that's necessary. It won't harm anything if you do it, but you're just wasting time.

 

[Rana]

I have to be very cautious with this system, so I tried to update step-wise. But already the first Bluefin update file (to 22.12.0) throws the error (i.e. is provided as squashFS whereas Anglefish expects something else). I can not find the correct update file. Can you tell me where to find the magic update file that gets me from TrueNAS-SCALE-22.02.4 (Anglefish) to Bluefin?

 

[sretalla]

OK, we're back to square one...

I recommend that you:

Remove your current boot drive (after taking a config backup).

Insert a new boot drive, built whatever version you feel comfortable with on that new boot drive, then restore your config.

If something you don't like happens, swap back to your old boot drive.

 

[Rana]

The eagle has landed!
We finally upgraded by booting from a USB drive, and most things work.
I appreciate the input from sretalla.
Still, I would like to understand what went wrong in the first place to not break the update mechanism (and potentially other things) again. Comments on the stack trace from my second post are welcome.