upgrade TrueNAS-22.12-RC.1 failed,it says "middleware in not running"

[henry_cui]

when I upgrade TrueNAS-22.02.4 to TrueNAS-22.12-RC.1,after reboot ,it prompts "middleware in not running.Press Enter to open root shell" and stoped,I check middlewared.log,it show lack field in sqlite
middlewared

 

[morganL]

If it fails on successive reboots, please report a bug with full hardware configuration.
Has anyone else seen this?

 

[exodus454]

I've had that problem too, every time I've tried to upgrade past .4

To the RC, nightly, clean configs, nothing has worked. Always "middleware not running"

 

[henry_cui]

If it fails on successive reboots, please report a bug with full hardware configuration.
Has anyone else seen this?

8 x 12th Gen Intel(R) Core(TM) i3-12100
Linux 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100)
pve-manager/7.3-3/c3928077

 

[morganL]

I've had that problem too, every time I've tried to upgrade past .4

To the RC, nightly, clean configs, nothing has worked. Always "middleware not running"

Its probably a bug, but triggered by either your specific hardware or configuration.

If you can document both and replicate the issue, please report a bug.

 

[scottkorvek]

Same here. Cannot update past 22.02.4, cannot delete the broken 22.12.0 environment.
Trying to install 22.12.0 on a new server with very different hardware as a net new (not migrating, adding for redundancy) environment - will report if I can get that one up.

 

[scottkorvek]

Same here. Cannot update past 22.02.4, cannot delete the broken 22.12.0 environment.
Trying to install 22.12.0 on a new server with very different hardware as a net new (not migrating, adding for redundancy) environment - will report if I can get that one up.

Able to create net new system- was not able to update it from an older install .iso- started over with 22.12 and am fine. The issue on the existing server is exactly as above- missing column in config database (system_settings.stg_ds_auth).

I tried the update via the UI as well as burning the full install image to a USB drive and doing a full upgrade which states everything worked ok and had no errors with the database migration step.

Should I try an overwrite install and try to apply my config settings from the backup file?

 

[morganL]

Able to create net new system- was not able to update it from an older install .iso- started over with 22.12 and am fine. The issue on the existing server is exactly as above- missing column in config database (system_settings.stg_ds_auth).

I tried the update via the UI as well as burning the full install image to a USB drive and doing a full upgrade which states everything worked ok and had no errors with the database migration step.

Should I try an overwrite install and try to apply my config settings from the backup file?

I think this means you have one system up on 22.12.0, but still having problems updating a 2nd system?

More details needed.. suggest a new thread.

 

[zfsuser9001]

I have the same issue going from angelfish current to bluefin release. It halts with 'middleware not running'.

Running systemctl status ix-update.service doesn't have anything terribly useful.
failed with result 'exit-code'

Supermicro X10DRi-C, Dual CPU E5-v4, 512GB ECC RAM

It seems like it doesn't even try to start the update, there is no lag. Booted from angelfish and redownloaded it again, same result.

 

[zfsuser9001]

Tried the update again from the GUI. While the update was spinning in the browser, this was the console output, repeated over 100x times before it rebooted and the update fails at boot console output.

Code:

Dec 28 13:06:28 freenas middlewared[70564]:   File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in __call__
Dec 28 13:06:28 freenas middlewared[70564]:     move_to_root_cgroups(os.getpid())
Dec 28 13:06:28 freenas middlewared[70564]:   File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 13, in move_to_root_cgroups
Dec 28 13:06:28 freenas middlewared[70564]:     with open(f"/sys/fs/cgroup/{new_name}/cgroup.procs", "w") as f2:
Dec 28 13:06:28 freenas middlewared[70564]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/unified/cgroup.procs'
Dec 28 13:06:29 freenas middlewared[70567]: Process zettarepl:
Dec 28 13:06:29 freenas middlewared[70567]: Traceback (most recent call last):
Dec 28 13:06:29 freenas middlewared[70567]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Dec 28 13:06:29 freenas middlewared[70567]:     self.run()
Dec 28 13:06:29 freenas middlewared[70567]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
Dec 28 13:06:29 freenas middlewared[70567]:     self._target(*self._args, **self._kwargs)
Dec 28 13:06:29 freenas middlewared[70567]:   File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in __call__
Dec 28 13:06:29 freenas middlewared[70567]:     move_to_root_cgroups(os.getpid())
Dec 28 13:06:29 freenas middlewared[70567]:   File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 13, in move_to_root_cgroups
Dec 28 13:06:29 freenas middlewared[70567]:     with open(f"/sys/fs/cgroup/{new_name}/cgroup.procs", "w") as f2:
Dec 28 13:06:29 freenas middlewared[70567]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/unified/cgroup.procs'
Dec 28 13:06:30 freenas middlewared[70570]: Process zettarepl:
Dec 28 13:06:30 freenas middlewared[70570]: Traceback (most recent call last):
Dec 28 13:06:30 freenas middlewared[70570]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Dec 28 13:06:30 freenas middlewared[70570]:     self.run()
Dec 28 13:06:30 freenas middlewared[70570]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
Dec 28 13:06:30 freenas middlewared[70570]:     self._target(*self._args, **self._kwargs)
Dec 28 13:06:30 freenas middlewared[70570]:   File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in __call__
Dec 28 13:06:30 freenas middlewared[70570]:     move_to_root_cgroups(os.getpid())
Dec 28 13:06:30 freenas middlewared[70570]:   File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 13, in move_to_root_cgroups
Dec 28 13:06:30 freenas middlewared[70570]:     with open(f"/sys/fs/cgroup/{new_name}/cgroup.procs", "w") as f2:
Dec 28 13:06:30 freenas middlewared[70570]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/unified/cgroup.procs'
Dec 28 13:06:31 freenas middlewared[70580]: Process zettarepl:

 

[morganL]

Has anyone reported the bug?
Please provide a bug-ID for everyone to track.

 

[zfsuser9001]

[NAS-119644] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[orztrickster]

It's possible it's the same problem

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

 

[zfsuser9001]

any chance this will get a fix before Feb? Seems like it might be any issue with some file or mount that isn't set up. This was a vanilla install from CD of the prior version of SCALE and I haven't futzed with any system files. So if files are missing it is from the vanilla install or one of the subsequent updates.

 

[vasquezmi]

I can't even submit a ticket through the Atlassian portal. Frustrating.

 

[zfsuser9001]

Looking in /data/update.failed in the shell, my case appears to be related to an issue in migrating iscsi services.