Upgrade from TrueNAS-22.02.4 -> TrueNAS-22.12.0 Bluefin fails - middleware not running

[dsalvat1]

This one has been a head scratcher for a while and I see many threads out there with folks who finally give up and reinstall. Personally, I think this is a problem which the upgrade process and how the SQL tables are being converted/created during the initialization of the update.

Upon reviewing the logs, I noticed that a folder was not being created

root@nas[/var/log]# cat messages|grep "Feb 11 19:03:44 nas" Feb 11 19:03:44 nas systemd-journald: Received SIGTERM from PID 1 (systemd). Feb 11 19:03:44 nas middlewared[2949]: Process zettarepl: Feb 11 19:03:44 nas middlewared[2949]: Traceback (most recent call last): Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap Feb 11 19:03:44 nas middlewared[2949]: self.run() Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run Feb 11 19:03:44 nas middlewared[2949]: self._target(*self._args, **self._kwargs) Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in __call__ Feb 11 19:03:44 nas middlewared[2949]: move_to_root_cgroups(os.getpid()) Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 13, in move_to_root_cgroups Feb 11 19:03:44 nas middlewared[2949]: with open(f"/sys/fs/cgroup/{new_name}/cgroup.procs", "w") as f2: Feb 11 19:03:44 nas middlewared[2949]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/unified/cgroup.procs'

So, I created it manually then ran the python script

mkdir /sys/fs/cgroup/unified/

then:

python /usr/lib/python3/dist-packages/middlewared/utils/cgroups.py

and the folder populated.
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.max 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.min 0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.numa_stat 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.oom.group 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.pressure 0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.stat 0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.swap.current 0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.swap.events 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.swap.high 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.swap.max 0 -r--r--r-- 1 root root 0 Feb 11 19:18 pids.current 0 -r--r--r-- 1 root root 0 Feb 11 19:18 pids.events 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 pids.max 0 -r--r--r-- 1 root root 0 Feb 11 19:18 rdma.current 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 rdma.max root@nas[/var/log]# ls -ls /sys/fs/cgroup/unified/cgroup.procs 0 -rw-r--r-- 1 root root 0 Feb 11 19:18 /sys/fs/cgroup/unified/cgroup.procs

Then ran the update again from TrueNAS-22.02.4 -> TrueNAS-22.12.0

This did not work, then I started looking into the alembic database upgrade schema. After the middleware failure screen, went to shell prompt and ran:

"alembic upgrade head"

error: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: iscsi_listen_port
[SQL: ALTER TABLE services_iscsitargetglobalconfiguration ADD COLUMN iscsi_listen_port INTEGER DEFAULT '3260' NOT NULL]

This is really a problem with the schema upgrade.

My specific issue is with the python script in /usr/lib/python3/dist-packages/alembic/versions/22.12/2022-07-27_01-00_portal_ips.py on line 28

all_ports.add(row['iscsi_target_portalip_port ']) else: if row['iscsi_target_portalip_ip'] == '0.0.0.0': listen_port = row['iscsi_target_portalip_port'] break if len(all ports) == 1:

 

[dsalvat1]

Update - After doing some digging, I realized the upgrade is unable to perform the schema update if iscsi was previously setup.

Solution:

- Boot into 22.02.4
- Noted iscsi configuration, and then removed entries, added "." for the basename
- Downloaded and applied bluefin update
- Restarted and system is now fully upgraded.