This one has been a head scratcher for a while and I see many threads out there with folks who finally give up and reinstall. Personally, I think this is a problem which the upgrade process and how the SQL tables are being converted/created during the initialization of the update.
Upon reviewing the logs, I noticed that a folder was not being created
So, I created it manually then ran the python script
then:
and the folder populated.
Then ran the update again from TrueNAS-22.02.4 -> TrueNAS-22.12.0
This did not work, then I started looking into the alembic database upgrade schema. After the middleware failure screen, went to shell prompt and ran:
"alembic upgrade head"
error: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: iscsi_listen_port
[SQL: ALTER TABLE services_iscsitargetglobalconfiguration ADD COLUMN iscsi_listen_port INTEGER DEFAULT '3260' NOT NULL]
This is really a problem with the schema upgrade.
My specific issue is with the python script in /usr/lib/python3/dist-packages/alembic/versions/22.12/2022-07-27_01-00_portal_ips.py on line 28
Upon reviewing the logs, I noticed that a folder was not being created
root@nas[/var/log]# cat messages|grep "Feb 11 19:03:44 nas"
Feb 11 19:03:44 nas systemd-journald: Received SIGTERM from PID 1 (systemd).
Feb 11 19:03:44 nas middlewared[2949]: Process zettarepl:
Feb 11 19:03:44 nas middlewared[2949]: Traceback (most recent call last):
Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Feb 11 19:03:44 nas middlewared[2949]: self.run()
Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
Feb 11 19:03:44 nas middlewared[2949]: self._target(*self._args, **self._kwargs)
Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in __call__
Feb 11 19:03:44 nas middlewared[2949]: move_to_root_cgroups(os.getpid())
Feb 11 19:03:44 nas middlewared[2949]: File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 13, in move_to_root_cgroups
Feb 11 19:03:44 nas middlewared[2949]: with open(f"/sys/fs/cgroup/{new_name}/cgroup.procs", "w") as f2:
Feb 11 19:03:44 nas middlewared[2949]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/unified/cgroup.procs'
So, I created it manually then ran the python script
mkdir /sys/fs/cgroup/unified/
then:
python /usr/lib/python3/dist-packages/middlewared/utils/cgroups.py
and the folder populated.
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.max
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.min
0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.numa_stat
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.oom.group
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.pressure
0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.stat
0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.swap.current
0 -r--r--r-- 1 root root 0 Feb 11 19:18 memory.swap.events
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.swap.high
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 memory.swap.max
0 -r--r--r-- 1 root root 0 Feb 11 19:18 pids.current
0 -r--r--r-- 1 root root 0 Feb 11 19:18 pids.events
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 pids.max
0 -r--r--r-- 1 root root 0 Feb 11 19:18 rdma.current
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 rdma.max
root@nas[/var/log]# ls -ls /sys/fs/cgroup/unified/cgroup.procs
0 -rw-r--r-- 1 root root 0 Feb 11 19:18 /sys/fs/cgroup/unified/cgroup.procs
Then ran the update again from TrueNAS-22.02.4 -> TrueNAS-22.12.0
This did not work, then I started looking into the alembic database upgrade schema. After the middleware failure screen, went to shell prompt and ran:
"alembic upgrade head"
error: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: iscsi_listen_port
[SQL: ALTER TABLE services_iscsitargetglobalconfiguration ADD COLUMN iscsi_listen_port INTEGER DEFAULT '3260' NOT NULL]
This is really a problem with the schema upgrade.
My specific issue is with the python script in /usr/lib/python3/dist-packages/alembic/versions/22.12/2022-07-27_01-00_portal_ips.py on line 28
all_ports.add(row['iscsi_target_portalip_port ']) else: if row['iscsi_target_portalip_ip'] == '0.0.0.0': listen_port = row['iscsi_target_portalip_port'] break if len(all ports) == 1: