SCALE Nightlies upgrade broken ?

[ato]

I can't upgrade in GUI. Same in CLI:
root@nas3[~]# cat /etc/version
22.02-MASTER-20220205-112923
root@nas3[~]# cli
[nas3]> system update update
[0%] Retrieving update manifest...
[0%] Verifying existing update...
[50%] Reading update file...
Error: Unable to downgrade from 22.02-MASTER-20220205-112923 to 22.02.1-MASTER-20220207-152925
So how to switch train to 0.2.1-MASTER (or to stay on head) ?

 

[ArchatParks]

I can't upgrade in GUI. Same in CLI:
root@nas3[~]# cat /etc/version
22.02-MASTER-20220205-112923
root@nas3[~]# cli
[nas3]> system update update
[0%] Retrieving update manifest...
[0%] Verifying existing update...
[50%] Reading update file...
Error: Unable to downgrade from 22.02-MASTER-20220205-112923 to 22.02.1-MASTER-20220207-152925
So how to switch train to 0.2.1-MASTER (or to stay on head) ?

View attachment 52928

I have the same issue at two different sites.

 

[Kris Moore]

Hey! Not sure the cause of this yet. Can you please file a ticket on https://jira.ixsystems.com so we can be sure to get this investigated?

 

[ArchatParks]

ok...just did!

 

[elvisimprsntr]

I had already submitted one

https://jira.ixsystems.com/browse/NAS-114710

Will we be stuck in an endless loop? Or can they release an update under the old version number convention with what ever patches are necessary to allow updates to the new 22.02.01 version numbering?

 

[ClaesBas]

Is it possible to fix it through CLI, with "system update" and some commands?
I also went into this problem yesterday..
Tried CLI: "system update update", today and got:
Error: Unable to downgrade from 22.02-MASTER-20220206-035100 to 22.02.1-MASTER-20220208-072925
Or is the only way to get out from it, to boot into "RC2"?

 

[elvisimprsntr]

@ClaesBas I tried via GUI, CLI, and manual update file. All result in same error.

The CLI interface does not seem to have a -force option that I saw.

 

[bsotak]

Same happens to me, trying to upgrade from 22.02-MASTER-20220127-092925.

 

[nemaddux]

Same here, trying to upgrade from 22.02-MASTER-20220206-152923.

 

[Phantomsfbw]

Same error here as well...

Error: [EFAULT] Unable to downgrade from 22.02-MASTER-20220206-232922 to 22.02.1-MASTER-20220208-132922

Otherwise the system has been rock solid!

 

[elvisimprsntr]

Seems a fix is in the works, but in order to update to the 22.02.1 nightlies, we have to first upgrade to the latest 22.02 nightly that includes the patch.

https://jira.ixsystems.com/browse/NAS-114710

fixed, but first you'll have to upgrade to the newest 22.02 nightly build (with this patch included), only then yo'll be able to proceed to 22.02.1

I have looked for a newer 22.02 nightly update file, but don't see one

Index of /62993/tnscale/truenas-scale-nightly/

https://update.freenas.org/scale/TrueNAS-SCALE-Angelfish-Nightlies/

Is there some trick via the CLI or another location with the updated 22.02 nightly update file which contains the patch?

UPDATE:

I figured out a work around without downloading a patched 22.02 nightly update file, since one does not seem to exist.

1. SSH into NAS
2. cd /usr/lib/python3/dist-packages/middlewared/plugins/update_/
3. nano utils.py
4. Insert the following after line 15. Re: https://github.com/truenas/middleware/commit/38ee827dc95cdd5637aeda9bf604ba3d48a55604

Code:

        if not x.isdigit() and y.isdigit():
            return True
        if x.isdigit() and not y.isdigit():
            return False

5. reboot
6. SSH into NAS
7. cli
8. system update update
9. exit
10. reboot


Successfully running 22.02.1 nightlies.

 

[bsotak]

Thanks, your workaround of including their update worked for me (without having to go to the latest 22.02).

 

[ato]

Seems a fix is in the works, but in order to update to the 22.02.1 nightlies, we have to first upgrade to the latest 22.02 nightly that includes the patch.

https://jira.ixsystems.com/browse/NAS-114710

I have looked for a newer 22.02 nightly update file, but don't see one

Index of /62993/tnscale/truenas-scale-nightly/

https://update.freenas.org/scale/TrueNAS-SCALE-Angelfish-Nightlies/

Is there some trick via the CLI or another location with the updated 22.02 nightly update file which contains the patch?

UPDATE:

I figured out a work around without downloading a patched 22.02 nightly update file, since one does not seem to exist.

1. SSH into NAS
2. cd /usr/lib/python3/dist-packages/middlewared/plugins/update_/
3. nano utils.py
4. Insert the following after line 15. Re: https://github.com/truenas/middleware/commit/38ee827dc95cdd5637aeda9bf604ba3d48a55604

Code:

        if not x.isdigit() and y.isdigit():
            return True
        if x.isdigit() and not y.isdigit():
            return False

5. reboot
6. SSH into NAS
7. cli
8. system update update
9. exit
10. reboot


Successfully running 22.02.1 nightlies.

It is not working for me:

# diff -u /usr/lib/python3/dist-packages/middlewared/plugins/update_/utils.py{-old,}
--- /usr/lib/python3/dist-packages/middlewared/plugins/update_/utils.py-old 2022-02-04 11:29:25.000000000 +0100
+++ /usr/lib/python3/dist-packages/middlewared/plugins/update_/utils.py 2022-02-10 00:16:46.407458500 +0100
@@ -13,6 +13,11 @@

def can_update(old_version, new_version):
for x, y in itertools.zip_longest(SEP.split(old_version), SEP.split(new_version), fillvalue=''):
+ if not x.isdigit() and y.isdigit():
+ return True
+ if x.isdigit() and not y.isdigit():
+ return False
+
for special in ['CUSTOM', 'MASTER', 'INTERNAL']:
if x == special and y != special:
return False
# cli -c 'system update update'
[0%] Retrieving update manifest...
[0%] Verifying existing update...
[50%] Reading update file...
Error: Unable to downgrade from 22.02-MASTER-20220205-112923 to 22.02.1-MASTER-20220209-152957
.......

 

[elvisimprsntr]

It is not working for me:

did you reboot or restart middleware after making the change?

 

[ClaesBas]

I managed to do a CLI system update manual path="<full path to dowloaded nightly update-file>"...

My nightly version 22.02-MASTER was "borked"...
I booted to my saved "RC2" and added a line with (spaces/indent and) return True at line 16 in utils.py.
(/usr/lib/python3/dist-packages/middlewared/plugins/update_/utils.py)
Rebooted back to my saved "RC2".
SSH:ed in and downloaded latest nightly update file.
(cd somewhere and do a: wget https://update.freenas.org/scale/TrueNAS-SCALE-Angelfish-Nightlies/TrueNAS-SCALE.update )
Get into CLI and do a manual update:
cli
system update manual path="/path-to-where-you-download-it-to/TrueNAS-SCALE.update"
exit
reboot
Wait for the system to restart, and voila: TrueNAS-SCALE-22.02.1-MASTER-20220209-172924 running...

 

[ato]

did you reboot or restart middleware after making the change?

After middleware restart I successfully upgraded. Thanks!

 

[murzik]

not working for me.
middleware errors after reboot.

 

[elvisimprsntr]

not working for me.
middleware errors after reboot.

check your edits to make sure it matches the following

Allow upgrading `22.02-MASTER` to `22.02.1-MASTER` · truenas/middleware@38ee827

TrueNAS CORE/Enterprise/SCALE Middleware Git Repository - Allow upgrading `22.02-MASTER` to `22.02.1-MASTER` · truenas/middleware@38ee827

github.com

 

[stavros-k]

not working for me.
middleware errors after reboot.

Better copy the whole file and paste it into.
Python is sensitive in spaces.

 

[murzik]

I've done it 4 times. Same error over and over again. Middleware won't load , errors in files I did not edit.
Not sure if that matters, but I am trying to switch to nightly from TrueNAS-SCALE-22.02-RC.2