New boot image does not activate after upgrade

[rmont]

I am running Truenas Scale 22.02.1. I tried to upgrade a few times to 22.02.2 and now 22.02.3, but the upgrade is not successful. I tried both the automatic upgrade and uploading the new image manually.

After some digging, I found out that the new boot image is properly loaded and set as to be activated after boot:

In reality, after the boot, the original image is loaded again and the new image is not activated.

I tried to check /var/log/messages. These are the first lines after the reboot:

Aug 30 14:47:17 truenas syslog-ng[5393]: syslog-ng shutting down; version='3.28.1' Aug 30 14:52:08 truenas syslog-ng[5425]: syslog-ng starting up; version='3.28.1' Aug 30 14:51:15 truenas kernel: microcode: microcode updated early to revision 0xea, date = 2021-03-08 Aug 30 14:51:15 truenas kernel: Linux version 5.10.109+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Tue May 3 15:35:59 UTC 2022 Aug 30 14:51:15 truenas kernel: Command line: BOOT_IMAGE=/ROOT/22.02.1@/boot/vmlinuz-5.10.109+truenas root=ZFS=boot-pool/ROOT/22.02.1 ro console=ttyS0,115200 console=tty1 libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1

Looks like it's still loading the old image: BOOT_IMAGE=/ROOT/22.02.1

Any suggestions on how to make it load the new image?

Thanks for helping

 

[Arwen]

Looks like Grub is not being properly updated. Could be a quirk with BIOS verses EFI booting.

You could try and see what Grub has for boot entries. If the newer ones are missing, then you have more information. You could even try manually adding the new one for a test.


On a rant: I hate Grub. It's a most convoluted, bloated, annoying tool. The so called modern Grub requires me to manually create my alternate boot environment entries for ZFS, (on Linux, but NOT TrueNAS).

 

[rmont]

Thanks for pointing me in the right direction Arwen.
I fiddled with grub a bit and solved the issue.
I ran update-grub and, while I was at it, I also replaced one of the disks in my boot pool (I was using 2 disks of different sizes and went back to 2 SSDs of the same size).

Initially update-grub didn't fix the issue, but after replacing the disk in the boot pool it did.
Not sure what created the issue, but the good news is that it works now.

Roberto