Failed upgrade from 22.12.4.2 Bluefin to 23.10.1.3 Cobia - Failed to load ZFS modules

[Keefy]

Issue:
When upgrading from 22.12.4.2 to 23.10.1.3 I fail to boot and see this error:

Essentially: Failed to load ZFS modules. Manually load the modules and exit
Followed by a initramfs shell which doesn't accept any keyboard inputs (normally I can navigate with my keyboard fine in the boot selector menu and when booted into 22.12.4.2

What I have tried:
I tried searching this forum and online but couldn't find anything relevant to this issue on TrueNAS Scale, I saw people with similar or the same errors on other Debian based OS, but unfortunately nothing seems to be viable suggestion to solve this issue.
I looked through my system config, and checked for updates to my storage volume ("upgrade"), but as this volume was only created very recently, this isn't the case.
I tried typing into this shell in hopes I could do as requested and "Manually load the modules", but I'm unable to accept keyboard inputs on this screen.
I tried reverting back to 22.12.4.2 and repeating the upgrade 3 times with identical results, each time removing 23.10.1.3 from the boot list and freshly downloading it.

I would greatly appreciate help resolving this as some of my services are inoperable on 22.12.4.2, now requiring 23.x

 

[Jorsher]

Back up your config and do a clean install from the image, maybe from a spare drive if possible so you can quickly switch back to 22 if needed?

 

[Keefy]

Back up your config and do a clean install from the image, maybe from a spare drive if possible so you can quickly switch back to 22 if needed?

It's taken longer than I'd like but this has resolved the issue.
I would like the updates to work without going through this process though.

 

[Jorsher]

It's taken longer than I'd like but this has resolved the issue.
I would like the updates to work without going through this process though.

For sure. That's what we all hope for. I've been using FreeNAS/TrueNAS for over a decade and have never had to do a clean install, but it seemed like you had already spent a bit of time and hadn't had any luck from the leads you followed.

Good to hear you're back up and running.

 

[Ziggy]

Somewhat similarly my bootup is stuck at ‘ZFS: Loaded module v2.2.2-1, ZFS pool version 5000, ZFS files on version 5’. I’ve tried new installations via .iso of Scale 22.10, 23.10, and even 24.04-BETA to no avail. My original problem was webgui and ssh access dropping out within 24 hours And I thought a fresh install might cure the problem. Since the attempt to go to 24.04-BETA told me my boot pool (a SATADOM 16GB) wasn’t big enough - even after attempting to delete previous versions of Scale on it - I foolishly, perhaps, wiped the SATADOM and since then am getting these ZFS errors. I have tried installing all of the above since also on a 128GB SSD with the same results. I am completely stumped now.

 

[Ziggy]

Although, now I think about it, the output is basically saying the existing pools (from previous installations) are using a higher version of ZFS that the module loading. So, how do I get around that?

 

[Jorsher]

I'm not seeing the error about the existing pools being a higher version in the screenshot you listed. Here's from my boot of 23.10:

Code:

[    7.875739] Disabling lock debugging due to kernel taint
[    9.781034] ZFS: Loaded module v2.2.2-1, ZFS pool version 5000, ZFS filesystem version 5
[   10.266328] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[   10.324371] systemd[1]: Inserted module 'autofs4'
[   10.410255] systemd[1]: systemd 252.6-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   10.435452] systemd[1]: Detected architecture x86-64.
[   10.469403] systemd[1]: Hostname set to <truenas>.
[   10.643546] systemd[1]: Queued start job for default target truenas.target.
[   10.690610] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.

What version are you trying to boot when you get an error the pools are too new? 22.10?

 

[Ziggy]

I'm not seeing the error about the existing pools being a higher version in the screenshot you listed. Here's from my boot of 23.10:

Code:

[    7.875739] Disabling lock debugging due to kernel taint
[    9.781034] ZFS: Loaded module v2.2.2-1, ZFS pool version 5000, ZFS filesystem version 5
[   10.266328] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[   10.324371] systemd[1]: Inserted module 'autofs4'
[   10.410255] systemd[1]: systemd 252.6-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   10.435452] systemd[1]: Detected architecture x86-64.
[   10.469403] systemd[1]: Hostname set to <truenas>.
[   10.643546] systemd[1]: Queued start job for default target truenas.target.
[   10.690610] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.

What version are you trying to boot when you get an error the pools are too new? 22.10?

Have tried both 23.10 and 22.12 - same result with the 'ZFS: Loaded module v2.2.2-1, ZFS pool version 5000, ZFS files on version 5’ output. Interestingly (and frustratingly) I have just tried installing Core 13 to see if I could then upgrade to Bluefin - the last stable Scale version for me - from there, but when the installation concludes and I reboot 23 and 22 versions of Scale were still available in the boot menu and no Core. Eventually realised that grub was obviously still pointing to the satadom where I had originally installed Scale (Core was now on another SSD), so I shutdown, pulled that out the satadom and rebooted. I still had multiple attempts to boot Core so, a much shorter version of what I did, I moved the SSD to a SATA port that was recommended for booting in the bios, reinstalled Core using the boot via bios option instead of UEFI, changed the relevant settings in the bios, rebooted, loaded Core in the WebGUI, upgraded to Bluefin Release, and have now started re-importing my pools. It looks like I may have to redo my one Ubuntu VM and the two apps previously installed. Am merely hoping now that the original network issues I was having will not re-occur.

I have a larger 32GB satadom on order. Will then have to figure out how to move the OS to that as I want to use the SATA port the SSD currently sits in as part of a mirrored pool. I'm wondering if dd will function for this, but then how do I point grub to the new satadom?

 

[Ziggy]

Eventually managed a re-install of 22.12 on a separate SSD - had to pull the satcom on which the OS had sat for many years - and so far all is well. Had to redo my one and only ubuntu vm and two apps, so not too painful. Might upgrade to Dragonfish 24 when the first RC is out. Won’t chance the BETA.