23.10.0 GRUB boot screen after ZFS upgrade

[Sawtaytoes]

I upgraded the boot-pool's ZFS today, and after restarting, I'm hitting a GRUB prompt in both UEFI and Legacy boot modes. Is there something broken about booting to a fully upgraded boot-pool?

Also, any idea how I can fix it?

 

[Sawtaytoes]

I tried reinstalling TrueNAS, and this came up:

Was it a bad USB image or a bug in the installer?

 

[Sawtaytoes]

Here's a `zfs list`. Looks like these are from my boot drives:

Anyone have an idea what I can do? I was going to do the upgrade install + format route, but I don't wanna lose my config.

 

[rudeone]

I also have the same situation with the same scenario.
Upgraded to 23.10.0, updated zfs piol
Next reboot was presented with Grub

 

[probain]

That's rough. I'm also guessing that you did this through shell, and not through the CLI?

Just don't upgrade the boot pool. That will break GRUB.
- @anodos (iXsystems)
source

Are you able to access your pools at all? Live-CD or similar? If so, then you should be able to copy the backed up configs. I don't remember which folder it's in, at the moment. But search the forum for topics like "auto save config" and such. And you should be able to find it. Then you could do a format of the boot-pool and reinstall from scratch.

 

[probain]

I hade extra time, so I took the opportunity to look up the folders I was alluding to.
Basically you should have your configs in the following folders

• /var/db/system/configs-<VERY-LARGE-INTEGER>/config/TrueNAS-Version/<date>.db (daily backups)
• /data/freenas-v1.db (I believe this is the current used one)

So if you could access those folders. Then you could take the configs. And use those.
If you can't access your drives contents, then I don't know what you could do.

There are good and simple ways to backup those configs to a secondary location too. I found this one to be the easiest.

 

[Sawtaytoes]

Thank you so much for your posts @probain! It's good to know, so I don't upgrade the ZFS level next time. I didn't realize the bootloader needed an older ZFS.

After doing the "keep everything and format" route in the installer and having it error there too, I lost all my configs on those drives.

I checked to make sure I had a recent-enough backup config (I did from the 23.10-RC upgrade). I used this to install TrueNAS from scratch. Then I imported my last backup, and it worked! Aside from logs, I don't think I lost anything important. Good thing I keep my apps on a separate zpool!

Still getting these system.journal warnings though. I'm assuming this is some sort of v23 bug:

I don't wanna hyjack my own thread though.

My current version is v23.10.0.1 now.

 

[LarsR]

You can upgrade the zfs version on your data pools, just not the boot-pool. If the boot-pool needs a zfs update, it will be done during the whole OS update process. But if there're no new zfs features you need it's also perfectly fine to not upgrade your data pools.

 

[Sawtaytoes]

My bad. I should've left it alone. Good to know for next time.

 

[probain]

The script I linked earlier. I run that daily as a pre-init script for my cloud backups (encrypted and salted of course). That way I know I should always be at most 24h behind on my configs.

Just make sure to explore and work out your restore plan. And document it. So you can get inte quickly if/when needed. Without having the stress or frustration of a non-working system hanging over you.

 

[bisento]

You can upgrade the zfs version on your data pools, just not the boot-pool. If the boot-pool needs a zfs update, it will be done during the whole OS update process. But if there're no new zfs features you need it's also perfectly fine to not upgrade your data pools.

I ran into the same problem like Sawtaytoes: I got the messages that a ZFS update is available and updated via the Shell with zpool upgrade datapool. After zpool status it showed still an available update for boot-pool and mindlessly I just did it... Now I am greeted with a blank GRUB prompt. Is there a step by step instruction how to rescue the system? I have a TrueNAS Mini X with the latest SCALE. Do I have to make a bootable USB-stick with a TrueNAS live system? with attaching a Monitor an keyboard and switching boot sequence? Is a restore available via a live system GUI? I am currently hitting myself over the head for the reckless zpool upgrade...but sooner or later I will have to tackle the boot problem :-/

 

[Sawtaytoes]

If you have a TrueNAS system, they offer support on those don't they? I'd recommend getting in touch with TrueNAS support.

---

If not, just take a bootable USB-drive (this is how I did it), and when booting into TrueNAS, you have some options to in-place upgrade or fresh reinstall. I had to fresh reinstall because the ZFS version would've still been messed up otherwise.

Once in TrueNAS, I imported my last saved config, and it restarted with the correct configuration.

If you don't have a backup config, you'll have to use their tools to copy it off. Ideally, you'd install TrueNAS into another drive, copy off your config, and then you could overwrite the original boot-pool with a fresh install.

 

[terry5008]

Yeah, I did the same boneheaded move on mine and hosed the boot-pool. Fortunately, I'm running mine in Proxmox and have VM backups set up and scheduled. With the TrueNAS VM shut down I just restored this morning's backup and then restarted it. POOF! Back from the dead!
I LOVE PROXMOX!!!!!

 

[vfiuchcikicshuusrch]

23.10.1 fixed this ?

now I boot from usb flash drive with zfsbootmenu

 

[tprelog]

23.10.1 fixed this ?

There's nothing to fix - You should not upgrade the boot-pool.

 

[vfiuchcikicshuusrch]

I'm already upgrade boot-pool.
proxmox, for example, used systemd-boot, for support new zfs, why truenas not ?

 

[HoneyBadger]

23.10.1 will prevent you from upgrading the boot-pool beyond a certain compatibility level.

It won't fix an already-upgraded one though.

 

[vfiuchcikicshuusrch]

when planed to support lattest zfs version by bootloader ?

 

[vfiuchcikicshuusrch]

23.10.1.3 also can't boot from lattest zfs ?

 

[Etorix]

Never upgrade your boot pool. Full stop.

There will always be new features. By definition, the bootloader from the previous version, which is installed on your boot drive, does not support the new features. —which breaks boot.
One never needs "new features" for a boot drive.