Bootloader not found after 9.3 to 9.10 upgrade and zpool upgrade.

[soulburn]

I used the GUI to upgrade from 9.3 to 9.10. Things seemed to have gone OK, but after the reboot I did zpool status and it told me I should upgrade "freenas-boot" and my main data volume so I did. After it did the boot volume it told me that I should run "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1". I did that for da12 and da13 (my mirrored boot USB sticks).

After reboot I got a "bootloader not found" error and could only restore via mounting the 9.10 install ISO and doing an upgrade. Now I'm back and things are OK, but apparently the "freenas-boot" volume still needs to be upgraded. Any ideas on what I'm doing wrong so I can upgrade the boot zpool volume to support the new flags in 9.10? Is this something I should even do to the boot volume?

Additionally, "zpool get version" doesn't report the actual version for my boot volume or my main data volume. I never even looked before as I did a fresh install and fresh data volume setup on 9.3 so I don't know if that's normal. Thanks for any help.

Code:

[root@freenas] ~# zpool status
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
  still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
  the pool may no longer be accessible by software that does not support
  the features. See zpool-features(7) for details.
  scan: none requested
config:

  NAME  STATE  READ WRITE CKSUM
  freenas-boot  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  da12p2  ONLINE  0  0  0
  da13p2  ONLINE  0  0  0

errors: No known data errors

Code:

[root@freenas] ~# zpool get version freenas-boot
NAME  PROPERTY  VALUE  SOURCE
freenas-boot  version  -  default

Code:

[root@freenas] ~# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

POOL  FEATURE
---------------
freenas-boot
  multi_vdev_crash_dump
  spacemap_histogram
  enabled_txg
  hole_birth
  extensible_dataset
  embedded_data
  bookmarks
  filesystem_limits
  large_blocks

[root@freenas] ~#

 

[SweetAndLow]

Don't bother with the boot pool freenas version. Pretty sure it will just break everything if you upgrade it.

 

[Eric Blau]

I just did the same "zpool upgrade" on my freenas-boot pool which is a set of mirrored USB sticks as well. I have not run the gpart command yet because I'm not sure if it is partitioned GPT or MBR. My data volume did not report a similar need to upgrade.

I figured that my upgrade to 9.10 was successful, why not proceed with the zpool upgrade.

What should I do at this point to fix this?

 

[dlavigne]

Create a bug report at bugs.freenas.org and post the issue number here.

 

[Ericloewe]

I vaguely remember this sort of thing happening early in the 9.3 days, until the error messages were suppressed. Apparently that little detail was lost during the transition from FreeBSD 9 to 10?

 

[Eric Blau]

https://bugs.freenas.org/issues/14317

Is the bottom line that the "zpool upgrade" should not be run on freenas-boot ever? I have not rebooted one system yet so is there a way for me to recover?

 

[Eric Blau]

@Ericloewe I found this old bug against 9.3:

pool upgrade -a is game over for freenas
https://bugs.freenas.org/issues/6312

I assume this is what you were talking about? I wish I had known about this...

 

[soulburn]

@Ericloewe I found this old bug against 9.3:

pool upgrade -a is game over for freenas
https://bugs.freenas.org/issues/6312

I assume this is what you were talking about? I wish I had known about this...

That sounds like what I experienced. I'm curious, if you run zpool-status on a boot volume in 9.3, does it also tell you that you should upgrade? I guess this bug means leave the boot volume alone, just like recommended by @SweetAndLow above.

 

[Eric Blau]

I never noticed zpool status showing the boot volume as upgradable in 9.3 and I'm sure I looked at the status output in the past.

In bug 6312 there were two potential solutions discussed: 1) update grub so that it can boot off newer ZFS versions or 2) suppress the zpool upgrade message. I can't tell which one was implemented. I'm guessing the zpool upgrade message suppression based on our experience.

@soulburn For recovery, I assume you booted off the 9.10 ISO, re-created your USB sticks and restored your old configuration. Is that how you recovered?

 

[soulburn]

I never noticed zpool status showing the boot volume as upgradable in 9.3 and I'm sure I looked at the status output in the past.

In bug 6312 there were two potential solutions discussed: 1) update grub so that it can boot off newer ZFS versions or 2) suppress the zpool upgrade message. I can't tell which one was implemented. I'm guessing the zpool upgrade message suppression based on our experience.

@soulburn For recovery, I assume you booted off the 9.10 ISO, re-created your USB sticks and restored your old configuration. Is that how you recovered?

Yes it would seem that it's set to suppress, as it appears that the alternative involves hacking GRUB to support the new feature flags. I wouldn't have noticed it except I knew to upgrade my data pool to support the new flags and it just popped up to remind me to upgrade the boot volume, too.

Yes, that's exactly correct. While booted with the ISO it gives the option to upgrade. It says it will wipe the USB sticks but after it does that it automatically applied my config on it's own. Have a config backup ready just in case, though.

 

[Eric Blau]

It is

Yes it would seem that it's set to suppress, as it appears that the alternative involves hacking GRUB to support the new feature flags. I wouldn't have noticed it except I knew to upgrade my data pool to support the new flags and it just popped up to remind me to upgrade the boot volume, too.

Yes, that's exactly correct. While booted with the ISO it gives the option to upgrade. It says it will wipe the USB sticks but after it does that it automatically applied my config on it's own. Have a config backup ready just in case, though.

Did you have a message in zpool status saying to upgrade your data pool? I'm asking because it only showed my boot pool as being upgradable. It said/says nothing about upgrading my data pool.

Thanks for verifying the recovery steps. I will work on this later this evening.

 

[soulburn]

It is


Did you have a message in zpool status saying to upgrade your data pool? I'm asking because it only showed my boot pool as being upgradable. It said/says nothing about upgrading my data pool.

Thanks for verifying the recovery steps. I will work on this later this evening.

To be completely honest I can't remember if the data pool was asking to be upgraded during the 9.3 to 9.10 upgrade as at that point as my mind went cloudy after I rebooted and lost my boot volume. I always just check after a version upgrade to see if there are new feature flags. You may very well be right and I didn't upgrade my data pool and just noticed it for the boot volume.

 

[Blues Guy]

It seems like there was a fix in 9.3 which suppressed the upgrade-message. This fix, however, was not implemented in 9.10 and so zpool showed the message again after upgrading. If you do a zpool upgrade -a, you will not be able to boot the machine again, because GRUB does not support feature flags. A fix is in build testing at the moment (See bug #14317 https://bugs.pcbsd.org/issues/14317)

 

[soulburn]

It seems like there was a fix in 9.3 which suppressed the upgrade-message. This fix, however, was not implemented in 9.10 and so zpool showed the message again after upgrading. If you do a zpool upgrade -a, you will not be able to boot the machine again, because GRUB does not support feature flags. A fix is in build testing at the moment (See bug #14317 https://bugs.pcbsd.org/issues/14317)

Thanks for the update. Glad to see they've merged it into the next build.

 

[Simon Mackenzie]

ran zpool upgrade freenas-boot
without "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1"
and everything appears fine on my mirrored boot disk (2 x USB sticks) after a reboot.

 

[soulburn]

ran zpool upgrade freenas-boot
without "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1"
and everything appears fine on my mirrored boot disk (2 x USB sticks) after a reboot.

Does "zpool status" not complain that the boot volume should be upgraded after you do that?

 

[Simon Mackenzie]

Apparently not.

Code:

[root@towerserver ~]# zpool status freenas-boot
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h6m with 0 errors on Mon Apr 4 14:01:04 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/012caca0-f9dd-11e4-948f-002590f4c81c ONLINE 0 0 0
da8p2 ONLINE 0 0 0

errors: No known data errors
[root@towerserver ~]#

 

[soulburn]

Apparently not.

Code:

[root@towerserver ~]# zpool status freenas-boot
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h6m with 0 errors on Mon Apr 4 14:01:04 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/012caca0-f9dd-11e4-948f-002590f4c81c ONLINE 0 0 0
da8p2 ONLINE 0 0 0

errors: No known data errors
[root@towerserver ~]#

The changes must have been committed in the latest round of updates for 9.10, because now my bool vol is not suggesting upgrade, whereas before (a few weeks ago) it was. Thanks, devs!

 

[styno]

You can indeed upgrade the zpool without the gpart-part. Now if you happen to run the gpart and break your freenas you can actually fix it by booting from your install media and run the update procedure. The Freenas will fix itself and boot again (with the previous config preserved).