SOLVED After reboot, boot-pool has unsupported features. I did not upgrade it.

[FrankWard]

I just rebooted my server gracefully and on reboot I'm met with boot-pool import failure. I DID NOT upgrade the boot-pool, and have read to never do so, so I'm not sure how it got upgraded. Any ideas how this could happen?!? I am now facing this error:

The pool uses the following features not supported by this system. org.openzfs:zilsaxattr.

I imported using the suggestion, zpool import -N -f -o readonly=on' and it got me to the prompt. From here, is there any way to remedy the situation? Can I update the boot loader? Am I stuck reinstalling TrueNAS? I've got recent config export from a few days ago.

 

[morganL]

I just rebooted my server gracefully and on reboot I'm met with boot-pool import failure. I DID NOT upgrade the boot-pool, and have read to never do so, so I'm not sure how it got upgraded. Any ideas how this could happen?!? I am now facing this error:



I imported using the suggestion, zpool import -N -f -o readonly=on' and it got me to the prompt. From here, is there any way to remedy the situation? Can I update the boot loader? Am I stuck reinstalling TrueNAS? I've got recent config export from a few days ago.

Please describe system.. including software versions?

Did you update recently? From what to what?

 

[FrankWard]

Please describe system.. including software versions?

Did you update recently? From what to what?

Thanks for reply Captn. Systems are detailed and up to date in my signature. I have this recent configs saved.

• TrueNAS-SCALE-23.10.1-20240205171852 (Cobia)

I was running 23.10.1 today when this occurred. I believe I last upgraded when the release addressing the ZFS corruption bug came out. I have rebooted plenty of times without this issue, so I'm not sure how the boot-pool got upgraded. I do not recall any prompts about upgrading it, and as stated, I'm aware I should never upgrade it.

Is there any documentation on when and where the zfs pool upgrade dialog appears? It's entirely possible I fat-fingered a dialog and didn't even notice. If that's possible, then perhaps the wizards behind the curtain should add that to the list of improvements.

In any case, I was able to get 23.10.1 back online and uploaded my config. All appears well. I just had to put in my ssh key for root.

 

[morganL]

I think there is a plan to remove the risk..... AFAIK it does require a user OK, but we'd prefer to remove the option.

 

[dak180]

I think there is a plan to remove the risk..... AFAIK it does require a user OK, but we'd prefer to remove the option.

There is actually an easy way to do this (at pool creation time, a bit more complicated later but still doable) just using built in ZFS tools: add -o compatibility=grub2 (or any other compatibility flag as appropriate, even a custom list) this will restrict upgrading the pool to only the designated feature flags.

 

[FrankWard]

There is actually an easy way to do this (at pool creation time, a bit more complicated later but still doable) just using built in ZFS tools: add -o compatibility=grub2 (or any other compatibility flag as appropriate, even a custom list) this will restrict upgrading the pool to only the designated feature flags.

Nice. How would you do this to the boot pool when it's created during TN install? Do you have any info on how to do this after the pool has already been created?

 

[Ericloewe]

zpool set compatiblity=grub2 truenas-boot or whatever the boot pool is called.

 

[dak180]

Do you have any info on how to do this after the pool has already been created?

It is just a property that is set on the pool; the only issue is that until iX adds the necessary files (in /usr/share/zfs/compatibility.d) you would need to specify the full path to where ever you put the file with the feature list you want for the pool.

 

[Ericloewe]

Yeah, that merits a feature request. Why are the files not included in TrueNAS?

 

[dak180]

Yeah, that merits a feature request. Why are the files not included in TrueNAS?

I remember adding such a suggestion to the bug tracker but I cannot find it now.

 

[HoneyBadger]

Yeah, that merits a feature request. Why are the files not included in TrueNAS?

They are. I don't recall the exact version they were introduced, but new installs should actually have the compatibility flag on boot-pool set to grub2.

 

[dak180]

They are. I don't recall the exact version they were introduced ...

Is that in scale only because I do not see them in core.

 

[HoneyBadger]

Is that in scale only because I do not see them in core.

Yes, it's in SCALE - I believe upgrades to 23.10.1 and newer will also set this on your boot-pool to prevent inadvertent upgrades from CLI.

 

[probain]

Yes, it's in SCALE - I believe upgrades to 23.10.1 and newer will also set this on your boot-pool to prevent inadvertent upgrades from CLI.

Can confirm that is has been set on my system automatically. I remember noticing and commenting on it on some release announcement some time ago.

 

[dak180]

Yes, it's in SCALE

Are the files somewhere in CORE or will they be added at some point? I can imagine that an improperly upgraded boot pool could prevent side-grading from CORE to SCALE successfully.

 

[Ericloewe]

Most Core boot pools have been incompatible with GRUB for a long time, though I'm not sure what new Core installs are doing.

 

[morganL]

NAS-125154 (its private unfortunately)

Jira

ixsystems.atlassian.net

Fixed in 23.10.2 and beyond.

 

[NugentS]

I don't know if this adds anything but this is what my boot fool looks like. I have like the OP not upgraded it

Code:

  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:01:03 with 0 errors on Thu Feb  8 03:46:05 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdr3    ONLINE       0     0     0
            sda3    ONLINE       0     0     0

errors: No known data errors
root@NewNAS[...gPool/SMB/NewNAS-Scripts/multi-report]# zpool get compatibility
NAME        PROPERTY       VALUE          SOURCE
AppPool     compatibility  off            default
BigPool     compatibility  off            default
SSDPool     compatibility  off            default
ScratchSSD  compatibility  off            default
boot-pool   compatibility  grub2          local

 

[FrankWard]

They are. I don't recall the exact version they were introduced, but new installs should actually have the compatibility flag on boot-pool set to grub2.

Yes, it's in SCALE - I believe upgrades to 23.10.1 and newer will also set this on your boot-pool to prevent inadvertent upgrades from CLI.

Lots of good info on the topic here. However, in regards to of the compatibility flag being automatically set to grub2 in 23.10 and on, how was my system's boot-pool affected? My system was GUI upgraded from 22.12.4.2 Cobia to 23.10.1 & running for a few weeks prior to my boot-pool getting hosed with the "The pool uses the following features not supported by this system. org.openzfs:zilsaxattr." issue. With this compatibility flag set, it should have prevented upgrades from the CLI and/or the GUI right?

 

[HoneyBadger]

Are the files somewhere in CORE or will they be added at some point? I can imagine that an improperly upgraded boot pool could prevent side-grading from CORE to SCALE successfully.

Could you file a feature request along these lines of "ensure CORE -> SCALE upgrade compatibility on boot-pool"?

Most Core boot pools have been incompatible with GRUB for a long time, though I'm not sure what new Core installs are doing.

The average CORE boot-pool will be compatible with GRUB as the unsupported features stay in enabled unless used, but there's two that might have been set to active through manual interaction - those being device_removal through a boot device resilver/change, and zstd_compress - if those are active then GRUB might have a problem loading.

My system was GUI upgraded from 22.12.4.2 Cobia to 23.10.1 & running for a few weeks prior to my boot-pool getting hosed with the "The pool uses the following features not supported by this system. org.openzfs:zilsaxattr." issue. With this compatibility flag set, it should have prevented upgrades from the CLI and/or the GUI right?

I'm confused, that error doesn't make sense as zilsaxattr is an allowed feature for the GRUB2 compatibility set. It shouldn't be complaining about that at all.