TrueNAS SCALE 22.02-RC.1 Now available!

[Samuel Tai]

Nuked the pool, recreated the pool, reinstalled the apps. Everything working.

That's rather the path of last resort. Should the release notes be updated to recommend starting from a completely fresh install, with a completely fresh pool?

 

[LarsR]

updated without problems, everything came right back up

 

[freqlabs]

There have been a few people in this thread confused about the xattr_compat feature. The formatting of the release notes got really messed up somehow, obscuring the examples of what is ok and what is a problem. The docs team are working on updating the release notes to better communicate the situation.

To clarify:
* enabled is fine
* active is problematic
* xattr_compat is not related to any boot issues

 

[masru]

* xattr_compat is not related to any boot issues

Thanks for clarification. Is the root cause of the boot issues already identified?

 

[j_r0dd]

There have been a few people in this thread confused about the xattr_compat feature. The formatting of the release notes got really messed up somehow, obscuring the examples of what is ok and what is a problem. The docs team are working on updating the release notes to better communicate the situation.

To clarify:
* enabled is fine
* active is problematic
* xattr_compat is not related to any boot issues

Is it possible to make simply enabled after it was active. I never activated this or use xattr's. I have been replicating all the datasets omitting the xattr_compat like so zfs send tank/mydataset | zfs receive -x xattr_fallback -x xattr_compat tank/mydatasetnew. I still have a few TB left. If I'm completely wasting my time please tell me. And if I read the docs correctly, this is no longer even being merged upstream now?

 

[crkinard]

Am I blind or are the SMTP settings hiding from me?

 

[bollar]

Am I blind or are the SMTP settings hiding from me?

Yeah. Click on the Alerts bell icon at the top right. Then the Settings gear icon. Then email.
After then you need to go to Alert Services to confirm the settings.

 

[Kasazn]

I hope there's a hotfix coming like how there was for BETA.1 to BETA.2.

 

[Tomaae]

Update from core was smooth.
Had some issues with dns section in containers, any customization there just gave yaml to xml error and options only had name and no value. Not sure if that is on truenas side or truecharts.
Cloud sync got bit messed up, schedule, source and target was broken in edit window, which was easy to fix. But snapshot cannot be turned on, with error that there are nested datasets, which is not true.
Had to rollback because consistent backups are important for me.
I didnt had time to look up if this was already reported yet.

 

[masru]

There have been a few people in this thread confused about the xattr_compat feature. The formatting of the release notes got really messed up somehow, obscuring the examples of what is ok and what is a problem. The docs team are working on updating the release notes to better communicate the situation.

To clarify:
* enabled is fine
* active is problematic
* xattr_compat is not related to any boot issues

Sorry to ask again, but let me phrase it differently:

Do you iXsystem folks need people with known broken updates to stay on BETA.2 to test a new upgrade implementation when ready?

This is a BETA test after all, so I'm perfectly happy to stay some more time on BETA.2 -- but if you decide in some weeks this is a do-not-fix or documentation-only issue I would be quite unhappy with wasting my time on BETA.2.

And I'm not asking for timelines or detailed information, I just want to know if I should leave my system in the current broken state.

 

[morganL]

Sorry to ask again, but let me phrase it differently:
Do you iXsystem folks need people with known broken updates to stay on BETA.2 to test a new upgrade implementation when ready?
This is a BETA test after all, so I'm perfectly happy to stay some more time on BETA.2 -- but if you decide in some weeks this is a do-not-fix or documentation-only issue I would be quite unhappy with wasting my time on BETA.2.
And I'm not asking for timelines or detailed information, I just want to know if I should leave my system in the current broken state.

@masru Thanks very much for the offer. We don't need you to say in this state, we can reproduce the issue in house.
We are currently working on better documentation that provides more instructions on what to do and how best to avoid the issues. Hope to get it out this week.

 

[HerrV]

The issue seems to be inherent to this release whatever the ZFS implementation.

A couple of days ago, I did a fresh install of the 21.08-BETA.2 with empty ZFS pools. I updated yesterday to the new release but got the infamous "Import ZFS Pool". Just like the others, I observe no issue when booting back on 21.08.

As it is a test server with no major data on it, I obliterated the system and formatted the storage drives. Even with a clean install the issue happens. I did the following steps :

1. Reinstalled the 22.02-RC.1 and it initializes properly ;
2. Reboot, again, no issue to be observed ;
3. Create new ZFS pools on the formatted drives, reboot, no issue to be observed ;
4. Define which pool I want to use for App containers and add TrueCharts repository, reboot, issues with "Import ZFS Pool".

 

[rmr]

This is congruous with what I'm observing. A fresh install of 22.02-RC.1 with subsequent import of the old pools works (including of course rebooting multiple times). When I then import the old configuration, reboot hangs. Further experiments lead me to believe it's the "ix-applications" dataset that causes issues (even though in my case I do not use TrueCharts).
I suspected the 'upgrade' of that dataset (if it was already on the pool and contained old cruft), but in your step #4, there was no prior dataset like that and you're still running into issues, correct?

The issue seems to be inherent to this release whatever the ZFS implementation.

A couple of days ago, I did a fresh install of the 21.08-BETA.2 with empty ZFS pools. I updated yesterday to the new release but got the infamous "Import ZFS Pool". Just like the others, I observe no issue when booting back on 21.08.

As it is a test server with no major data on it, I obliterated the system and formatted the storage drives. Even with a clean install the issue happens. I did the following steps :

1. Reinstalled the 22.02-RC.1 and it initializes properly ;
2. Reboot, again, no issue to be observed ;
3. Create new ZFS pools on the formatted drives, reboot, no issue to be observed ;
4. Define which pool I want to use for App containers and add TrueCharts repository, reboot, issues with "Import ZFS Pool".

 

[HerrV]

This is congruous with what I'm observing. A fresh install of 22.02-RC.1 with subsequent import of the old pools works (including of course rebooting multiple times). When I then import the old configuration, reboot hangs. Further experiments lead me to believe it's the "ix-applications" dataset that causes issues (even though in my case I do not use TrueCharts).
I suspected the 'upgrade' of that dataset (if it was already on the pool and contained old cruft), but in your step #4, there was no prior dataset like that and you're still running into issues, correct?

Indeed, fresh install with fresh ZFS pools.

 

[j_r0dd]

This whole xattr_compat issue leaves a bad taste in my mouth. A change like this never should have made it any form on any build until merged upstream. Breaking samba, nfs, middleware, etc is one thing, but messing with pool data is reckless imo. It was a nightmare moving one of my pools over to ubuntu from scale. Not only do I need to compile the debs from truenas source, but also need to now do some git reverts since it has been disabled in the source. I feel like SCALE was such a community effort with testing and submitting jira tickets to get to this point and now the testers are the one footing the bill on this one. I've been using freenas/truenas for many years now, but this is no bueno. I'm in the process of replicating all my datasets with zfs send tank/mydataset | zfs receive -x xattr_fallback -x xattr_compat tank/mydatasetnew and once done I will dry reimporting the pool with feature not active. Not trying to start drama or anything. Kris was helpful a few weeks back in the discord and one of the devs also was on github. I will report back once done. Only 14TB more to go....

actually just tried on a test pool....
cannot import pool 'k8s': property 'feature@xattr_compat' can only be set to 'disabled' at creation time
So it can not be disabled without destroying. Simply 'enabled' will import just fine, but 'active' is the headache. That is what I am trying to fix on this 1 pool I moved to Ubuntu. I'll update later......

Just to provide an update to this. I finished replicating all my datasets and exported the pool and tried re-importing it and xattr_compat stays active. So this was all a waste of time, just like me testing and submitting Jira tickets for SCALE. Good luck with your data guys.

 

[Kasazn]

Just to provide an update to this. I finished replicating all my datasets and exported the pool and tried re-importing it and xattr_compat stays active. So this was all a waste of time, just like me testing and submitting Jira tickets for SCALE. Good luck with your data guys.

So the solution for now is to completely destroy the pools and do a fresh install of SCALE and start new pools and datasets.

 

[sl-net]

My pools came back online just fine and were originally created in TrueNAS Core, however my applications are displayed as not installed.

However Plex is functioning while NextCloud is not.
Beta2 to RC1 upgrade.

 

[freqlabs]

Thanks for clarification. Is the root cause of the boot issues already identified?

It seems encryption is a common factor. In the system I've been able to inspect, the middleware was stuck in a job trying to recursively unlock the pool.

 

[c77dk]

It seems encryption is a common factor. In the system I've been able to inspect, the middleware was stuck in a job trying to recursively unlock the pool.

Hmm, I had a nested dataset with it's own encryption key so might be something there (don't have that anymore, since it was only a test)

 

[sl-net]

My pools came back online just fine and were originally created in TrueNAS Core, however my applications are displayed as not installed.

However Plex is functioning while NextCloud is not.
Beta2 to RC1 upgrade.

The installed applications appeared after some time. So no dramas. My apologies.