TrueNAS SCALE Bluefin Quality Update

[morganL]

TrueNAS SCALE Bluefin 22.12.0 was released on Dec 13 --- 10 days ago. Since then we've seen over 7,000 early adopters and many reports of success and some reports of issues. Overall, the quality has been very good for a .0 release and has benefited from all the work done by the community on Angelfish. From this base, we're expecting the normal lifecycle improvements with more systems in operation, more testing and more reported issues.

The primary issue being reported is the Host Path validation safety belt. We understand it is annoying to add these checks, but without it some Apps can ignore file permissions and corrupt data. The TrueNAS team has a philosophy of making sure the software is usable by Enterprises for important data, so we can't ignore this risk. The release notes include instructions (and there are many forum posts). We will be checking to see how we can improve this in future releases.

• New security checks are present for host paths in use by various system services. If you have host paths that are shared by multiple system services (e.g. Apps and SMB), please read the 22.12.0 Known Issues and take steps to create unique host paths for each in-use system service.

The known issues in the release notes has been updated with newly found issues .. please read it: https://www.truenas.com/docs/scale/scale22.12/#known-issues

There are a few issues identified in 22.12.0 which are slated for fixing in 22.12.1 which is expected by mid-Feb. All of the issues that say "targeted 22.12 (Bluefin)" were resolved in 22.12.0. So, the remaining count of "known" issues is quite low (<10) and several have workarounds.

In the "Software status" page we now recommend all SCALE systems move to Bluefin. Obviously, Angelfish systems with no known issues may stay there until there until there is a pressing need.

We're looking forward to delivering even better software in 2023. Many thanks for all the tremendous contributions to and from the Community.

Hope everyone has a TRUEly great holiday season.

 

[danb35]

The known issues in the release notes has been updated with newly found issues

You seem to be missing "apps don't exist after upgrade":

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

See also:

k3s not starting after upgrade

After upgrading to Bluefin, my Apps weren't running. When going to Apps -> Installed Apps, it's blank and says: Applications are not running [View Catalog] My pool is set and should be fine still. After a bit I get this error in the alert-area: Failed to configure PV/PVCs support: Cannot...

www.truenas.com

 

[morganL]

You seem to be missing "apps don't exist after upgrade":

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

See also:

k3s not starting after upgrade

After upgrading to Bluefin, my Apps weren't running. When going to Apps -> Installed Apps, it's blank and says: Applications are not running [View Catalog] My pool is set and should be fine still. After a bit I get this error in the alert-area: Failed to configure PV/PVCs support: Cannot...

www.truenas.com

That issue hasn't been reviewed or resolved to a level where it is a known bug... if anyone else sees the same issue please add it to the ticket (and report here).

Its worthwhile checking to see if the host path safety checks are causing the issue.

 

[danb35]

Its worthwhile checking to see if the host path safety checks are causing the issue.

The problem persists with them disabled for a half-hour now.

 

[morganL]

The problem persists with them disabled for a half-hour now.

There's clearly some issue... whether it is configuration or timing related is hard to tell. Truecharts indicated its impacting 1% of users. Are you running a large number of Apps?

Ideally, we need a system that reproduces the issue where the system can be used for diagnostics (outage time doesn't matter). The simpler the system the better.

 

[danb35]

Are you running a large number of Apps?

At the moment I'm running zero apps (which is the problem), but I had been running 23 (though four of those are "external service" apps, which AIUI just act as reverse proxy entries for Traefik). I wouldn't think that counts as a large number, but I really don't know. The system (details in my .sig) certainly ought to be able to handle them, and it has been under Angelfish. But I'd think the debug files from myself and another user (OP on the other thread) on that ticket would at least have enough information to determine where the problem is.

I'll see if I can come up with a reproducible case in a VM, but of course no guarantee there.

 

[morganL]

At the moment I'm running zero apps (which is the problem), but I had been running 23 (though four of those are "external service" apps, which AIUI just act as reverse proxy entries for Traefik). I wouldn't think that counts as a large number, but I really don't know. The system (details in my .sig) certainly ought to be able to handle them, and it has been under Angelfish. But I'd think the debug files from myself and another user (OP on the other thread) on that ticket would at least have enough information to determine where the problem is.

I'll see if I can come up with a reproducible case in a VM, but of course no guarantee there.

23 is a lot..... typical would be <5. Your update time may be much longer than normal and perhaps beyond what we have tested.
Its also more likely with more Apps that there is a configuration issue Bluefin is not handling well. Thanks for any help you can provide.

 

[Kasazn]

I can confirm that apps cannot be run with an exception of TrueChart's PiHole.

 

[browntiger]

Anyone reported that the annoying disk reports hoover button: dashboards > storage > (threee disks) zfs pools not working.

 

[Daisuke]

Works fine for me, from both places: Reporting > ZFS or Dashboard > Storage > Disk Reports.

 

[morganL]

I can confirm that apps cannot be run with an exception of TrueChart's PiHole.

If you do have an issue, please start a separate thread. Lets see if there are any config issues or whether its a software issue.

 

[danb35]

Ideally, we need a system that reproduces the issue where the system can be used for diagnostics (outage time doesn't matter).

I think it's worth pointing out that your engineers have had a test system available for a week now, if they'd actually done anything with that ticket.

 

[browntiger]

Mine does absolutely nothing.... Other than reports link address "http://trunas.local/ui/storage/status/1"

 

[sybreeder]

Can i downgrade to older 22.02 by just booting older kernel? Or any issues may occur after that with docker apps for example ?
I'm keep getting this error message.
Apps are not stable. One reboot works another kubernetes doesn't want to start..

 

[danb35]

Add to the list, "false alert about root user having password disabled". See:

[NAS-119599] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[morganL]

Can i downgrade to older 22.02 by just booting older kernel? Or any issues may occur after that with docker apps for example ?
I'm keep getting this error message.
Apps are not stable. One reboot works another kubernetes doesn't want to start..
View attachment 61545

I'd suggest you start a thread with all the details.

 

[morganL]

Wrapping this thread up now that TrueNAS SCALE 22.12.1 has been released with about 300+ changes.

Hopefully, many of the issues are resolved and remaining ones will be resolved in 22.12.12 which is targetted for end of March.

TrueNAS SCALE Bluefin 22.12.1 has been released!

TrueNAS SCALE 22.12.1 has been released. It includes many new features and improved functionality that span initial effort for high availability (HA) feature support and improvements, and new or improved features in SCALE applications, services, ACLs, and shares. For more details, see the...

www.truenas.com