TrueNAS SCALE 23.10.0 has been released!

[nasfalse]

I also can't use apps after upgrade from 22.12.4.2. At first the webui shows "initalizing" for a long time, then I unset the pool and try to choose the pool again but the system just reboot. If I try to choose another pool it'll initialize without error.

 

[ChrisRJ]

I have not upgraded the pool so I can go back to the original SSD for further troubleshooting as needed.

A pool upgrade is never something that should just be done, because I am offered the option. My personal view is that I only ever do this, if I really need a new feature. That has not happened for quite a few years. Perhaps this is a somewhat extreme view, but so far it has worked out well. And while the size of my operation is small, it is still in a commercial/professional context. In other words, I have a considerable portion of my money depending on the well-being of my data.

 

[Tony-1971]

Am I the only one affected by apps not starting after the upgrade? I upgraded from 23.10-RC.1 and I see an "Initializing Apps Service" message with a spinning wheel. Moreover, kubectl shows that there are no ix-* namespaces:

Code:

$ k3s kubectl get ns
NAME              STATUS   AGE
default           Active   8h
kube-system       Active   8h
kube-public       Active   8h
kube-node-lease   Active   8h
openebs           Active   8h

Hello,
There are also others with your problem: https://www.truenas.com/community/threads/cobia-final-available.113642/
Not yet upgrade my system because I'm using Storj Node app.
Best Regards,
Antonio

 

[ABain]

Am I the only one affected by apps not starting after the upgrade? I upgraded from 23.10-RC.1 and I see an "Initializing Apps Service" message with a spinning wheel. Moreover, kubectl shows that there are no ix-* namespaces:

Code:

$ k3s kubectl get ns
NAME              STATUS   AGE
default           Active   8h
kube-system       Active   8h
kube-public       Active   8h
kube-node-lease   Active   8h
openebs           Active   8h

I think we already have one bug report of a similar nature : https://ixsystems.atlassian.net/browse/NAS-124776
If you can send me a debug by pm I can get this reviewed.

 

[BossyBear]

A pool upgrade is never something that should just be done, because I am offered the option. My personal view is that I only ever do this, if I really need a new feature. That has not happened for quite a few years. Perhaps this is a somewhat extreme view, but so far it has worked out well. And while the size of my operation is small, it is still in a commercial/professional context. In other words, I have a considerable portion of my money depending on the well-being of my data.

In my case this is s new system with no data on it so no risk.

 

[anodos]

In my case this is s new system with no data on it so no risk.

Just don't upgrade the boot pool. That will break GRUB.

 

[molay]

Hi, Dear Administrator,
I just upgraded TrueNAS SCALE 22.12.4 to 23.10.0 through the Web UI this afternoon.
I found that there is a problem with the in/out real-time information of NIC in the Network widget in the Dashboard page.
It shows that dozens of MiB of data are constantly coming in and out, and even a GiB level display appears.
However, my mini server is only connected to the 1GbE network. This can't happen.
The data of the Network chart in the Report UI is normal.

 

[ABain]

@molay believe this has already been reported: https://ixsystems.atlassian.net/browse/NAS-124779

 

[Tony-1971]

Hello,
Upgrade done without problems (all apps are OK).
Best Regards,
Antonio

 

[LarsR]

Today i had no problems with download speeds, update went smooth, just had to clear browser cache for the disk temp reporting to populate.

 

[Pmantis]

Am I the only one affected by apps not starting after the upgrade? I upgraded from 23.10-RC.1 and I see an "Initializing Apps Service" message with a spinning wheel. Moreover, kubectl shows that there are no ix-* namespaces:

Code:

$ k3s kubectl get ns
NAME              STATUS   AGE
default           Active   8h
kube-system       Active   8h
kube-public       Active   8h
kube-node-lease   Active   8h
openebs           Active   8h

Any fix to this yet? I just upgraded my production system (less important one but I want it working). I tried to reboot, nothing. Rolled back the boot environment to 22.12.3.3, nothing. I tried to rollback DVR/ix-applications, no help. The notification in the UI states:

Failed to configure kubernetes cluster for Applications: Missing "DVR/ix-applications/docker" dataset(s) required for starting kubernetes.
2023-10-27 11:58:52 (America/New_York)

Indeed this is empty, but there is no snapshot for this directory either, so it should be in the main ix-systems dataset snapshot. I checked a snapshot from midnight today and yesterday, which are both empty.

root@dvrnas[/mnt/DVR/ix-applications/docker]# zfs list | grep docker
root@dvrnas[/mnt/DVR/ix-applications/docker]# ls -l
total 0
root@dvrnas[/mnt/DVR/ix-applications/docker]# k3s kubectl get ns
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
root@dvrnas[/mnt/DVR/ix-applications/docker]#
root@dvrnas[/mnt/DVR/ix-applications/docker]# cd ../..
root@dvrnas[/mnt/DVR]# ls -l ix-applications-auto-2023-10-27_00-00-clone/docker
total 0
root@dvrnas[/mnt/DVR]# ls -l ix-applications-auto-2023-10-26_00-00-clone/docker
total 0
root@dvrnas[/mnt/DVR]#

I also restored my configuration from just before the upgrade. No help.
If it makes any difference, I built the system on an encrypted pool, but the apps system created the ix-applications dataset without encryption... and the UI complains that this is not supported. Very odd.

I'm not feeling very good about the apps stability at this point... I'm hoping to get this back soon.

 

[BossyBear]

I was able to upgrade using the downloaded .update file.

 

[joeschmuck]

I did a little test, cloned my active boot configuration (SCALE 22.12.4.1) and set the clone as active, then rebooted. After that was working I switched trains and let her rip. The upgrade seemed to go smoothly.

What issues did I face?
My pool was not online. I had to carefully export it and then import it, that resolved the initial issue.
I noticed my Network report was in alarm condition, yet my network connection was working. The error was [Errno 2] No such file or directory: 'rrdtool' even after a reboot. And of course this applied to all charts as well.

Then I thought, let's upgrade from 23.10-RC1 and see what happens. In RC1 I still have the 'rddtool' error message to start with.
After the upgrade I have the exact same problems.

To be honest, not a smooth upgrade if your pool is not available after the upgrade. If you are not careful you could destroy your pool when you export it. To me this signifies a huge risk.

Now to come clean about my configuration which may play a part in the issues I faced.
I am running on ESXi 7.0 U3 but I don't feel this is the issue.
My pool was created way back when running FreeNAS and I have not rebuilt it nor upgraded any ZFS features in roughly 6 years. I really shouldn't have to completely rebuild my NAS pools nor update the ZFS feature set but it may be a contributor to the problems.
My errors are completely repeatable, on my system.

I know others have witnessed several errors as well.

I did not do any further testing of anything else.

My cautionary tail here is to Not upgrade your ZFS Features so you have the option to roll back to your previous version, and I did successfully rolled back to 22.12.4.1 and all is good in the world. If all works well for you, I'd still wait on the ZFS Feature Set update if there are new features that would benefit you and I'd still wait a good month before updating just to make sure the software is working properly.

For those of you who had a great experience, I'm happy for you, but I need to wait for the next release I guess.

 

[morganL]

FYI.. we've passed 4,000 systems with SCALE 23.10.0. That've very fast uptake in less than 4 days.

There is likely to be a minor update to resolve simple, but annoying issues. We'll be making a decison this week on what and when.

Thanks for reporting any issues you have found.

 

[Arwen]

In someways, any major release should have a plan to release bug fix version within a month of the major release. Just common sense today with the complex software and newer hardware that is available.

I've been in software development, and it was never fun to take a good release and find it has bugs. But, if you know and plan on a bug fix release after Alpha & Beta testers find the bugs, then you can avoid the surprise and disappointment.

Now I am not suggesting / proposing that major changes be made, even if some consider it a bug. But, that a purely bug release be planned and if needed, released, BEFORE the next major release.

 

[morganL]

In someways, any major release should have a plan to release bug fix version within a month of the major release. Just common sense today with the complex software and newer hardware that is available.

I've been in software development, and it was never fun to take a good release and find it has bugs. But, if you know and plan on a bug fix release after Alpha & Beta testers find the bugs, then you can avoid the surprise and disappointment.

Now I am not suggesting / proposing that major changes be made, even if some consider it a bug. But, that a purely bug release be planned and if needed, released, BEFORE the next major release.

Agreed.

However, we need to be clear that only certain types of bugs can be fixed without going through a normal QA cycle. Anything complex and critical will have to wait.

 

[danb35]

It would be good if you could at least explain what the issue is/was (as it has apparently been resolved). Having gone through complete loss of all my apps when I migrated from Angelfish to Bluefin, I have no appetite for that again. And it looks like this time you have multiple users with the same problem.

 

[morganL]

It would be good if you could at least explain what the issue is/was (as it has apparently been resolved). Having gone through complete loss of all my apps when I migrated from Angelfish to Bluefin, I have no appetite for that again. And it looks like this time you have multiple users with the same problem.

Yes, we'll be adding to the release notes.

There is one major issue for the situation where the Apps dataset is unencrypted but it located on a fully encrypted pool. This is not the recommended configuration, but several users have seen it.

I expect this to be resolved in the minor update: https://github.com/truenas/middleware/pull/12402

 

[joeschmuck]

@morganL I hope you take our criticism with the intent it is given. I don't feel that most of us are upset or irritated with the release, concerned is my perspective and of course cautioning other users before they take the plunge, that is part of a caring community. I have no doubt that iXsystems will release an update fairly quick, that is what normally happens. And I sincerely understand proper QA. My concern at this time is people having pools that fail to mount and applications that fail to run. Thankfully none if the issues caused a loss of data.

I'm curious if part of the problem could be not as many beta testers (aka RC1 testers). I haven't been tracking it and I failed to report the issues I had when I tested RC1 because I figured they were so obvious that they would be reported, but I must have been wrong. When 23.10.1 is released I will test it and report any problems.

So please take our comments constructively if you can.

 

[Tomasz Elendt]

There is one major issue for the situation where the Apps dataset is unencrypted but it located on a fully encrypted pool. This is not the recommended configuration, but several users have seen it.

I'm not too fond of this message. It indicates that the users affected by the apps not starting issue in the latest release did something wrong.

My whole pool is encrypted. And if I recall correctly, I didn't create the ix-applications dataset myself - the Apps automatic setup did that.
I'm pretty sure there was no warning triggered at any step that would tell me that I did something wrong.

I'm curious if part of the problem could be not as many beta testers (aka RC1 testers).

And here's the interesting part - I ran RC1! And my apps were running stable there (I rebooted the server multiple and every time things started correctly).
It's the upgrade to the final that broke things. Which means the bug was introduced there.

That should never happen. There should be hardly any changes between RC and final - no new features introduced, no non-critical changes.

I really don't want any iXsystems to feel bad. I paid nothing to anyone and I have no right to demand anything. All I ask is to not blame it on users.