TrueCharts Nextcloud won't start after update to 24.0.2_15.2.2, stuck on DEPLOYING 2/3

[radomirpolach]

Code:

$ sudo k3s kubectl get pods -n ix-nextcloud
NAME                               READY   STATUS             RESTARTS       AGE
svclb-nextcloud-zh58q              1/1     Running            0              4h39m
nextcloud-redis-0                  1/1     Running            0              167m
nextcloud-postgresql-0             1/1     Running            0              167m
nextcloud-cronjob-27621850-cdh99   0/1     Completed          0              8m53s
nextcloud-cronjob-27621855-sxjbl   0/1     Completed          0              3m53s
nextcloud-6df8c5bf9f-rz7nz         2/4     CrashLoopBackOff   66 (82s ago)   167m

Code:

2022-07-08 19:34:15.196959+00:00Waiting for notify_push file to be available...
2022-07-08 19:34:25.198465+00:00Notify_push not found... waiting...
2022-07-08 19:34:35.199720+00:00Notify_push not found... waiting...
...
2022-07-08 19:56:35.241216+00:00Waiting for notify_push file to be available...
2022-07-08 19:56:35.241268+00:00Waiting for Nextcloud to start...
2022-07-08 19:56:35.257084+00:00Nextcloud not found... waiting...
2022-07-08 19:56:45.265797+00:00Nextcloud not found... waiting...
...

What is going on?

 

[truecharts]

Code:

$ sudo k3s kubectl get pods -n ix-nextcloud
NAME                               READY   STATUS             RESTARTS       AGE
svclb-nextcloud-zh58q              1/1     Running            0              4h39m
nextcloud-redis-0                  1/1     Running            0              167m
nextcloud-postgresql-0             1/1     Running            0              167m
nextcloud-cronjob-27621850-cdh99   0/1     Completed          0              8m53s
nextcloud-cronjob-27621855-sxjbl   0/1     Completed          0              3m53s
nextcloud-6df8c5bf9f-rz7nz         2/4     CrashLoopBackOff   66 (82s ago)   167m

Code:

2022-07-08 19:34:15.196959+00:00Waiting for notify_push file to be available...
2022-07-08 19:34:25.198465+00:00Notify_push not found... waiting...
2022-07-08 19:34:35.199720+00:00Notify_push not found... waiting...
...
2022-07-08 19:56:35.241216+00:00Waiting for notify_push file to be available...
2022-07-08 19:56:35.241268+00:00Waiting for Nextcloud to start...
2022-07-08 19:56:35.257084+00:00Nextcloud not found... waiting...
2022-07-08 19:56:45.265797+00:00Nextcloud not found... waiting...
...

What is going on?

Just noticed this second thread of yours, this indeed looks like the storage migration procedure hasn't been executed/checked correctly.
We really advice everyone to follow the announcements and directly throw our staff a ticket when something doesn't work out :)

 

[sstruke]

I have the same problem. After the update, nextcloud does not start.

 

[truecharts]

I have the same problem. After the update, nextcloud does not start.

Please apply the data migration procedure manually (preferably using TrueTool) and, if stuck, please file a ticket with our support staff.

 

[chri5]

Please apply the data migration procedure manually (preferably using TrueTool) and, if stuck, please file a ticket with our support staff.

What's TrueTool? My nextcloud doesn't work after update but it has the same issue when there was a bug a few weeks back. Traefik removes the nextcloud ingress and even editing and reapplying ingress doesn't work.

 

[truecharts]

What's TrueTool? My nextcloud doesn't work after update but it has the same issue when there was a bug a few weeks back. Traefik removes the nextcloud ingress and even editing and reapplying ingress doesn't work.

Please be aware that we do not offer support here.
If you encounter any issues please be sure to file a ticket with our support staff.

However: Any issues of the Nextcloud App related to "a few weeks back" are not of relevance anymore.
Because it has completely been rewriten in the mean time, that's what version 15.x.x is all about.

 

[chri5]

Please be aware that we do not offer support here.
If you encounter any issues please be sure to file a ticket with our support staff.

However: Any issues of the Nextcloud App related to "a few weeks back" are not of relevance anymore.
Because it has completely been rewriten in the mean time, that's what version 15.x.x is all about.

The version from 'a few weeks back' is very relevant if it's the only one you can roll back to after the 'completely rewritten' one doesn't work. I have put a ticket in on the discord.

 

[chri5]

Also I was told by your support on discord hold off upgrading when I had trouble with the old version. I saw it had a major update so thought now was the time to upgrade. Sadly that was a mistake, as now, after rolling back, ingress no longer works and I don't have access to nextcloud at all.

 

[truecharts]

The version from 'a few weeks back' is very relevant if it's the only one you can roll back to after the 'completely rewritten' one doesn't work. I have put a ticket in on the discord.

To be very clear:
Rolling back after a major version change, is NOT supported and might lead to dataloss.
This is quite a normal practice with software versioning, for example: Nextcloud itself also does not support this.

While we opened up support for people with issues with the Nextcloud Major update, everyone reverting after a major update likely has permanently broken their Nextcloud installation. Or at least damaged it in ways we likely cannot provide support for.

 

[chri5]

To be very clear:
Rolling back after a major version change, is NOT supported and might lead to dataloss.
This is quite a normal practice with software versioning, for example: Nextcloud itself also does not support this.

While we opened up support for people with issues with the Nextcloud Major update, everyone reverting after a major update likely has permanently broken their Nextcloud installation. Or at least damaged it in ways we likely cannot provide support for.

Great!

 

[truecharts]

Great!

You might be able to figure something out though, not all is lost!
But it would involve manually going over all nextcloud data and seeing what's still there.

In most cases we've seen, the userdata itself and the DB are still salvageable (though we do not support database migrations).
In the previous cases, people had a lot of success saving the usersdata (/var/www/html/data) folder and using that as a hostPath(simple) mountpoint with a new 15.x.x install of Nextcloud.

Right away using that new sweet feature of version 15.x.x to keep the userdata outside of the App by default :)

 

[chri5]

You might be able to figure something out though, not all is lost!
But it would involve manually going over all nextcloud data and seeing what's still there.

In most cases we've seen, the userdata itself and the DB are still salvageable (though we do not support database migrations).
In the previous cases, people had a lot of success saving the usersdata (/var/www/html/data) folder and using that as a hostPath(simple) mountpoint with a new 15.x.x install of Nextcloud.

Right away using that new sweet feature of version 15.x.x to keep the userdata outside of the App by default :)

Well you won't believe this but my nextcloud login prompt appeared after getting a constant 404 for the last 30 min. I seem to be back in and on the latest version. my ingress appeared in Traefik and all seems fine. It might have been my network or a cloudfare glitch.

 

[kofeyh]

I've been (repeatedly) unable to upgrade to any of the 15.x.x versions of this App for Nextcloud, every single update has failed to complete, typically getting stuck on endlessly trying to deploy the same pod, as of the latest update:

Container image "tccr.io/truecharts/nextcloud-fpm:v24.0.2@sha256:402be26ad63d87469fd31bed1056a0e51a7a8a6122a15d7804abc592711cbf3a" already present on machine

Pretty much every single build has been broken in the 15.x.x tree, even the latest app available, 24.0.2_15.2.5, is still busted and ends up stuck in a loop of 2/3 deployed, endlessly trying to redeploy nextcloud-fpm.

I have gone back to the last known-good version I managed to install (14.0.12) rolling back the snapshot as well to preserve data. I am going to hold off upgrading until the code base for 15.x.x has settled; I am not sure why an entirely new branch is being presented as 'stable' as it clearly isn't and should probably have been put into one of the other branches until it was stable enough to reasonably safely allow 14.x users to upgrade.

I am happy to raise this, but knowing where would help.

 

[truecharts]

I've been (repeatedly) unable to upgrade to any of the 15.x.x versions of this App for Nextcloud, every single update has failed to complete, typically getting stuck on endlessly trying to deploy the same pod, as of the latest update:

Container image "tccr.io/truecharts/nextcloud-fpm:v24.0.2@sha256:402be26ad63d87469fd31bed1056a0e51a7a8a6122a15d7804abc592711cbf3a" already present on machine

It's important to leave bugtracing to people with experience reading these logs.
In this case, this is not an error, warning or anything abnormal at all. It just gives info that it already has downloaded the docker container and is not downloading again. Which happens every restart of any App that has not changed docker version.

Pretty much every single build has been broken in the 15.x.x tree, even the latest app available, 24.0.2_15.2.5, is still busted and ends up stuck in a loop of 2/3 deployed, endlessly trying to redeploy nextcloud-fpm.

We want to highlight that this is considered a completely new App, hence the "breaking change".

While we try to help people through migration and even have added some automated migration tooling, we cannot guarantee migration with breaking changes like these.

The correct troubleshooting steps would be:
0. Remove any mention of "nextcloud_datadir" either in env-vars or config.php, we never supported this due to us expecting this change long beforehand
1. Remove any custom storage
2. Ensure config is set to "PVC"
3. Add "127.0.0.1" to the TRUSTED_PROXY setting like this: 172.16.0.0/16 127.0.0.1
5. Stop and Start the old version to double-ensure the settings are applied
6. Apply the update
7. wait for a good while (go grab lunch, eat it and drink a coffee afterwards)
8. Check if it worked out

If not:
9. Manually ensure your nextcloud datastructure is correct inside the App in both /var/www/html and /var/www/html/data
10. Double check if datadirectory is not set to anything other than /var/www/html/data (not set at-all is fine though)
11. restart

If, at this point, it still fails...
The only solution is to file a supportticket with our staff, but most likely you will have to reinstall.

I have gone back to the last known-good version I managed to install (14.0.12) rolling back the snapshot as well to preserve data. I am going to hold off upgrading until the code base for 15.x.x has settled; I am not sure why an entirely new branch is being presented as 'stable' as it clearly isn't and should probably have been put into one of the other branches until it was stable enough to reasonably safely allow 14.x users to upgrade.

We have, by now, reviewed the new version of the App, together with multiple users and staffmembers, and concluded the App itself is generally stable.
There are some minor bugs, but nothing that inherently breaks the App out-of-the-blue.

But: This is not "aimed" to be a smooth upgrade.
Rather: We would heavily advice users to start from scratch (or just reusing userdata) with this new "version".

There are no plans to have a smoother migration path in the future. As in case the above fails, there is something else inherently preventing you form migrating to the new version.

So TLDR:
The only thing that isn't stable, is the migration from older version. Which it was never expected to be.
We're sorry some users are displeased with this, but the Nextcloud App was long due for an overhaul and has generally improved significantly on all fronts.

I am happy to raise this, but knowing where would help.

We generally only offer support via our support ticket system on discord.

 

[chri5]

It would be helpful if the app refused to roll back to a version that is not compatible. As you say it's not supported, not recommended and would likely break things, then close that option. Or give a big warning. I rolled back without thinking, luckily nothing broke (so far).

 

[truecharts]

It would be helpful if the app refused to roll back to a version that is not compatible. As you say it's not supported, not recommended and would likely break things, then close that option. Or give a big warning. I rolled back without thinking, luckily nothing broke (so far).

Actually, after carefull consideration... rollback with snapshot rollback should work...
Not that we support it though.

That being said:
You seem to overestimate what we have control over. We have absolutely zero control over errors and/or (allowing) the rollback system.

We'll leave it at this when it comes to Nextcloud, as we're not here to provide support.
If people need support or ask our devs something, they need to get in contact with our staff directly. This is mostly just a service-account to nudge people that reach out to us in the right direction, nothing more.