Nextcloud postgresql container crash loop

[redsocks157]

Hoping someone can help me out here, because I am not sure exactly what's gone wrong.

This morning, Nextcloud (TrueNAS official version) recommended that I use Redis for file locking. I installed Redis (TrueNAS official version) and then added the env variables for Redis to Nextcloud through the GUI. I clicked "Update", but the app couldn't restart. I managed to get logs which point to a postgresql issue:

Code:

2024-01-12 10:46:20.058369-05:002024-01-12T10:46:20.058369068-05:00
2024-01-12 10:46:20.058423-05:00PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-01-12 10:46:20.058441-05:002024-01-12T10:46:20.058441396-05:00
2024-01-12 10:46:20.094628-05:002024-01-12 15:46:20.094 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2024-01-12 10:46:20.094839-05:002024-01-12 15:46:20.094 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-01-12 10:46:20.094868-05:002024-01-12 15:46:20.094 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-01-12 10:46:20.096844-05:002024-01-12 15:46:20.096 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-12 10:46:20.100899-05:002024-01-12 15:46:20.100 UTC [26] LOG:  database system was shut down at 2024-01-12 13:00:12 UTC
2024-01-12 10:46:20.101250-05:002024-01-12 15:46:20.101 UTC [26] LOG:  invalid resource manager ID 101 at 2/858258C0
2024-01-12 10:46:20.101285-05:002024-01-12 15:46:20.101 UTC [26] LOG:  invalid primary checkpoint record
2024-01-12 10:46:20.101312-05:002024-01-12 15:46:20.101 UTC [26] PANIC:  could not locate a valid checkpoint record
2024-01-12 10:46:20.102390-05:002024-01-12 15:46:20.102 UTC [1] LOG:  startup process (PID 26) was terminated by signal 6: Aborted
2024-01-12 10:46:20.102415-05:002024-01-12 15:46:20.102 UTC [1] LOG:  aborting startup due to startup process failure
2024-01-12 10:46:20.104057-05:002024-01-12 15:46:20.103 UTC [1] LOG:  database system is shut down

If I'm reading this correctly, the postgres database is corrupted! I have no idea how that happened - all I did was add 3 env variables.

I've tried getting into the postgres container shell, but I can't access it. That means I can't run any commands that might recover the database from whatever has happened here. In the nextcloud app settings, it says there is a "Postgres Backup Volume" but I have no idea how to use that to restore my database.

Really hoping someone has an idea here!

 

[Lexden]

Hey I just started getting this same issue... I tried rolling back to previous snapshots of the postgres-data as well as the whole ix-applications dataset, but nothing seems to be getting it into a usable state... Have you figured out anything?
I found this: https://stackoverflow.com/questions...ource-manager-id-in-primary-checkpoint-record
But as you noted, the postgres container shell is inaccessible for some reason. It seems as though the logs stay around, but the pod immediately dies? When I try to list my pods "k3s kubectl get pods", it finds no resources. Doesn't really make sense given that the init-postgresdb container is actively spitting out new logs reporting that it is still waiting for postgres. Whether trying to attach to it via kubectl or via the shell in the apps list, it doesn't seem to find it.

 

[redsocks157]

I ended up installing the TrueCharts version of Nextcloud and haven't had a problem since. This did result in my having tostart over with some things because the database was a mess, but nothing I tried was able to keep postgres online for very long. It would run for awhile, then break down again. All of my files were fine but extensions that stored all of their data in SQL were lost.

Here's a few of the links I found that were related:

• https://sysopspro.com/fix-postgresql-error-panic-could-not-locate-a-valid-checkpoint-record/
• https://www.crunchydata.com/blog/how-to-recover-when-postgresql-is-missing-a-wal-file

Hopefully you have better luck than I did!

 

[Lexden]

That's fantastic, thank you! I coped all of my NextCloud data and I have no extensions, so I think I will have to try installing TrueCharts NextCloud since this is the second time I've had to deal with database issues. Thank you very much!