SOLVED: Nextcloud doesn't work anymore after upgrade from .14 to .16 on TrueNAS-SCALE-22.12.0

[mihies]

After minor nextcloud (official) upgrade from .14 to .16 it doesn't work anymore - if I click on Web Portal button it opens a web page saying "It can't connect" (not an error from Nextcloud). Otherwise it doesn't report any error, 2/2 pods deployed.

Looking at logs and application events I can't find anything obvious. I've found something odd though. It seems that I have 4 pods. If I look into Shell or Logs, I can see four options for pods (two nextcloud and two nextcloud-progress that differs in postfix numbers). Two of them have containers, two of them does not. It seems to me that I have ended somehow with duplicate pods which is the reason for problems.

Any idea what to look for or how to fix it?

nextcloud-postgres logs

023-01-21 18:30:34.983304+00:002023-01-21T18:30:34.983304058Z
2023-01-21 18:30:34.983362+00:00PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-01-21 18:30:34.983378+00:002023-01-21T18:30:34.983378622Z
2023-01-21 18:30:35.011283+00:002023-01-21 18:30:35.011 UTC [1] LOG: starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2023-01-21 18:30:35.011349+00:002023-01-21 18:30:35.011 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-01-21 18:30:35.011376+00:002023-01-21 18:30:35.011 UTC [1] LOG: listening on IPv6 address "::", port 5432
2023-01-21 18:30:35.012727+00:002023-01-21 18:30:35.012 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-21 18:30:35.015906+00:002023-01-21 18:30:35.015 UTC [26] LOG: database system was shut down at 2023-01-21 18:30:30 UTC
2023-01-21 18:30:35.019128+00:002023-01-21 18:30:35.018 UTC [1] LOG: database system is ready to accept connections

nextcloud/nextcloud logs

2023-01-21 18:30:36.483313+00:00Conf remoteip disabled.
2023-01-21 18:30:36.483344+00:00To activate the new configuration, you need to run:
2023-01-21 18:30:36.483353+00:00service apache2 reload
2023-01-21 18:30:36.562745+00:00AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.16.1.13. Set the 'ServerName' directive globally to suppress this message
2023-01-21 18:30:36.573006+00:00AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.16.1.13. Set the 'ServerName' directive globally to suppress this message
2023-01-21 18:30:36.587776+00:00[Sat Jan 21 18:30:36.587623 2023] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.54 (Debian) PHP/8.1.14 configured -- resuming normal operations
2023-01-21 18:30:36.587797+00:00[Sat Jan 21 18:30:36.587647 2023] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

nextcloud/nginx logs

2023-01-21 18:30:36.260394+00:00/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
2023-01-21 18:30:36.260491+00:00/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
2023-01-21 18:30:36.264807+00:00/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
2023-01-21 18:30:36.312419+00:0010-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
2023-01-21 18:30:36.318336+00:0010-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
2023-01-21 18:30:36.318412+00:00/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
2023-01-21 18:30:36.320464+00:00/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
2023-01-21 18:30:36.321430+00:00/docker-entrypoint.sh: Configuration complete; ready for start up

nextcloud/initdb-postgres logs

2023-01-21 18:30:33.851456+00:00nextcloud-postgres:5432 - no response
2023-01-21 18:30:33.851846+00:00waiting for postgres
2023-01-21 18:30:35.878729+00:00nextcloud-postgres:5432 - accepting connections

Application Events

2023-01-21 19:30:36
Container image "nginx:1.23.3" already present on machine
2023-01-21 19:30:36
Created container nginx
2023-01-21 19:30:36
Started container nginx
2023-01-21 19:30:36
Container image "nextcloud:25.0.3" already present on machine
2023-01-21 19:30:36
Created container nextcloud
2023-01-21 19:30:36
Started container nextcloud
2023-01-21 19:30:34
Created container nextcloud-postgres
2023-01-21 19:30:34
Started container nextcloud-postgres
2023-01-21 19:30:33
Add eth0 [172.16.1.12/16] from ix-net
2023-01-21 19:30:33
Add eth0 [172.16.1.13/16] from ix-net
2023-01-21 19:30:33
Container image "postgres:13.1" already present on machine
2023-01-21 19:30:33
Container image "postgres:13.1" already present on machine
2023-01-21 19:30:33
Created container init-postgresdb
2023-01-21 19:30:33
Started container init-postgresdb
2023-01-21 19:30:32
Scaled up replica set nextcloud-74b7848655 to 1
2023-01-21 19:30:32
Created pod: nextcloud-74b7848655-6cnnz
2023-01-21 19:30:32
Successfully assigned ix-nextcloud/nextcloud-74b7848655-6cnnz to ix-truenas
2023-01-21 19:30:32
Scaled up replica set nextcloud-postgres-689d44d69c to 1
2023-01-21 19:30:32
Created pod: nextcloud-postgres-689d44d69c-msvjm
2023-01-21 19:30:32
Successfully assigned ix-nextcloud/nextcloud-postgres-689d44d69c-msvjm to ix-truenas
2023-01-21 19:30:30
Created container nextcloud-postgres
2023-01-21 19:30:30
Started container nextcloud-postgres
2023-01-21 19:30:30
Stopping container nextcloud-postgres
2023-01-21 19:30:29
Stopping container init-postgresdb
2023-01-21 19:30:28
Add eth0 [172.16.1.10/16] from ix-net
2023-01-21 19:30:28
Container image "postgres:13.1" already present on machine
2023-01-21 19:30:28
Add eth0 [172.16.1.11/16] from ix-net
2023-01-21 19:30:28
Created container init-postgresdb
2023-01-21 19:30:28
Container image "postgres:13.1" already present on machine
2023-01-21 19:30:28
Started container init-postgresdb
2023-01-21 19:30:28
Deleted pod: nextcloud-76d6c58664-sctrv
2023-01-21 19:30:28
Deleted pod: nextcloud-postgres-cdff8f547-gzfrh
2023-01-21 19:30:27
Scaled up replica set nextcloud-postgres-cdff8f547 to 1 from 0
2023-01-21 19:30:27
Created pod: nextcloud-postgres-cdff8f547-gzfrh
2023-01-21 19:30:27
Successfully assigned ix-nextcloud/nextcloud-postgres-cdff8f547-gzfrh to ix-truenas
2023-01-21 19:30:27
Scaled up replica set nextcloud-76d6c58664 to 1 from 0
2023-01-21 19:30:27
Created pod: nextcloud-76d6c58664-sctrv
2023-01-21 19:30:27
Successfully assigned ix-nextcloud/nextcloud-76d6c58664-sctrv to ix-truenas

 

[mihies]

Even more interesting is, that if I install a new nextcloud instance .16 on a different port (default settings, just set Nextcloud host IP to one that server has assigned) and run it, it behaves in the same way - everything looks fine except host:port refuses connection. I assume something with nginx?

 

[mihies]

Next odd thing is that if I try Shell into nextcloud pod, I always end up in the container with the Apache installed (I assume that's nextcloud container) no matter what container I select (nextcloud, nginx or init-postgresdb). Perhaps I'm assuming wrong, but i.e. on nginx container I can't find nginx at all (i.e. service nginx status has no idea of nginx) and all of the three end up in the same starting directory (var/www/html) with same files. This is the same for my original nextcloud and for test instance from the post above.

 

[mihies]

After fiddling around more I've found the cause of the problem. Server has two IPs assigned and until now nextcloud was working perfectly fine with one of those. However, after upgrade it stopped for some reason. It started working again when I've changed IP to the other one in chart settings (Host owner). I'm not sure why exactly it stopped, will check it out later, right now I'm just happy it works again.