Nextcloud cannot access WebUI that worked before

[Kosygor]

Hello

I am positing this here since it seems like k3s problem.
I had working nextcloud app on SCALE. After few hw improvements (I cannot tell exacly when it went down) I cannot acces WEBUI anymore. Strange thing is that sometimes nextcloud gives output to log inside k3s instance sometimes not.
Inside nextcloud.log:

Code:

{"reqId":"FbRjMwpTUaDlAwnHFd1j","level":3,"time":"2021-12-18T23:25:51+00:00","remoteAddr":"172.16.0.1","user":"--","app":"base","method":"GET","url":"/status.php","message":"Failed to start session","userAgent":"kube-probe/1.21","versi>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (output started at /var/w>
{"reqId":"T14ne96lXu7hMOAYegXZ","level":3,"time":"2021-12-18T23:25:57+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Cannot modify header information - headers already sent by (38:50 +0100] "GET /status.php HTTP/1.1" 200 1669 "-" "kube-probe/1.21"
2021-12-18 23:39:00.758525+00:00172.16.0.1 - - [19/Dec/2021:00:39:00 +0100] "GET /status.php HTTP/1.1" 200 1669 "-" "kube-probe/1.21"

In App logs I have

Code:

2021-12-18 23:58:49.813856+00:00AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.16.25.236. Set the 'ServerName' directive globally to suppress this message
2021-12-18 23:58:49.824498+00:00[Sun Dec 19 00:58:49.824376 2021] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.51 (Debian) PHP/8.0.13 configured -- resuming normal operations
2021-12-18 23:58:49.824548+00:00[Sun Dec 19 00:58:49.824385 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
2021-12-18 23:58:53.819704+00:00172.16.0.1 - - [19/Dec/2021:00:58:53 +0100] "GET /status.php HTTP/1.1" 200 1677 "-" "kube-probe/1.21"
2021-12-18 23:58:53.819754+00:00172.16.0.1 - - [19/Dec/2021:00:58:53 +0100] "GET /status.php HTTP/1.1" 200 1677 "-" "kube-probe/1.21"
2021-12-18 23:59:03.633944+00:00172.16.0.1 - - [19/Dec/2021:00:59:03 +0100] "GET /status.php HTTP/1.1" 200 1672 "-" "kube-probe/1.21"
2021-12-18 23:59:03.633999+00:00172.16.0.1 - - [19/Dec/2021:00:59:03 +0100] "GET /status.php HTTP/1.1" 200 1674 "-" "kube-probe/1.21"

One time Truenas itself gave me alert in main dashboard about kube-probe but I did not copied it since then I did not notice that NC was not accesible. Could not reproduce it either.
What I have tried so far:
setting NC into maintenance mode
sudo -u www-data php occ maintenance:repair

Code:

- Repair MySQL collation
     - Not a mysql database -> nothing to do
 - Repair mime types
 - Clean tags and favorites
     - 0 tags of deleted users have been removed.
     - 0 tags for delete files have been removed.
     - 0 tag entries for deleted tags have been removed.
     - 0 tags with no entries have been removed.
 - Repair invalid shares
 - Move .step file of updater to backup location
 - Add move avatar background job
     - Repair step already executed
 - Add preview cleanup background jobs
 - Fix potential broken mount points
     - No mounts updated
 - Repair language codes
 - Install new core bundle components
 - Add log rotate job
 - Clear frontend caches
     - Image cache cleared
     - SCSS cache cleared
     - JS cache cleared
 - Clear every generated avatar on major updates
 - Add preview background cleanup job
 - Queue a one-time job to cleanup old backups of the updater
 - Cleanup invalid photocache files for carddav
 - Add background job to cleanup login flow v2 tokens
 - Remove potentially over exposing share links
     - No need to remove link shares.
 - Clear access cache of projects
 - Reset generated avatar flag
 - Keep legacy encryption enabled
 - Check encryption key format
 - Remove old dashboard app config data
 - Add job to cleanup the bruteforce entries
 - Queue a one-time job to check for user uploaded certificates
 - Repair DAV shares
 - Add background job to set the lookup server share state for users
 - Upgrading Circles App
 - Fix component of birthday calendars
     - 4 birthday calendars updated.
 - Regenerating birthday calendars to use new icons and fix old birthday events without year
     - Repair step already executed
 - Fix broken values of calendar objects
    0 [->--------------------------]
 - Registering building of calendar search index as background job
     - Repair step already executed
 - Register building of social profile search index as background job
 - Registering background jobs to update cache for webcal calendars
     - Added 0 background jobs to update webcal calendars
 - Registering building of calendar reminder index as background job
     - Repair step already executed
 - Clean up orphan event and contact data
     - 0 events without a calendar have been cleaned up
     - 0 properties without an events have been cleaned up
     - 0 changes without a calendar have been cleaned up
     - 0 cached events without a calendar subscription have been cleaned up
     - 0 changes without a calendar subscription have been cleaned up
     - 0 contacts without an addressbook have been cleaned up
     - 0 properties without a contact have been cleaned up
     - 0 changes without an addressbook have been cleaned up
 - Remove activity entries of private events
     - Removed 0 activity entries
 - Clean up old calendar subscriptions from deleted users that were not cleaned-up
    0 [----->----------------------]
     - 0 calendar subscriptions without an user have been cleaned up
 - Fix the share type of guest shares when migrating from ownCloud
 - Copy the share password into the dedicated column
 - Set existing shares as accepted
 - Update OAuth token expiration times
 - Switches from default updater server to the customer one if a valid subscription is available
     - Repair step already executed
 - Send an admin notification if monthly report is disabled
 - Add background job to check for backup codes
 - Populating added database structures for workflows

above commands forced me to increase memory limit since I have got

Code:

PHP Fatal error: Allowed memory size of x bytes exhausted (tried to allocate x bytes) in

by doing:

Code:

/usr/local/etc/php/conf.d/nextcloud.ini
memory_limit=512M

During fixing i have also updatedet app to 23.0.0_11.0.4 version which did not helped.
What am I missing?

Edit: My next discoveries are:
- Sometimes page looks like it is almost there gray background sometimes Cloudflare just spits Error 522.
- apache 2 service was not running after manually starting it nothing changes, also there is nothing in paache2 logifile.
>
In trafeik I have found error:

2021-12-19 19:35:43.828646+00:00time="2021-12-19T20:35:43+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-chmura serviceName=chmura-nextcloud servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=chmura-nextcloud

2021-12-19 19:35:43.828694+00:00time="2021-12-19T20:35:43+01:00" level=error msg="Skipping service: no endpoints found" ingress=chmura-nextcloud namespace=ix-chmura servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=chmura-nextcloud providerName=kubernetes

>
temporarily striking this trough since Apps have accessible storage.
I will try to work with that.
Ok so traefik sometimes gives errors sometimes not. I have noticed that my ssd (used only for VM with Home assistant and Apps) is 91 % full and I have got another thing that I cannot get my head aroud:

Inside TrueNAS shell:

Code:

root@truenas[~]# sudo du -hc --max-depth=1 /mnt/Apps
1.7G    /mnt/Apps/ix-applications
1.7G    /mnt/Apps
1.7G    total
root@truenas[~]# 

Inside VM:

Code:

sudo du -hc --max-depth=1 /

Says that there is total of 42 GB data (except of few dirs in /proc that even root did not had acces to) .
As far as I remeber I have assigned 120 GB to this VM. If I am not wrong this space is reserved for VM only.
1. so 42 GB is around 40 GiB so that matches.
2 I have still about 20 Gib left for apps.
Can this be a problem ? I know that disks tend to run slower when full....

>
Still trying to figure it out. Thought that will deploy another app to check if error is on traefik side od NC.
One lesson learned is that stopping app is not infulencing Traefik config (configs from Ingress config section of Apps are used to make Traefik config file ?).
I have switched Log level to warnings and got this :

Code:

2021-12-20T19:13:00.262068028Z time="2021-12-20T20:13:00+01:00" level=info msg="Configuration loaded from flags."
2021-12-20T19:13:00.282250254Z time="2021-12-20T20:13:00+01:00" level=warning msg="ExternalName service loading is enabled, please ensure that this is expected (see AllowExternalNameServices option)" providerName=kubernetes
2021-12-20T19:13:00.505798410Z time="2021-12-20T20:13:00+01:00" level=error msg="Skipping service: no endpoints found" serviceName=chmura-nextcloud servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=chmura-nextcloud namespace=ix-chmura
2021-12-20T19:13:00.505841108Z time="2021-12-20T20:13:00+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-readarr serviceName=readarr servicePort="&ServiceBackendPort{Name:,Number:8787,}" providerName=kubernetes ingress=readarr
2021-12-20T19:13:08.369824103Z time="2021-12-20T20:13:08+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-chmura providerName=kubernetes serviceName=chmura-nextcloud servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=chmura-nextcloud
2021-12-20T19:13:08.369943578Z time="2021-12-20T20:13:08+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-readarr serviceName=readarr providerName=kubernetes ingress=readarr servicePort="&ServiceBackendPort{Name:,Number:8787,}"
2021-12-20T19:13:17.552253925Z time="2021-12-20T20:13:17+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-chmura serviceName=chmura-nextcloud servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=chmura-nextcloud
2021-12-20T19:13:17.552316702Z time="2021-12-20T20:13:17+01:00" level=error msg="Skipping service: no endpoints found" ingress=readarr namespace=ix-readarr servicePort="&ServiceBackendPort{Name:,Number:8787,}" serviceName=readarr providerName=kubernetes
2021-12-20T19:13:17.557455878Z time="2021-12-20T20:13:17+01:00" level=error msg="Skipping service: no endpoints found" namespace=ix-chmura providerName=kubernetes ingress=chmura-nextcloud serviceName=chmura-nextcloud servicePort="&ServiceBackendPort{Name:,Number:80,}"
2021-12-20T19:13:17.557519348Z time="2021-12-20T20:13:17+01:00" level=error msg="Skipping service: no endpoints found" ingress=readarr namespace=ix-readarr serviceName=readarr servicePort="&ServiceBackendPort{Name:,Number:8787,}" providerName=kubernetes
2021-12-20T19:13:34.015658515Z time="2021-12-20T20:13:34+01:00" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=chmura-nextcloud providerName=kubernetes ingress=chmura-nextcloud namespace=ix-chmura
2021-12-20T19:13:34.015812135Z time="2021-12-20T20:13:34+01:00" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:8787,}" providerName=kubernetes ingress=readarr namespace=ix-readarr serviceName=readarr
2021-12-20T19:13:51.411223511Z time="2021-12-20T20:13:51+01:00" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=chmura-nextcloud namespace=ix-chmura serviceName=chmura-nextcloud providerName=kubernetes
2021-12-20T19:13:51.531750907Z time="2021-12-20T20:13:51+01:00" level=warning msg="SSLRedirect is deprecated, please use entrypoint redirection instead." entryPointName=websecure routerName=readarr-ix-readarr-chmura-sztorm2077-win@kubernetes middlewareName=default-basic-secure-headers@kubernetescrd middlewareType=Headers
2021-12-20T19:13:51.531804462Z time="2021-12-20T20:13:51+01:00" level=warning msg="SSLForceHost is deprecated, please use RedirectScheme middleware instead." entryPointName=websecure routerName=readarr-ix-readarr-chmura-sztorm2077-win@kubernetes middlewareName=default-basic-secure-headers@kubernetescrd middlewareType=Headers
2021-12-20T19:13:51.531879715Z time="2021-12-20T20:13:51+01:00" level=warning msg="SSLRedirect is deprecated, please use entrypoint redirection instead." middlewareType=Headers
2021-12-20T19:13:51.531904667Z time="2021-12-20T20:13:51+01:00" level=warning msg="SSLForceHost is deprecated, please use RedirectScheme middleware instead." middlewareType=Headers

 

[Kosygor]

After investigating by trucharts team this was resolved. Cause was ... change of my external IP.

 

[coltenkrauter]

Did you have to make any changes to fix this? I am seeing similar errors after upgrading to TrueNAS Scale Cobia. My public IP has not changed, it is static.