Nextcloud App Storage Configuration Woes

[Policy0702]

Personal Experience
If I must say the obvious. I am new to TrueNAS. Currently I am home user who is starting to do some IT work for individuals and small businesses. I have been using linuxmint as my daily driver for 3+ years now so I have a fairly decent understanding of the concepts but a weaker understanding of the particulars (commands/syntax) on how debian based systems run. I have no experience with distros other than debian based.

Objective
I have a turnkeylinux nextcloud vm running in XCP-ng that I want move to TrueNAS Scale. I also want to add more instances of Nextcloud, (personal/business, etc) so I would like to run several apps simultaneously if that is possible. I must be able to access/backup the nextcloud data in a way that is independent of TrueNAS as well as being able to pull files out of the backups if necessary.

Current Setup
Dell T3600 with an Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (8 cores with hyper-theading) and 31.3GiB (ECC) RAM
TrueNAS-SCALE-22.12.3.3 freshly installed with no important data on it so not a huge deal if I nuke it.
Boot Pool is on a 240G PNY SSD (no redunacy)
Data Pool is 1 vdev with 2 PNY 1TB SSDs striped. (Once I have the money I'll do it right but for now I'll just restore offsite backups in the event of a drive failure)

Attempted Setup
I created a dataset that I want to use exclusively for this one nextcloud app. I created 3 directories inside of that dataset, ~data/, ~db/, ~dbbk/, each for their own mapping to the nextcloud app. I installed the Nextcloud app via the webui and with the Nextcloud data directory mapped to ~data/, Postgress mapped to ~db/, and the backup mapped to ~dbbk/ .

Result
This setup would not work. According to the logs, postgres would come up and be fine but the nextcloud container would seem to loop, spin up, fail, restart, etc. By repeatedly trying to view the logs, I was occasionally able to view them while the container was up. The logs were showing read errors first for /var/www/html/occ. I tweaked permissions on the dataset and was finally able to resolve that, only to have errors by Apache not being able to access .htaccess. I don't remember when or how I got that resolved but eventually I was able to resolve all the errors in the logs for the containers.
I had the best success by exactly matching the file permissions in my dataset to the "ix-applications" dataset.

Current Problem
Nextcloud's login screen loads like expected but I cannot login. I double and triple checked my password but it simply will not work. It just repeatedly asks for my password but does not show any "incorrect password" error. (My active NC does) After enough attempts it blocks my IP address. I can see the login activity in the log for the nextcloud container and somehow it is getting my IP banned but seems unable to write anything else.

My Conclusions/Assumptions
It seems that NC is not able to authenticate my login due not being able to write data to disk. (Why no errors is beyond me)
This most certainly has to do with my mapping the "nextcloud" dataset to the app instead of using the defaults since it works when I leave those fields empty.
I am at a loss of where to go next. I do not understand how data is stored in the ix-applications dataset and I am on the impression that is not what it is for.

Later:
After looking at more forum posts it is looking to me like TrueNAS scale apps are just simply not working well at this point. If no one has a solution I believe I will go back to using VMs on xcp-ng and just use TrueNAS for file storage. I was really looking forward to being able to use some of the extra compute power I have on my TrueNAS machine by running apps or VMs on it.

If anyone has any pointers I would be happy try things.
Thanks!

 

[lailakas]

Hi Policy 0702,

Did you try leaving everything as default? If it works using the ix-application dataset then I suggest you double check the permission of the ~data/ directory for mounting nextcloud data. It MUST be owned by the "www-data" user and group. You said you matched the file permissions in my dataset to the "ix-applications" dataset, which in my case, is root. According to your description, it seems like you've created a directory for the data. I suggest making 3 datasets instead of directories for ~data/, ~db/, ~dbbk/, it is easier to manage permissions on the webui this way. Additionally, you may want to have the database backup dataset on a different pool if possible. I don't know what are the permissions required for database datasets to work, I have those two owned by the "apps" user and group, and full control by "www-data". At least I am not facing any issue.

I recently had my nextcloud migrated from truenas core to truenas scale app. It took a lot effort but It is all working now. Let me know if you run into other issues, our discussion may help someone else finding their way here.

 

[Policy0702]

Thanks lailakas,
Hearing from someone does give me renewed hope.

So, here is what I tried:
I nuked the app and dataset so I was starting from a clean slate and started over with the app install.
I set the created datasets and set the the permissions like this:

Code:

root@truenas[/home/admin]# ls -lR /mnt/PNY\ SSD\ Stripe\ 2x1TB/app-storage
'/mnt/PNY SSD Stripe 2x1TB/app-storage':
total 1
drwx------ 5 root root 5 Aug  4 16:04 nextcloud.aamast.family

'/mnt/PNY SSD Stripe 2x1TB/app-storage/nextcloud.aamast.family':
total 2
drwxrwxrwx 2 www-data www-data 2 Aug  4 16:44 data
drwxrwxrwx 2 apps     apps     2 Aug  4 16:45 db
drwxrwxrwx 2 apps     apps     2 Aug  4 16:04 dbbk

'/mnt/PNY SSD Stripe 2x1TB/app-storage/nextcloud.aamast.family/data':
total 0

'/mnt/PNY SSD Stripe 2x1TB/app-storage/nextcloud.aamast.family/db':
total 0

'/mnt/PNY SSD Stripe 2x1TB/app-storage/nextcloud.aamast.family/dbbk':
total 0
root@truenas[/home/admin]# 

After the installing the app the permissions were like this:

Code:

root@truenas[/home/admin]# ls -la /mnt/PNY\ SSD\ Stripe\ 2x1TB/app-storage/nextcloud.aamast.family
total 19
drwx------  5 root     root      5 Aug  4 16:04 .
drwxr-xr-x  3 root     root      3 Aug  4 16:03 ..
drwxrwxrwx  9 www-data www-data  9 Aug  4 16:59 data
drwx------ 19      999 apps     26 Aug  4 16:59 db
drwxrwxrwx  2 apps     apps      2 Aug  4 16:04 dbbk
root@truenas[/home/admin]# 

Which must meant that the installation of the app changes the owner and permissions of the directory. The dbbk did not change but there is also nothing in it yet.

The behavior is the same as it had been yesterday. Login page loads and looks good but silently eats my passwords without any errors.

In the log for the container

Code:

Application Name:nextcloud-aamast-familyPod Name:nextcloud-aamast-family-86c7d656f-kg757Container Name:nextcloud

I find this associated with the login attempt.

Code:

2023-08-04 22:07:13.590841+00:00172.16.0.1 - - [04/Aug/2023:22:07:13 +0000] "GET /status.php HTTP/1.1" 200 1836 "-" "kube-probe/1.25"
2023-08-04 22:07:13.590908+00:00172.16.0.1 - - [04/Aug/2023:22:07:13 +0000] "GET /status.php HTTP/1.1" 200 1832 "-" "kube-probe/1.25"
2023-08-04 22:07:20.936736+00:00192.168.1.29 - - [04/Aug/2023:22:07:20 +0000] "POST /login HTTP/1.1" 303 2163 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
2023-08-04 22:07:21.075025+00:00192.168.1.29 - - [04/Aug/2023:22:07:20 +0000] "GET /login?direct=1&user=am HTTP/1.1" 200 7643 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
2023-08-04 22:07:21.851952+00:00192.168.1.29 - - [04/Aug/2023:22:07:21 +0000] "GET /cron.php HTTP/1.1" 200 1598 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
2023-08-04 22:07:23.591601+00:00172.16.0.1 - - [04/Aug/2023:22:07:23 +0000] "GET /status.php HTTP/1.1" 200 1832 "-" "kube-probe/1.25"
2023-08-04 22:07:23.591673+00:00172.16.0.1 - - [04/Aug/2023:22:07:23 +0000] "GET /status.php HTTP/1.1" 200 1832 "-" "kube-probe/1.25"

Here is the complete log from the postgress container.

Code:

2023-08-04 21:59:03.583931+00:00The files belonging to this database system will be owned by user "postgres".
2023-08-04 21:59:03.584015+00:00This user must also own the server process.
2023-08-04 21:59:03.584033+00:002023-08-04T21:59:03.584033228Z
2023-08-04 21:59:03.584252+00:00The database cluster will be initialized with locale "en_US.utf8".
2023-08-04 21:59:03.584301+00:00The default database encoding has accordingly been set to "UTF8".
2023-08-04 21:59:03.584333+00:00The default text search configuration will be set to "english".
2023-08-04 21:59:03.584348+00:002023-08-04T21:59:03.584348991Z
2023-08-04 21:59:03.584370+00:00Data page checksums are disabled.
2023-08-04 21:59:03.584400+00:002023-08-04T21:59:03.584400166Z
2023-08-04 21:59:03.584448+00:00fixing permissions on existing directory /var/lib/postgresql/data ... ok
2023-08-04 21:59:03.585826+00:00creating subdirectories ... ok
2023-08-04 21:59:03.586145+00:00selecting dynamic shared memory implementation ... posix
2023-08-04 21:59:03.615375+00:00selecting default max_connections ... 100
2023-08-04 21:59:03.643623+00:00selecting default shared_buffers ... 128MB
2023-08-04 21:59:03.688895+00:00selecting default time zone ... Etc/UTC
2023-08-04 21:59:03.690568+00:00creating configuration files ... ok
2023-08-04 21:59:03.952139+00:00running bootstrap script ... ok
2023-08-04 21:59:04.773233+00:00performing post-bootstrap initialization ... ok
2023-08-04 21:59:06.704764+00:00syncing data to disk ... ok
2023-08-04 21:59:06.704864+00:002023-08-04T21:59:06.704864781Z
2023-08-04 21:59:06.704903+00:002023-08-04T21:59:06.704903461Z
2023-08-04 21:59:06.704925+00:00Success. You can now start the database server using:
2023-08-04 21:59:06.704946+00:002023-08-04T21:59:06.704946177Z
2023-08-04 21:59:06.704966+00:00pg_ctl -D /var/lib/postgresql/data -l logfile start
2023-08-04 21:59:06.705001+00:002023-08-04T21:59:06.705001463Z
2023-08-04 21:59:06.704849+00:00initdb: warning: enabling "trust" authentication for local connections
2023-08-04 21:59:06.705053+00:00You can change this by editing pg_hba.conf or using the option -A, or
2023-08-04 21:59:06.705074+00:00--auth-local and --auth-host, the next time you run initdb.
2023-08-04 21:59:06.785104+00:00waiting for server to start....2023-08-04 21:59:06.784 UTC [48] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2023-08-04 21:59:06.786496+00:002023-08-04 21:59:06.786 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-04 21:59:06.814872+00:002023-08-04 21:59:06.814 UTC [49] LOG:  database system was shut down at 2023-08-04 21:59:04 UTC
2023-08-04 21:59:06.823614+00:002023-08-04 21:59:06.823 UTC [48] LOG:  database system is ready to accept connections
2023-08-04 21:59:06.826364+00:00done
2023-08-04 21:59:06.826417+00:00server started
2023-08-04 21:59:07.278639+00:00CREATE DATABASE
2023-08-04 21:59:07.279525+00:002023-08-04T21:59:07.279525377Z
2023-08-04 21:59:07.279839+00:002023-08-04T21:59:07.279839730Z
2023-08-04 21:59:07.279895+00:00/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2023-08-04 21:59:07.279918+00:002023-08-04T21:59:07.279918589Z
2023-08-04 21:59:07.281858+00:002023-08-04 21:59:07.281 UTC [48] LOG:  received fast shutdown request
2023-08-04 21:59:07.282872+00:00waiting for server to shut down....2023-08-04 21:59:07.282 UTC [48] LOG:  aborting any active transactions
2023-08-04 21:59:07.284594+00:002023-08-04 21:59:07.284 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
2023-08-04 21:59:07.284633+00:002023-08-04 21:59:07.284 UTC [50] LOG:  shutting down
2023-08-04 21:59:07.298563+00:002023-08-04 21:59:07.298 UTC [48] LOG:  database system is shut down
2023-08-04 21:59:07.382269+00:00done
2023-08-04 21:59:07.382328+00:00server stopped
2023-08-04 21:59:07.382506+00:002023-08-04T21:59:07.382506866Z
2023-08-04 21:59:07.382549+00:00PostgreSQL init process complete; ready for start up.
2023-08-04 21:59:07.382566+00:002023-08-04T21:59:07.382566696Z
2023-08-04 21:59:07.414463+00:002023-08-04 21:59:07.414 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2023-08-04 21:59:07.414751+00:002023-08-04 21:59:07.414 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-08-04 21:59:07.414793+00:002023-08-04 21:59:07.414 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-08-04 21:59:07.417102+00:002023-08-04 21:59:07.417 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-04 21:59:07.422287+00:002023-08-04 21:59:07.422 UTC [76] LOG:  database system was shut down at 2023-08-04 21:59:07 UTC
2023-08-04 21:59:07.427461+00:002023-08-04 21:59:07.427 UTC [1] LOG:  database system is ready to accept connections

Now, upon further examination I see that the permissions are being set by the postgress container and that they need to be owned by the user postgress.

The user postgres does not exist on the host system (TrueNAS)
Inside the postgres container, the user postgres does exist and is UID 999
The user www-data is UID 33 on the host system and in both nextcloud and postgres containers.
The user postgres does not exist in the nextcloud container.

I am not sure what to make of this.

 

[Policy0702]

I think I figured it out! Yay!
So, at the end of my last post I actually had a working configuration. The real issue then was browser caching in a strange way because clearing cookies did not help and clearing the cache did not help but I had to clear both the cookies and the cache at the same time. I don't understand that but it does not matter in this context. This is also reminder for me to start with the small things and test stuff with a private/incognito window.

In summary, this is what needs to happen: (I will come back and update this if I find it to not be the case)

1. The nextcloud data needs to be owned by the user and group www-data.
2. There is another user that needs to have access for the installation. I assume it the apps users but I don't know. (Update:Does not seem to be apps, I don't know what it is) I was able to successfully install by allowing "other" full control until the install was complete then I removed all permission for "other".
3. It does not matter who initially owns the postgres directory. Postgres will fix the ownership/permissions on first run. See logs from postgres container in my post above.
4. Currently I do not know what the permissions on the postgres backup directory needs to be. (I will update this if I find out.)
5. Remember to check for browser caching when trouble shooting.

Conclusion:
This does not seem to be best practice but the only way I am able to get it to work is by removing ACLs on the datasets, setting permissions to: user:root, group:root rwxrwxr-x for all directories up to my nextcloud data directory. (so that www-data can traverse those directories). I then set the the nextcloud data directory to user:www-data group:www-data rwxrwxrwx for the install. After the install is complete I was able to set permissions to rwxrwx---.

If someone has a better solution I would like to know about it.

Thanks lailakas for the nudge to keep trying.

Edit: Updated some requirements in the above list.
Edit: 19Aug23 Added "conclusion" above. As well as some other tweeks

 

[lailakas]

As a additional note. Apparently, if you are mounting any extra host path volumes to the Nextcloud app to be used as external storage. Those paths also have to be owned by the "www-data" user and group. Otherwise, even if you give "www-data" full control permission to that path, it will still be shown as an empty folder in Nextcloud.

Be careful that many apps will run chown automatically to change the ownership of mounted paths to the "apps" user and group (can be found in app description), that will cause Nextcloud to fail. For example, file browser. There are several workarounds if you need other apps to mount the same volume.

1. Run all other apps as user www-data (UID 33).
Note: not every app allows you to modify the user that runs it, just like nextcloud. Luckily, Nextcloud is the only app that I use which requires ownership while does not allow changing the running user, so I got away with this workaround.

2. Use Truecharts version of the app instead. Running chown is optional for mounting host path volumes for apps on the Truecharts catalog.
Note: some apps may not function properly without ownership, and there is always risk associated with using a 3rd party catalog. For the apps that I tested (those that runs chown on startup), file browser and various servarr apps run fine without ownership.

3. Share the dataset with NFS/SMB/etc and mount it in Nextcloud.
Note: since you need other apps to access the dataset that is shared, you may need to disable hostPathValidation, which you will risk data corruption. See detailed discussion here. I don't think it will be an issue for small scale use at home though.