Is it possible that some charts (paperless-ng) need ACL?

[indivision]

With the latest version of scale (TrueNAS-SCALE-22.12.1), I moved to NFS for all charts in order to avoid host path validation issues. That has so far worked great for all charts except for one: Paperless-NG.

This was working in the past via hostpath settings (and possible ACL). But, with NFS shares I can't seem to get it to work.

I can see in the logs that it tries to run a bunch of chown operations on the new folders that it creates and gets a permission denied error. To troubleshoot, I've created isolated dataset, user and group just for this chart. I've double-checked the permissions. I added the UID and GID environment variables from the paperless docs. I'm not seeing why it is coming up short on permissions.

I would expect this to be simpler with NFS and no ACL. But, is it possible that this chart uses and needs ACL?

 

[FrostyCat]

If you want to use your own group, make sure you change the fsGroup when deploying the chart, normally it's set to 568 (apps).
If the container tries to do funny stuff with ownership, make sure you also change the uid and gid in the Pod Security Context section when deploying the chart. What you set as environment variables may or may not matter at all.

Let me know if it works, if not I may try it myself and let you know how I got it to work.

 

[indivision]

If you want to use your own group, make sure you change the fsGroup when deploying the chart, normally it's set to 568 (apps).

This is set.

I've used this with every other app that needs external data and it has worked. But, something about NFS + Paperless is more complicated than other apps.

If the container tries to do funny stuff with ownership, make sure you also change the uid and gid in the Pod Security Context section when deploying the chart. What you set as environment variables may or may not matter at all.

I was advised to never change runAsUser or runAsGroup as some apps may depend on specific ID. But, I tried this anyway. :)

I noticed that when I added the UID and GID environment variables, it actually updated runAsUser and runAsGroup as well after install. So, I've been sticking to that method to alter those values.

Let me know if it works, if not I may try it myself and let you know how I got it to work.

Thank you. I do think it needs another set of eyes.

I was able to get it to boot and it looks functional. But, there are still two issues:

A) I had to use a very undesirable ID scheme. The consume folder I use is actually an FTP folder. I have a printer/scanner that sends documents there via FTP and then Paperless processes them and moves them from the consume folder to the documents folder.

I could find no UID/GID combination that would allow Paperless to successfully move the processed files from consume to document storage UNLESS I made Paperless run as the FTP user/group and made both consume and document folders match exactly. Just sharing GID would not work.

B) Even the above does not eliminate all permission errors from the log. It still tries and fails to chown the consume folder:

chown: changing ownership of '/consume/': Operation not permitted

The app appears to be working even with this error. But, I'm not sure if it's breaking something that I will realize down the road. My experience with this app is that it's pretty fragile. Even when using SMB, something would end up fouling it up (updates?) and I would have to re-install and rebuild the document archive.

So, I'm looking to get it installed "perfectly" if possible as it's the kind of app that needs to be stable to be useful.

I probably could change permissions to 777. But, I'd really rather not...

 

[FrostyCat]

Ok, thanks for that, I'm going to give it a shot myself and report back. Normally I expect ACLs to be able to provide the necessary permissions in order to have the documents moved out of that folder.

One last question, what kind of ACLs do you have your dataset set to? (I think the setting is called Share Type, and can be either Generic or SMB).

 

[ksimm1]

Bluefin introduced a new storage driver called overlayfs which has poor performance on chown operations on top of zfs CoW overhead.

Recommendation is to migrate your ix-applications dataset to an SSD pool, which still suffers from the same overhead but much less pronounced than having ix-applications on rust. You can migrate to another pool under Apps -> Settings -> Choose Pool (check Migrate applications to the new pool)

Other container images like LSIO that have large chown operations will also have the issue you're experiencing w/ paperless.

For paperless there is a workaround where you can set the Process User ID (under Change PUID/UMASK values) and fsGroup to 1000, which is Paperless' default user, so that it skips the chown operation entirely. Example below

 

[indivision]

Ok, thanks for that, I'm going to give it a shot myself and report back. Normally I expect ACLs to be able to provide the necessary permissions in order to have the documents moved out of that folder.

One last question, what kind of ACLs do you have your dataset set to? (I think the setting is called Share Type, and can be either Generic or SMB).

I may not have explained the circumstances well. My apologies.

My previous Paperless install (that worked) used "Host Path" for the mounts and it used ACL on one of the mounted paths. That dataset still exists. But, I couldn't find where Share Type is set.

That said, troubleshooting that setup is no longer the goal because using "Host Path" and accessing the same data via SMB has become invalid in the latest versions of Scale. So, I've switched everything to use NFS in place of "Host Path". My understanding is that ACL in SMB was part of the problem. So, using NFS prevents issues with charts due to the use of ACL.

Switching my apps to use NFS was a simple copy and paste since the permissions are all already in place. And doing this has worked for every app except for Paperless. To narrow this problem down, I've created new, separate datasets just for Paperless that are never touched by SMB and have no ACL. They are only NFS shares.

So, the troubleshooting goal now is: Install Paperless using NFS shares for the "consume" and "documents" locations without any permissions errors, ideally with a permissions scheme that relies only on group permissions. Once an app requires user permissions in a certain way it forces users to go with awkward if not insecure schemes.

Bluefin introduced a new storage driver called overlayfs which has poor performance on chown operations on top of zfs CoW overhead.

Recommendation is to migrate your ix-applications dataset to an SSD pool, which still suffers from the same overhead but much less pronounced than having ix-applications on rust. You can migrate to another pool under Apps -> Settings -> Choose Pool (check Migrate applications to the new pool)

Thank you. My apps are on an SSD pool.

Other container images like LSIO that have large chown operations will also have the issue you're experiencing w/ paperless.

For paperless there is a workaround where you can set the Process User ID (under Change PUID/UMASK values) and fsGroup to 1000, which is Paperless' default user, so that it skips the chown operation entirely. Example below

View attachment 64203

Interesting. Changing the Process User ID is a new one for me. I will experiment with that.

In your setup, are you using only NFS shares and are not seeing any chown errors in the log?