Recommended Plex workflow with sharing when accounting for HostPath validation?

[Daisuke]

have to disable host path validation for Plex specifically

Like all apps, Plex needs to have the hostPath set to read only, as explained into Kubernetes Applications section from my guide. Then you don't care about any permissions anymore.

 

[kofeyh]

A short response might be warranted:

Our advised solution is to combine SMB+NFS storage and use our "NFS" storage option inside the App storage configuration.
This is not (likely) to be disabled in the future and does not have the same issues has hostPath does.

Fears for databases over NFS are, mostly, unwarranted with TrueCharts:
- We always heavily advice to never change main App config storage, often aptly named "config", away from the "PVC" default.
- Storage for other databases is not user accessible.

When in doubt our support staff is always available to help out on discord :)

Interesting. However is NFS+Apps (ie both k3s and nfs services accessing the same dataset) going to be considered the same as SMB+Apps accessing the same dataset and thus prevented? I am not sure "not likely" really covers it if ix systems are going to go all-in on apps cannot share with other services. I believe a patch is potentially coming to catch the trickery of using nested datasets for apps whilst sharing a parent.

I guess it will simply come down to whether an app accessing an nfs path from the same server is deemed the same apparent security risk; never say never and all that. If the argument is that it's a security concern, then the end goal is eventually only an app or a service can access a given path (not both, irrespective of how that data is accessed).

edit: sorry, trying to not sound like an idiot, probably failing.

 

[oblivioncth]

Interesting. However is NFS+Apps (ie both k3s and nfs services accessing the same dataset) going to be considered the same as SMB+Apps accessing the same dataset and thus prevented? I am not sure "not likely" really covers it if ix systems are going to go all-in on apps cannot share with other services. I believe a patch is potentially coming to catch the trickery of using nested datasets for apps whilst sharing a parent.

I guess it will simply come down to whether an app accessing an nfs path from the same server is deemed the same apparent security risk; never say never and all that. If the argument is that it's a security concern, then the end goal is eventually only an app or a service can access a given path (not both, irrespective of how that data is accessed).

edit: sorry, trying to not sound like an idiot, probably failing.

This is a good question.

At the moment, it is of course SMB only that is blocked; however, that could conceivably change in the future.

TrueCharts own initial blog post on this matter noted:

This safetycheck makes sure apps and sharing services (SMB, NFS, etc) do not use the same data. This is done to avoid permissions issues, as there are a lot of apps that change permissions without giving the user a warning, or just plain do not work with ACL's.

While it could have simply been an accidentally overly broad statement, notice it does mention NFS. Though, I swore I saw something from iXSystems (or at least someone not from TrueCharts) include NFS when talking about Host Path validation as well.

Of course it would be ideal to pick a solution that you know will continue to work for some time, but at this point I think the precedent has been set that while NFS shares to pods or disabling Host Path verification (when reasonable and appropriate) are unlikely to be removed, you shouldn't count on anything 100%.

As for disabling Host Path verification, it's worth noting that there are plans to allow disabling it on a per-app basis in a future SCALE release, which I think would make doing so more appropriate in some circumstances when it's known that a particular app doesn't have issues with ACLs.

 

[chri5]

For me plex has to work and not be a pain. Disabling host path validation for individual apps sounds like a good solution.

 

[truecharts]

Please try not to speculate about TrueCharts features, in this case:
hostPath validation prohibits combining hostPath(!) with ANY other service on the host.
That includes SMB, NFS and so forth.

Our NFS solution loads the NFS server via NFS, NOT via hostPath.
We're using the share normally. Just like the share would be used from any other server. It also uses the same system as Democratic CSI, which is an iX Sanctioned tool for connecting non-scale Kubernetes clusters to storage.

Our NFS feature is not going to be removed, the same case as with hostPath simply does not apply here.
We designed the feature to be in full compliance with iX-Systems.

---
If you want to discuss our features, it's best to reach out directly.
This prevent needless fud being spread on a forum we don't really check that much.

 

[chri5]

Please try not to speculate about TrueCharts features, in this case:
hostPath validation prohibits combining hostPath(!) with ANY other service on the host.
That includes SMB, NFS and so forth.

Our NFS solution loads the NFS server via NFS, NOT via hostPath.
We're using the share normally. Just like the share would be used from any other server. It also uses the same system as Democratic CSI, which is an iX Sanctioned tool for connecting non-scale Kubernetes clusters to storage.

Our NFS feature is not going to be removed, the same case as with hostPath simply does not apply here.
We designed the feature to be in full compliance with iX-Systems.

---
If you want to discuss our features, it's best to reach out directly.
This prevent needless fud being spread on a forum we don't really check that much.

So, are you saying use nfs for your plex library?

 

[truecharts]

So, are you saying use nfs for your plex library?

We stand with our previous advisory, yes.

 

[Daisuke]

Disabling host path validation for individual apps sounds like a good solution.

On Reddit you would be downvoted to death.

 

[chri5]

On Reddit you would be downvoted to death.

I'm slowly changing my mind now I've figured out nfs...

 

[kofeyh]

Our NFS solution loads the NFS server via NFS, NOT via hostPath.
We're using the share normally. Just like the share would be used from any other server. It also uses the same system as Democratic CSI, which is an iX Sanctioned tool for connecting non-scale Kubernetes clusters to storage.

Our NFS feature is not going to be removed, the same case as with hostPath simply does not apply here.
We designed the feature to be in full compliance with iX-Systems.

I will take a look at NFS presentation in lieu of hostpath; as the latter looks like it will eventually be a dead-end street as it's effectively removing one of the key reasons to use in the first place (ie place data somewhere it can be more readily accessed).

It may not be a bad idea to actually show this as a wiki example or something of that nature. NFS isn't used as commonly as it used to be, and I am not super familiar with what I'll need to do, to swap from hostpath to NFS for the very small number of instances where SMB+Hostpath exist currently.

 

[kofeyh]

I'm slowly changing my mind now I've figured out nfs...

I have yet to figure it out..

 

[mistermanko]

I can't say anything in favor of the truecharts NFS solution, but I'm using NFS between my Dockerhost VM and the TrueNAS host just fine. Multiple apps are connection through the Debian VM hosting docker to my plex library via NFS. I guess truecharts' solutions works just as well.

 

[tfcbx]

Yes, another Bluefin thread related to HostPath validation...

I'm trying to move my setup, that I thought was pretty simple, from ESXi that had an Unraid VM with my storage and Ubuntu VM with some services like Plex, but now I'm not sure that was a good idea because of this unfortunate HostPath validation issue.

To be clear, I want to do things "the right way", and keeps things as clean as possible as long a nothing conflicts with my absolute requirements. So, I understand why this check was implemented, and would like to abide by the restriction, but I'm honestly not sure I even can while achieving close what I had before.

There are many specific sub-questions I have in regards to trying to setup my datasets correctly to work with Plex, but let me just keep this simple:

How do people that use Plex on Bluefin regularly add media to their library?

In my situation the person that needs to add media is not that tech savvy, and so the solution needs to be something straightforward that they can do without me.

The way I see it there are several approaches, but because of restrictions most are not actually possible.

1) SMB share the dataset to both users and the Plex container. Containers can't mount SMB shares
2) SMB share the dataset to users, NFS to the Plex container. Mixing protocols leads to potential permissions and file locking issues. In this particular instance I could probably get away with a read only NFS share to Plex, but that might not fly for other containers and still has the overhead of NFS.
3) Set the SMB share as the HostPath and disable validation. As I said, I'd like to not do this. It's not supported and can lead to issues.

So really in the end, I only see three actually options, two of which seem quite flawed to me and the third is very unoptimal:

1) Use the SMB share as the HostPath but keep SMB disabled. Then, when adding files, kill Plex, enable the share and add files, then kill the share and restart Plex. This is awful for obvious reasons as is a nonstarter for me because it's too complicated for the end user. But regardless of that, AFAIK this violates the purpose of HostPath validation anyway because you're ultimately still having the container and share interact with the dataset, with the potential of stomping on each others permissions, just not simultaneously.

2) Set the SMB share as the dataset root (note not the root dataset) as I was planning anyway, and then set Plex's HostPath to a subfolder within that path. According to here and here this is possible to do while leaving HostPath validation on, but this really seems like an unintentional hack to me that still violates the same principle and will likely be patched out in a future release.

3) Share the dataset via NFS only to both the container and users. I believe this is only possible with TrueCharts containers, has the overhead of NFS with large files (which of course movies for plex all are), and requires mounting in windows via the CLI, which means I always have to set up new shares for the users on their systems. This one technically isn't a deal breaking but its quite unsatisfying.

Or finally, say YOLO and disable the validation anyway.

While I get that all of this was done with enterprise in mind, Plex is enormously popular and there's lots of people saying they use it on SCALE, but at this point I'm just not seeing how it's feasible in a clean way because of this. There's plenty of tutorials you can find for setting it up, but most of it is on Angelfin or just shows people adding their media dataset via HostPath with all of their files already magically in the dataset and they make no comment on when the time comes to modify their library.

So again, how do people that use Plex on Bluefin handle managing media in their library while also abiding by these restrictions?

As a complete Truenas novice I really appreciate all the detail you've provided. No doubt there may be many posts on this topic but yours is very informative and complete. That said, I'm curious about #3:

"3) Share the dataset via NFS only to both the container and users. I believe this is only possible with TrueCharts containers, has the overhead of NFS with large files (which of course movies for plex all are), and requires mounting in windows via the CLI, which means I always have to set up new shares for the users on their systems. This one technically isn't a deal breaking but its quite unsatisfying."

I like the NFS option as it appears to work just fine on Truenas. Is there any reason you can't simply share the same directory via SMB and thus allow Windows users easy access? So I'd like to use NFS (Truenas only) and SMB for general access. Does this work or is there a technical detail I'm missing?

Thanks!

 

[oblivioncth]

As a complete Truenas novice I really appreciate all the detail you've provided. No doubt there may be many posts on this topic but yours is very informative and complete. That said, I'm curious about #3:

I like the NFS option as it appears to work just fine on Truenas. Is there any reason you can't simply share the same directory via SMB and thus allow Windows users easy access? So I'd like to use NFS (Truenas only) and SMB for general access. Does this work or is there a technical detail I'm missing?

Thanks!

You can, but there is one main caveat.

The file locking mechanisms for SMB and NFS are completely separate from each other, with no way for one to be aware of the other.

Normally, if two users/processes attempted to access the same file at the same time via the same protocol, the system that requested access second should be notified that the file is already open somewhere else and as such handle this interaction correctly. But since SMB and NFSv4 don't "communicate" with each other, if this same scenario occurred but one access was through SMB and the other NFSv4, the locking mechanism of the one that opened it first wouldn't be respected and you could experience one of two things.

Both accessors trying to modify: Data loss
1) X opens the file through SMB
2) Y opens the file through NFS, system unaware that the file is already being edited
3) Y edits the file then saves it, new data is written to the system (if the same protocol was used this would be prevented)
4) X then finishes their changes and saves the file, overwriting the changes by Y, leaving Y totally unaware this occurred

One accessor trying to modify, the other just read: Receiving corrupt data or the read outright failing
1) X opens the file through SMB
2) Y opens the file through NFS just to read it, system unaware that the file is already being edited
3) X saves edits to the file while Y is busy reading it causing the data stream for Y to become stale/invalid. I don't know exactly how this is handled for each protocol but it would either result in a truncated read or the read outright failing I'd assume.

My guess is that in most cases this would cause an error to occur where one can then simply just try the read again to get the updated data.

Either of these situations occurring are generally unlikely in a home setting with few users, though the chance are a little higher if you have software that regularly access the same paths since you might never be sure if a file is in use or not vs. when it's a person accessing it. That or types of file sharing where it's likely for people to want to access the same data repeatedly (e.g. antiquated document co-authoring).

You can avoid the first example entirely if you are able to do what you need by having one of the protocols access the data as read-only, leaving yourself open only to the potential of the much less serious second example.

So for the Plex example, if you don't care about being able to delete media through the Plex UI (the only reason Plex generally would need write access), then mounting the share through NFS to the Plex chart as read-only, and then through SMB to your desktops as read-write is a pretty good alternative to disabling hostpath validation.

Overall, if you do this, just be aware of the above and make judicious use of restricting write access for one of the protocols wherever possible.

 

[skittlebrau]

This entire debacle with Bluefin gave me the motivation to shift all of my services/apps to a separate server and as a result my whole experience has been much more stable and easier to backup/restore as well.

Like the OP, I want to do things the 'right way' so disabling host path validation was not something I wanted to consider and I wasn't satisfied with the workaround of using filesharing protocols to access files on the host. At that point, I might as well just use a different host.

I applaud the team at Truecharts for doing what they can within the confines of TN SCALE's capabilities; I just think I'm happier overall keeping TN as a storage appliance only.

 

[oblivioncth]

This entire debacle with Bluefin gave me the motivation to shift all of my services/apps to a separate server and as a result my whole experience has been much more stable and easier to backup/restore as well.

Like the OP, I want to do things the 'right way' so disabling host path validation was not something I wanted to consider and I wasn't satisfied with the workaround of using filesharing protocols to access files on the host. At that point, I might as well just use a different host.

I applaud the team at Truecharts for doing what they can within the confines of TN SCALE's capabilities; I just think I'm happier overall keeping TN as a storage appliance only.

I think at that point the potential of SCALE is being wasted a bit and is almost a little overkill; though, I don't blame you, and it's still useful as a nice GUI for setting up snapshots, replication, scrubs, etc. and having everything related to your storage just work.

I think ultimately using NFS shares or disabling hostpath validation on a per-chart basis when you're mounting them as read-only or know that the chart is OK with ACLs will be the way forward and not a big deal at all. While I completely empathize with iXsystems given the complicated nature of this project, I think the main reason why this blew up is because it wasn't like this from the beginning. The fact it was reneged on later is what has caused all of the confusion and frustration.

 

[browntiger]

Every Unix device has this possibility: because samba or nfs are not listening / or fully OS integrated.
Couple of weeks ago, at work, had Oracle SuperCluster M8 locked-up some processing because of NFS mount file share conflict. [reboot fixed it, lol] And ... Not the end of the world.

That was a real world mission-critical part of app.

So, the Plex has an ability to may be temporary lock, when someone at my house is deleting a file via smb or nfs that while plex is processing... BTW the same thing could happen on a windows servers that do have extra APIs processing to avoid conflicts (Yes, CreateFile/NtCreateFile APIs can hang up).

And should I panic? [Or the life goes on?] This happens in Unix world with both NFS and SAMBA.

 

[chri5]

I have yet to figure it out..

Setup an NFS share on your TrueNAS server and give your user (UID 3000) full permissions to the share.

On Windows... (edit if you are on Windows Home edition you can't add the NFS service.)

Install NFS Client (Services for NFS) in Apps > Optional features > More Windows features

Open command prompt and enter the following, replacing the IP and path to the nfs share with yours

Code:

mount -o anon 192.168.00.000:/mnt/slow/nfs N:

This mounts the server's NFS share to local drive letter N:\ on Windows but you wont be able to write to it yet.

Open powershell and paste the following (assuming your UID & GID for the share is 3000).

Code:

New-ItemProperty HKLM:\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default -Name AnonymousUID -Value 3000 -PropertyType "DWord"

New-ItemProperty HKLM:\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default -Name AnonymousGID -Value 3000  -PropertyType "DWord"

You should have drive letter N: on your Windows PC that you can write to.

If you just type mount in command prompt it will show you what has been mounted and which UID & GID are assigned.

Code:

Local    Remote                                 Properties
-------------------------------------------------------------------------------
N:       \\192.168.00.000\mnt\slow\nfs          UID=3000, GID=3000
                                                rsize=1048576, wsize=1048576
                                                mount=soft, timeout=1.6
                                                retry=1, locking=yes
                                                fileaccess=755, lang=ANSI
                                                casesensitive=no
                                                sec=sys

This is a quick solution that solves my plex library host path issue. If this is bad then please show us the way.

 

[kofeyh]

I have yet to figure it out..

I've now figured it out. The only reason I used hostpath was to present a location within storage that was specifically intended to be accessible outside of the container. This is obviously a security concern from an enterprise perspective, and there are some arguments about permission collisions, hence the decision made by ix-systems - but I am not an enterprise user and nobody would (or even should) do such a thing in anger in a business context anyway.

Any apps that have a location that I'd like to have parallel access to (eg media location for plex) now use nfs instead of host path for the app. Each nfs share has permissions set match the uid, gid of the app for NFS and point to the top level directory of the chosen data set; smb (samba) is then used to share this out.

I run several apps, but only two have NFS set up, as the rest have no reason to use shared host paths. It seems somewhat overkill to present shared data to an app from the same server the data is on, via NFS (because this has escaped unscathed thus far) however this, at least for now, seems to be an 'okay' option.

I didn't pick truenas scale because I wanted to have a complex app and storage experience. I would argue that further constraints on 'shared' access to data across services pretty much makes the entire point of running k3s and apps on truenas scale somewhat redundant as a concept, but that is likely a discussion for another day and thread. :)

 

[oblivioncth]

For the record, the workaround of using a parent dataset path as an SMB share and child path under that as a hostpath for an app, which was believed by myself and several others to be an oversight, has indeed been patched out as of 22.12.1.

So:
- Use NFS share if using TrueCharts containers
- Less ideally, disable hostpath validation.