Understanding the TrueNAS SCALE "hostPathValidation" setting

[HoneyBadger]

What is the “hostPathValidation” setting?

With the recent release of TrueNAS SCALE "Bluefin" 22.12.1, there have been a number of reports of issues with the Kubernetes "hostPathValidation" configuration setting, and requests for clarification regarding this security measure.

The “hostPathValidation” check is designed to prevent the simultaneous sharing of a dataset over a file-level protocol (SMB/NFS) while also being presented as hostPath storage to Kubernetes. This safety check prevents a container application from having the ability to accidentally perform a change in permissions or ownership to existing data in place on a ZFS dataset, or overwrite existing extended attribute (xattr) data, such as photo metadata on MacOS.

What’s the risk?

Disabling the hostPathValidation checkbox under Apps -> Settings -> Advanced Settings allows for this “shared access” to be possible, and opens up a small possibility for data loss or corruption when used incorrectly.

For example, an application that transcodes media files might, through misconfiguration or a bug within the application itself, accidentally delete an “original-quality” copy of a file and retain the lower-resolution transcoded version. Even with snapshots in place for data protection, if the problem is not detected prior to snapshot lifetime expiry, the original file could be lost forever.

Users with complex ACL schemes or who make use of extended attributes should take caution before disabling this functionality. The same risk applies to users running CORE with Jails or Plugins accessing data directly.

A change of this manner could result in data becoming unavailable to connected clients; and unless the permissions were very simplistic (single owner/group, recursive) reverting a large-scale change would require reverting to a previous ZFS snapshot. If no such snapshot exists, recovery would not be possible without manually correcting ownership and permissions.

When was this setting implemented?

In the initial SCALE release, Angelfish 22.02, there was no hostPathValidation check. As of Bluefin 22.12.0, the hostPathValidation setting was added and enabled by default. A bypass was discovered shortly thereafter, which allowed users to present a subdirectory or nested dataset of a shared dataset as a hostPath without needing to uncheck the hostPathValidation setting - thus exposing the potential for data loss. Another bypass was to stop SMB/NFS, start the application, and then start the sharing service again.

Both of these bypass methods were unintended, as they exposed a risk of data loss while the “hostPathValidation” setting was still set. These bugs were corrected in Bluefin 22.12.1, and as such, TrueNAS SCALE Apps that were dependent on these bugs being present in order to function will no longer deploy or start unless the hostPathValidation check is removed.

What’s the future plan for this setting?

We have received significant feedback that these changes and the validation itself have caused challenges. In a future release of TrueNAS SCALE, we will be moving away from a system-wide hostPathValidation checkbox, and instead providing a warning dialog that will appear during the configuration of the hostPath storage for any TrueNAS Apps that conflict with existing SMB/NFS shares.

Users can make the decision to proceed with the hostPath configuration at that time, or cancel the change and set up access to the folder through another method.

If data must be shared between SMB and hostPath, how can these risks be mitigated?

Some applications allow for connections to SMB or NFS resources within the app container itself. This may require additional network configuration, such as a network bridge interface as described in the TrueNAS docs “Accessing NAS from a VM” as well as creating and using a user account specific to the application.

https://www.truenas.com/docs/scale/scaletutorials/virtualization/accessingnasfromvm/

Users who enable third-party catalogs, such as TrueCharts, can additionally use different container path mount methods such as connecting to an NFS export. Filesystem permissions will need to be assigned to the data for the apps user in this case.

We at iX appreciate your feedback.

 

[Back that NAS up]

Thank you for listening and all your teams work.

 

[NickF]

This is the perfect response :)

 

[neofusion]

For example, an application that transcodes media files might, through misconfiguration or a bug within the application itself, accidentally delete an “original-quality” copy of a file and retain the lower-resolution transcoded version. Even with snapshots in place for data protection, if the problem is not detected prior to snapshot lifetime expiry, the original file could be lost forever.

I am probably misunderstanding this part, but I fail to see how the quoted example is specific to "shared access"-scenarios or indeed how keeping the hostpath setting enabled would prevent it.

If we imagine a path that is uniquely used by an single application, a path that is for the sake of argument not accessed by any other protocol or service. What would prevent your scenario from happening there?
My answer to that would be: Nothing. If something has write access to a file that something can destroy or corrupt the file.

That kind of dataloss, either due to user error or misbehaving application, is why you have snapshots and external backups that you regularity vet.

While I think this thread is long overdue, I do appreciate the official statement.
In my use case I chose to disable the hostpath-check. My apps only have read access to the share anyway, so it's a non-issue.

 

[morganL]

I am probably misunderstanding this part, but I fail to see how the quoted example is specific to "shared access"-scenarios or indeed how keeping the hostpath setting enabled would prevent it.

If we imagine a path that is uniquely used by an single application, a path that is for the sake of argument not accessed by any other protocol or service. What would prevent your scenario from happening there?
My answer to that would be: Nothing. If something has write access to a file that something can destroy or corrupt the file.

That kind of dataloss, either due to user error or misbehaving application, is why you have snapshots and external backups that you regularity vet.

While I think this thread is long overdue, I do appreciate the official statement.
In my use case I chose to disable the hostpath-check. My apps only have read access to the share anyway, so it's a non-issue.

The issue is that someone might share a dataset with valuable data with an App... and then find that data is unexpected deleted.
We wanted a mechanism to warn people that this was possible.

However, we agree that there are other ways of having a similar issue. External backup is recommended, but we know that many users may not have this setup. Snapshots do help, but require the user to realize the data has been deleted and then revert the key data.

It's one of those "Damned if you do and damned if you don't" situations.

 

[sammael]

Excuse my inability to grasp this, but I read this thread and the warning it gives when you switch it off, and I still don't really get what the setting does.

I have pool named apps and dataset named app_data. Under the dataset app_data are several subdirectories that are assigned to apps as host paths. Neither of these is shared. Then there is one subdirectory that is NOT used by any app AND is shared via nfs.

In my understanding in this scenario the warning does not really apply to me and no possibility of data loss exists? Could you just please confirm if this is true, or if I completely misunderstood the issue and the shared subfolder matters even if not used by an app? In case it does, is really all that is needed for truenas to not complain to move the subdir from under the dir where apps have their host paths?

Thanks!

 

[NickF]

Excuse my inability to grasp this, but I read this thread and the warning it gives when you switch it off, and I still don't really get what the setting does.

I have pool named apps and dataset named app_data. Under the dataset app_data are several subdirectories that are assigned to apps as host paths. Neither of these is shared. Then there is one subdirectory that is NOT used by any app AND is shared via nfs.

In my understanding in this scenario the warning does not really apply to me and no possibility of data loss exists? Could you just please confirm if this is true, or if I completely misunderstood the issue and the shared subfolder matters even if not used by an app? In case it does, is really all that is needed for truenas to not complain to move the subdir from under the dir where apps have their host paths?

Thanks!

You are understanding correctly. Unless you have more than one app mapped to the same "subdirectory" dataset, you are fine. One of the issues is that there is a concept called a file lock and the container/app will have no knowledge of another's use of a file through a share. SMB makes this worse because Windows and Linux handle this very differently.

ps I want to believe.

 

[morganL]

You are understanding correctly. Unless you have more than one app mapped to the same "subdirectory" dataset, you are fine. One of the issues is that there is a concept called a file lock and the container/app will have no knowledge of another's use of a file through a share. SMB makes this worse because Windows and Linux handle this very differently.

ps I want to believe.

That would be a good addition the post... Apps that modify files are more at risk than those that just add files.
@HoneyBadger

 

[NickF]

@anodos I am sure can articulate it better than me lol.

 

[Rarity]

Help me out with a real world example. I am using Plex. Plex has access to my video data and let's me watch my videos through its web interface. Awesome. However, I will add more videos to my collection over time. Plex offers no way to upload content through its web interface; I am meant to go behinds the scenes so to speak and put my content there. The only way I know to upload files is to run samba on my TrueNas system. That's worked for years. I upload my files, Plex does its web stuff, we're all happy. These recent changes make it so that I can't have this workflow anymore. TrueNas is now upset that I dare share data with Plex AND samba at the same time. One solution would be to have only one on at a time, but that's not acceptable. I could toggle that warning off, but how long will that toggle be available before some update removes that ability?

I can't be the only person with a workflow where I wish to upload content behind the scenes and have some service deliver that content for me, right? How am I meant to upload data to my NAS now that samba is dead?

 

[sammael]

@Rarity I am pretty much in the same boat, with a lot more apps, and have the hostpathcheck off since it came out and never had single issue (this is not to say you or others wouldn't. I still don't fully understand the issue, but it would seem in my usecase it's a moot point).

I recently reinstalled most of the apps to truecharts apps and converted host mounted storage to pvc (or in case of the -arrs into nfs shares), but some apps I failed to convert so 2 or 3 still remain with host path mounted.

I ran radarr, sonarr, lidarr, bazarr, prowlarr, qbittorrent, navidrome and jellyfin with host paths to /mnt/pool1/media_vault, while /mnt/pool1/media_vault is being shared via samba. Without going into incriminating details, there's thousands of media files that were accessed by all the apps constantly and by other machines in household via samba, and not a single time was there any conflict. It is absolutely possible that neither of these apps/machines happened to try to access single file at the same time yet and I am just lucky. I would not think they'd implement the checkbox in the 1st place if they planned on removing it.
edit: for timeframe we're talking 2+ years and even more for some apps.

Take this as my experience not advice.

 

[Rarity]

@sammael Maybe I'm misremembering, but I could have sworn I tried the strategy of top/app1, top/app2 where each app gets their own folder and I share samba with the top folder. That strategy used to work, but due to an update I'm sure that's now no longer possible. The error message now catches the overlap. I'm in the middle refactoring my zfs stuff anyways so nothing is running at the moment so I can't verify what I've said at the moment.

 

[anodos]

@sammael Maybe I'm misremembering, but I could have sworn I tried the strategy of top/app1, top/app2 where each app gets their own folder and I share samba with the top folder. That strategy used to work, but due to an update I'm sure that's now no longer possible. The error message now catches the overlap. I'm in the middle refactoring my zfs stuff anyways so nothing is running at the moment so I can't verify what I've said at the moment.

That strategy was exploiting a validation bug that was since fixed.

 

[sammael]

@Rarity I finally migrated all apps to use PVC or NFS share instead of host path and turned host path checking on on both Scales for peace of mind, but this was more of a something I wanted to do to consolidate apps spread around various docker host's vms and raspberrys onto single host instead of something I felt was important to protect my data (this is a home media server which if it died I'd be sad, but it wouldn't be the end of the world)

I had the host path check off since the update it came in, with multiple users and apps accessing the same data via both nfs and samba and never had a single issue. What I'm trying to say is unless it's a production mission critical data just turn it off and forget about it if it's a homelab or home media server.

 

[morganL]

The next phase of per App checking is being delayed into the Cobia release because of its complexity.
https://ixsystems.atlassian.net/browse/NAS-119483

 

[cosine5000]

I've read all the info around hostpathvalidation, multiple times, and am still pretty baffled.

1. How does this setting remove the risk? What does it actually DO?
2. If I have a dataset that is also an SMB share how does switching the app's mounting from host path to NFS reduce the risk as compared to just old-fashioned host path mounting?
3. Is this not a problem for ANY dataset that is touched my more than one app, regardless of Host path or SMB or NFS? I've never heard of any serious file serving setup where data is not manipulated by multiple apps, how is this just a problem for TrueNAS?

 

[morganL]

I've read all the info around hostpathvalidation, multiple times, and am still pretty baffled.

1. How does this setting remove the risk? What does it actually DO?

It avoids an App accessing an SMB share without thought and deleting valuable data.

2. If I have a dataset that is also an SMB share how does switching the app's mounting from host path to NFS reduce the risk as compared to just old-fashioned host path mounting?

Not a lot. Except that the App is loaded on TrueNAS and we have some (limited) responsibility for it being well behaved.

3. Is this not a problem for ANY dataset that is touched my more than one app, regardless of Host path or SMB or NFS? I've never heard of any serious file serving setup where data is not manipulated by multiple apps, how is this just a problem for TrueNAS?

Its a problem for any App that is not well behaved. For example, an App might automatically take a high res video file, downgrade it to a small low-res file, then delete the original to save space. That could cause the loss of valuable quality info.

The question is who is responsible for preventing this?

The Community has been clear.. the safety belt is not appreciated and the warning bell that sounds when not buckled up is annoying.

In Cobia the plan has been changed to just a general warning..... Users will need to take responsibility for the Apps they choose to run and configure.

 

[Tywele]

As a beginner to TrueNAS and selfhosting in general what I'm wondering now is: If setting the Host Path of an app to the same path as an SMB share is not the recommended way to get files from a Windows machine to the app what is the correct/recommended way to achieve this?

 

[sretalla]

what is the correct/recommended way to achieve this?

WinSCP (SFTP or file transfer over SSH). FileZilla also has that option.

 

[Tazlord]

I consider myself to be pretty well versed in many things tech and in TrueNAS/FreeNAS. However, I am having a hard time grasping this concept. I'm trying to understand what it's "protecting".

For instance, let's say I install the Plex app. It will end up installing itself into the 'ix-applications' dataset in the chosen pool.
Let's say I have another dataset on the root of the pool named 'media'. That dataset contains child datasets that will contain all of the media (movies, music, pictures, etc). Content is added to those datasets via SMB. The 'media' dataset and all of its children all have their own SMB shares (not all media devices need access to the entire 'media' folder).

1. Will Plex have access to the 'media' dataset from where it lives?
2. The 'media' dataset will have permissions in place to only allow read (and traverse) access to Plex or any other streaming app.
3. Does the 'host path validation' configuration setting have anything to do with a setup like this?