23.10.2 - ix-applications problems - Upgrade failed - resource mapping not found

[FrankWard]

This issue started after upgrading to 23.10.2. I haven't changed anything, but I noticed there were ix-applications read-only errors occurring during replication. Here's the cryptic error. It also throws errors trying to upgrade these apps.

Code:

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "cloudnative-pg" namespace: "" from "": no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "prometheus-operator-kps-operator" namespace: "ix-prometheus-operator" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to update App: Error: UPGRADE FAILED: cannot patch "cloudnative-pg" with kind Deployment: Deployment.apps "cloudnative-pg" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"cloudnative-pg", "app.kubernetes.io/name":"cloudnative-pg"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Any idea what's the cause/remedy for this?

 

[morganL]

This issue started after upgrading to 23.10.2. I haven't changed anything, but I noticed there were ix-applications read-only errors occurring during replication. Here's the cryptic error. It also throws errors trying to upgrade these apps.

Code:

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "cloudnative-pg" namespace: "" from "": no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "prometheus-operator-kps-operator" namespace: "ix-prometheus-operator" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to update App: Error: UPGRADE FAILED: cannot patch "cloudnative-pg" with kind Deployment: Deployment.apps "cloudnative-pg" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"cloudnative-pg", "app.kubernetes.io/name":"cloudnative-pg"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Any idea what's the cause/remedy for this?

Can you tell us the whole situation
... did all Apps restart successfully?
What Apps are you running... standard or from other catalogs?

 

[dlc]

Same issue, cannot update CloudNative PG. For me, I am running truecharts catalog apps.

Also, cannot update prometheus-operator, same message.

 

[morganL]

There's a similar problem and potential resolution here:

SOLVED - Cobia, Error: "prometheus-operator" has no deployed releases, it can't upgrade & delete but running

TrueNAS-SCALE-23.10.1 One of my installed app somehow shows the "prometheus-operator" has no deployed releases Error, It cannot be upgrade and delete but it is running without issues. helm list --namespace ix-prometheus-operator...

www.truenas.com

 

[FrankWard]

Can you tell us the whole situation
... did all Apps restart successfully?
What Apps are you running... standard or from other catalogs?

Thanks Morgan. All apps are started and running as they were before the upgrade. I have three truecharts apps running. Prometheus/Cloudnative/WatchMyLan. So nothing crazy. I can certainly try to delete and reinstall the app, but I was hoping for a more elegant solution. I see that this is more of a TrueCharts issue, so I'll deal with that with them.

As for the replication tasks failing for ix-applications. Not sure if the two are connected, but I tried to delete my 'ix-applications' dataset on the remote and resync it to but was met with the same result below. No idea how or why that would suddenly have an issue replicating this dataset.

Code:

cannot mount '/mnt/Archive/POOL/ix-applications/k3s/kubelet': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-9ccbb51e-7272-4fbd-9188-105c495e6cc2': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-d1c6a0fb-8e1e-4f02-b8e4-8b2d98c06a80': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-ddd6754c-e5fb-41c4-9a3b-6c811779b2de': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/watchyourlan/volumes/pvc-84fd95ad-20ff-4abf-9916-666718d23264': failed to create mountpoint: Read-only file system

 

[morganL]

Thanks Morgan. All apps are started and running as they were before the upgrade. I have three truecharts apps running. Prometheus/Cloudnative/WatchMyLan. So nothing crazy. I can certainly try to delete and reinstall the app, but I was hoping for a more elegant solution. I see that this is more of a TrueCharts issue, so I'll deal with that with them.

As for the replication tasks failing for ix-applications. Not sure if the two are connected, but I tried to delete my 'ix-applications' dataset on the remote and resync it to but was met with the same result below. No idea how or why that would suddenly have an issue replicating this dataset.

Code:

cannot mount '/mnt/Archive/POOL/ix-applications/k3s/kubelet': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-9ccbb51e-7272-4fbd-9188-105c495e6cc2': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-d1c6a0fb-8e1e-4f02-b8e4-8b2d98c06a80': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-ddd6754c-e5fb-41c4-9a3b-6c811779b2de': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/watchyourlan/volumes/pvc-84fd95ad-20ff-4abf-9916-666718d23264': failed to create mountpoint: Read-only file system

I'd suggest trying to resolve the App errors with TrueCharts first... they might be causing a filesystem issue which is then impacting replication.

If the replication issue persists, lets submit a bug report.

 

[FrankWard]

I'd suggest trying to resolve the App errors with TrueCharts first... they might be causing a filesystem issue which is then impacting replication.

If the replication issue persists, lets submit a bug report.

• FYI - The apps 'watchmylan' and 'netdata' upgraded with no issues unlike the other two apps.
• Removed both offending truecharts apps (prometheus, cloudnative-pg) using the k3s command in shell
• Delete remote ix-applications dataset
• Re-ran the replication task which failed with just one readonly error with netdata, but not watchmylan.

I could delete netdata and re-install, but I'd really like to keep my historical performance data if possible.

 

[morganL]

• FYI - The apps 'watchmylan' and 'netdata' upgraded with no issues unlike the other two apps.
• Removed both offending truecharts apps (prometheus, cloudnative-pg) using the k3s command in shell
• Delete remote ix-applications dataset
• Re-ran the replication task which failed with just one readonly error with netdata, but not watchmylan.

I could delete netdata and re-install, but I'd really like to keep my historical performance data if possible.

Can netdata store the data in an external dataset??

 

[FrankWard]

Can netdata store the data in an external dataset??

It's possible but I haven't looked into it yet. Would prefer to understand the underlying problem before moving forward. Something happened after, but no necessarily due to the last update that caused replication of the ix-applications folder to fail with read-only errors. Very strange. My remote NAS is running Core 13.0-U6.1.