23.10.2 - ix-applications problems - Upgrade failed - resource mapping not found

FrankWard

Explorer
Joined
Feb 13, 2023
Messages
71
This issue started after upgrading to 23.10.2. I haven't changed anything, but I noticed there were ix-applications read-only errors occurring during replication. Here's the cryptic error. It also throws errors trying to upgrade these apps.

Code:
[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "cloudnative-pg" namespace: "" from "": no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "prometheus-operator-kps-operator" namespace: "ix-prometheus-operator" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to update App: Error: UPGRADE FAILED: cannot patch "cloudnative-pg" with kind Deployment: Deployment.apps "cloudnative-pg" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"cloudnative-pg", "app.kubernetes.io/name":"cloudnative-pg"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable


Any idea what's the cause/remedy for this?
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
This issue started after upgrading to 23.10.2. I haven't changed anything, but I noticed there were ix-applications read-only errors occurring during replication. Here's the cryptic error. It also throws errors trying to upgrade these apps.

Code:
[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "cloudnative-pg" namespace: "" from "": no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to upgrade App: Error: UPGRADE FAILED: resource mapping not found for name: "prometheus-operator-kps-operator" namespace: "ix-prometheus-operator" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first

[EFAULT] Failed to update App: Error: UPGRADE FAILED: cannot patch "cloudnative-pg" with kind Deployment: Deployment.apps "cloudnative-pg" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"cloudnative-pg", "app.kubernetes.io/name":"cloudnative-pg"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable


Any idea what's the cause/remedy for this?

Can you tell us the whole situation
... did all Apps restart successfully?
What Apps are you running... standard or from other catalogs?
 

dlc

Cadet
Joined
Mar 4, 2024
Messages
1
Same issue, cannot update CloudNative PG. For me, I am running truecharts catalog apps.

Also, cannot update prometheus-operator, same message.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694

FrankWard

Explorer
Joined
Feb 13, 2023
Messages
71
Can you tell us the whole situation
... did all Apps restart successfully?
What Apps are you running... standard or from other catalogs?
Thanks Morgan. All apps are started and running as they were before the upgrade. I have three truecharts apps running. Prometheus/Cloudnative/WatchMyLan. So nothing crazy. I can certainly try to delete and reinstall the app, but I was hoping for a more elegant solution. I see that this is more of a TrueCharts issue, so I'll deal with that with them.

As for the replication tasks failing for ix-applications. Not sure if the two are connected, but I tried to delete my 'ix-applications' dataset on the remote and resync it to but was met with the same result below. No idea how or why that would suddenly have an issue replicating this dataset.

Code:
cannot mount '/mnt/Archive/POOL/ix-applications/k3s/kubelet': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-9ccbb51e-7272-4fbd-9188-105c495e6cc2': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-d1c6a0fb-8e1e-4f02-b8e4-8b2d98c06a80': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-ddd6754c-e5fb-41c4-9a3b-6c811779b2de': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/watchyourlan/volumes/pvc-84fd95ad-20ff-4abf-9916-666718d23264': failed to create mountpoint: Read-only file system
 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Thanks Morgan. All apps are started and running as they were before the upgrade. I have three truecharts apps running. Prometheus/Cloudnative/WatchMyLan. So nothing crazy. I can certainly try to delete and reinstall the app, but I was hoping for a more elegant solution. I see that this is more of a TrueCharts issue, so I'll deal with that with them.

As for the replication tasks failing for ix-applications. Not sure if the two are connected, but I tried to delete my 'ix-applications' dataset on the remote and resync it to but was met with the same result below. No idea how or why that would suddenly have an issue replicating this dataset.

Code:
cannot mount '/mnt/Archive/POOL/ix-applications/k3s/kubelet': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-9ccbb51e-7272-4fbd-9188-105c495e6cc2': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-d1c6a0fb-8e1e-4f02-b8e4-8b2d98c06a80': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/netdata/volumes/pvc-ddd6754c-e5fb-41c4-9a3b-6c811779b2de': failed to create mountpoint: Read-only file system
cannot mount '/mnt/Archive/POOL/ix-applications/releases/watchyourlan/volumes/pvc-84fd95ad-20ff-4abf-9916-666718d23264': failed to create mountpoint: Read-only file system
I'd suggest trying to resolve the App errors with TrueCharts first... they might be causing a filesystem issue which is then impacting replication.

If the replication issue persists, lets submit a bug report.
 

FrankWard

Explorer
Joined
Feb 13, 2023
Messages
71
I'd suggest trying to resolve the App errors with TrueCharts first... they might be causing a filesystem issue which is then impacting replication.

If the replication issue persists, lets submit a bug report.
  • FYI - The apps 'watchmylan' and 'netdata' upgraded with no issues unlike the other two apps.
  • Removed both offending truecharts apps (prometheus, cloudnative-pg) using the k3s command in shell
  • Delete remote ix-applications dataset
  • Re-ran the replication task which failed with just one readonly error with netdata, but not watchmylan.
I could delete netdata and re-install, but I'd really like to keep my historical performance data if possible.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
  • FYI - The apps 'watchmylan' and 'netdata' upgraded with no issues unlike the other two apps.
  • Removed both offending truecharts apps (prometheus, cloudnative-pg) using the k3s command in shell
  • Delete remote ix-applications dataset
  • Re-ran the replication task which failed with just one readonly error with netdata, but not watchmylan.
I could delete netdata and re-install, but I'd really like to keep my historical performance data if possible.

Can netdata store the data in an external dataset??
 

FrankWard

Explorer
Joined
Feb 13, 2023
Messages
71
Can netdata store the data in an external dataset??

It's possible but I haven't looked into it yet. Would prefer to understand the underlying problem before moving forward. Something happened after, but no necessarily due to the last update that caused replication of the ix-applications folder to fail with read-only errors. Very strange. My remote NAS is running Core 13.0-U6.1.
 
Last edited:
Top