Failed to update chart release: another operation (install/upgrade/rollback) is in progress

[oxyle]

Good afternoon everyone,

After a few weeks I finally decided to make the move to 22.12... All apps seem to be running and working, however I had to do a reboot at one point and I can't figure out why (when I change the specifications/settings) I get this error all the time:

[EFAULT] Failed to update chart release: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Whenever I open for example Overseer, try to change ports or any configuration at all I get that error.

Code:

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 553, in do_update
    await self.middleware.call('chart.release.helm_action', chart_release, chart_path, config, 'update')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1266, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1169, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 44, in helm_action
    raise CallError(f'Failed to {tn_action} chart release: {stderr.decode()}')
middlewared.service_exception.CallError: [EFAULT] Failed to update chart release: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

 

[bigjme]

I'm having the exact same issue. I even upgrades from Angelfish to Bluefin to see if that would help and its the exact same

Currently unable to upgrade or edit any docker containers

 

[oxyle]

I'm having the exact same issue. I even upgrades from Angelfish to Bluefin to see if that would help and its the exact same

Currently unable to upgrade or edit any docker containers

I had this fixed by rolling back one version. And then upgrading back to most recent version!

 

[bigjme]

Looks like i may be stuck as i only have one version, the container is question has never been upgraded :) never mind, i will keep looking

 

[Lockszmith]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

 

[adornala]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

This did the trick. Thank you.

 

[spvedula]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

This worked for me as well on TrueNAS-SCALE-22.12.1

 

[G2bb]

@Lockszmith I am fairly new to TrueNAS and Kubernetes, would you be willing to walk me through what your solution does and how I can find which app/app_name is causing my problem so that I can get my services back up and running?

 

[alieg]

@Lockszmith I am fairly new to TrueNAS and Kubernetes, would you be willing to walk me through what your solution does and how I can find which app/app_name is causing my problem so that I can get my services back up and running?

Did you figure this out?

 

[Starrbuck]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

You, sir, are a genius. Thanks.

 

[zari90]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

I'm getting an error zsh: parse error near '\n' when running APP_NAME=<APP_NAME> anything I might be doing incorrectly?

 

[zari90]

and when running the delete secret command

 

[zari90]

and when running the delete secret command
View attachment 67019

 

[mainstream]

try:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v1 (or the version you want to delete)

 

[Nicholas Flamy]

The solution mentioned in this SO answer worked for me:

Code:

alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like

Code:

NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m

However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:

Code:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>

THANK YOU SO MUCH

 

[Sebastian Bordon]

Worked flawlessly since I couldn't rollback properly as well. I appreciate it.