Failed to update chart release: another operation (install/upgrade/rollback) is in progress

O

oxyle

Cadet
Joined
Jan 15, 2023
Messages
3
Good afternoon everyone,

After a few weeks I finally decided to make the move to 22.12... All apps seem to be running and working, however I had to do a reboot at one point and I can't figure out why (when I change the specifications/settings) I get this error all the time:

[EFAULT] Failed to update chart release: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Whenever I open for example Overseer, try to change ports or any configuration at all I get that error.

Code:
 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 553, in do_update
    await self.middleware.call('chart.release.helm_action', chart_release, chart_path, config, 'update')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1266, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1169, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 44, in helm_action
    raise CallError(f'Failed to {tn_action} chart release: {stderr.decode()}')
middlewared.service_exception.CallError: [EFAULT] Failed to update chart release: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
 
B

bigjme

Dabbler
Joined
May 16, 2021
Messages
19
I'm having the exact same issue. I even upgrades from Angelfish to Bluefin to see if that would help and its the exact same

Currently unable to upgrade or edit any docker containers
 
O

oxyle

Cadet
Joined
Jan 15, 2023
Messages
3
bigjme said:
I'm having the exact same issue. I even upgrades from Angelfish to Bluefin to see if that would help and its the exact same

Currently unable to upgrade or edit any docker containers
Click to expand...
I had this fixed by rolling back one version. And then upgrading back to most recent version!
 
B

bigjme

Dabbler
Joined
May 16, 2021
Messages
19
Looks like i may be stuck as i only have one version, the container is question has never been upgraded :) never mind, i will keep looking
 
L

Lockszmith

Cadet
Joined
Mar 29, 2022
Messages
3
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
 
adornala

adornala

Cadet
Joined
Dec 2, 2022
Messages
3
Lockszmith said:
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
Click to expand...
This did the trick. Thank you.
 
S

spvedula

Cadet
Joined
Apr 6, 2022
Messages
9
Lockszmith said:
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
Click to expand...
This worked for me as well on TrueNAS-SCALE-22.12.1
 
G2bb

G2bb

Cadet
Joined
Feb 22, 2023
Messages
2
@Lockszmith I am fairly new to TrueNAS and Kubernetes, would you be willing to walk me through what your solution does and how I can find which app/app_name is causing my problem so that I can get my services back up and running?
 
alieg

alieg

Dabbler
Joined
Jul 12, 2013
Messages
44
G2bb said:
@Lockszmith I am fairly new to TrueNAS and Kubernetes, would you be willing to walk me through what your solution does and how I can find which app/app_name is causing my problem so that I can get my services back up and running?
Click to expand...
Did you figure this out?
 
S

Starrbuck

Dabbler
Joined
Jul 7, 2016
Messages
15
Lockszmith said:
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
Click to expand...
You, sir, are a genius. Thanks.
 
Z

zari90

Cadet
Joined
Mar 5, 2023
Messages
4
Lockszmith said:
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
Click to expand...
I'm getting an error zsh: parse error near '\n' when running APP_NAME=<APP_NAME> anything I might be doing incorrectly?
 
Z

zari90

Cadet
Joined
Mar 5, 2023
Messages
4
and when running the delete secret command
1685541292919.png
 
M

mainstream

Dabbler
Joined
Jan 7, 2022
Messages
18
try:

kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v1 (or the version you want to delete)
 
N

Nicholas Flamy

Dabbler
Joined
Feb 25, 2023
Messages
28
Lockszmith said:
The solution mentioned in this SO answer worked for me:
Code:
alias kubectl='env sudo k3s kubectl '
alias helm='env sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm '

APP_NAME=<APP_NAME>

kubectl get secrets --namespace=ix-$APP_NAME "--sort-by=.metadata.creationTimestamp" | sed "s/$APP_NAME/<APP_NAME>/g"

# Results SHOULD look something like
Code:
NAME                                TYPE                                  DATA   AGE
default-token-gr4tv                 kubernetes.io/service-account-token   3      200d
sh.helm.release.v1.<APP_NAME>.v36   helm.sh/release.v1                    1      62d
sh.helm.release.v1.<APP_NAME>.v39   helm.sh/release.v1                    1      47d
sh.helm.release.v1.<APP_NAME>.v45   helm.sh/release.v1                    1      36d
sh.helm.release.v1.<APP_NAME>.v49   helm.sh/release.v1                    1      30d
sh.helm.release.v1.<APP_NAME>.v58   helm.sh/release.v1                    1      14m
sh.helm.release.v1.<APP_NAME>.v59   helm.sh/release.v1                    1      13m
<APP_NAME>-tls-0-ixcert-4-59        kubernetes.io/tls                     2      13m


However, if the ixcert line isn't the last one, it probably means something got stuck.
Try removing the latest secret, and re-deploy:
Code:
kubectl delete secret --namespace=ix-$APP_NAME sh.helm.release.v1.$APP_NAME.v<Latest Version>
Click to expand...
THANK YOU SO MUCH
 
You must log in or register to reply here.

Similar threads

B
Failed to upgrade chart: Error: pre-upgrade hook already exists
Replies
4
Views
6K
Neillman
Neillman
P
Error: INSTALLATION FAILED: create: failed to create: secrets
Replies
5
Views
2K
xzibit
xzibit
Kasazn
All Apps unable to update (TrueNAS-SCALE-22.02.4)
Replies
10
Views
4K
om1d3
om1d3
H
Apps won't update
Replies
5
Views
3K
derlinus
D
H
my app is normal before upgrade to TrueNAS-SCALE-22.12.1,but they all can't start now.
Replies
3
Views
2K
LarsR
L
Top