SOLVED Truenas SCALE nextcloud image Error: ErrImagePull

[mnbk91]

I have several different apps installed on my scale and i installed many of them after starting the nextcloud install, all of them are pulling images from truecharts as all of them are truechart apps. But only when i try to install nextcloud i get the imagepull error.

I went through the forum and the discord server trying to figure out what the problem was, mostly i cam to know this is my network issue.
I tried using several different DNS servers Quad9, cloudflare, google after changing my DNS server every time i restarted my SCALE server.
But none of the DNS changes seem to help

Other than that i've tried deleting and re-adding my catalog, changing the image pool etc.. but nothing seems to work

Code:

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m57s  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         2m56s  default-scheduler  Successfully assigned ix-nextcloud/nextcloud-5db659b468-qwnsx to ix-truenas
  Normal   AddedInterface    2m50s  multus             Add eth0 [172.16.3.49/16] from ix-net
  Normal   Pulled            2m50s  kubelet            Container image "ghcr.io/truecharts/multi-init:v0.0.1@sha256:d947b94180365c8b7294b610dcd7138f7997134050a95bcd52bdef957247f33a" already present on machine
  Normal   Created           2m31s  kubelet            Created container prepare
  Normal   Started           2m30s  kubelet            Started container prepare
  Normal   Pulling           85s    kubelet            Pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:a307c7d49d5c7c691916f2e6bdbb10bc698bbd58498de12c4db992c4755a7018"
  Warning  Failed            3s     kubelet            Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:a307c7d49d5c7c691916f2e6bdbb10bc698bbd58498de12c4db992c4755a7018": rpc error: code = Unknown desc = context canceled
  Warning  Failed            3s     kubelet            Error: ErrImagePull
  Normal   BackOff           3s     kubelet            Back-off pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:a307c7d49d5c7c691916f2e6bdbb10bc698bbd58498de12c4db992c4755a7018"
  Warning  Failed            3s     kubelet            Error: ImagePullBackOff

P.S: My server is behind a CGNAT and i have a working cloudflare tunnel

i'm also wondering if my internet connection speed might be the reason for this error, if so is there a way to increase the timeout for image pull??

 

[morganL]

I doubt the timeout is because of slow internet. Its more likely loss/lack of connectivity or CDN/Server error.

 

[mnbk91]

still unable to find a solution, i have installed more apps after this, has anyone else faced similar issue??

every time i get Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:a307c7d49d5c7c691916f2e6bdbb10bc698bbd58498de12c4db992c4755a7018": rpc error: code = Unknown desc = context canceled

error code = unknown & desc = canceled , doesn't matter if i use PVC or host path for my user data storage

Has anyone else had this error, any help would be much appreciated.

 

[mnbk91]

I have a old HP DL380p with 20GB of ram, latest Truenas scale build & latest catalog of Truecharts

 

[aekt]

i did ran into this myself, the only solution is to create new app instance with a different name. Most likely whatever setup the vnet is messed up.

 

[mnbk91]

tried to do a fresh install under different name, it does not work.

i observed something, an app fails to pull it it has redis and postgresql as additional images in the container

for example nextcloud install is not working for me similarly outline which has the same two dependencies doesn't work and gets stuck in deployment 2/3 error image pull

 

[aekt]

An alternative would be to use system shell,
execute

docker image pull

On the image that failed to pull
So that the image is caches locally

Then try to launch the app again, it should pick up image.

Also another thing to try would be to perform a prune with docker on stale images,

docker image prune -f

there could be checksum failure at local cache which caused the cancellation of container launch

 

[mnbk91]

if i try to pull the image manually it get stuck in extracting the second part

 

[mnbk91]

It took quite a while and my gui shell kept disconnecting, got frustrated and ssh into my machine and pulled the image manually took around 15 mins to extract everything, but after that tried to install the nextcloud app from truecharts and it worked.

thank you very much man @aekt

 

[crk1918]

still unable to find a solution, i have installed more apps after this, has anyone else faced similar issue??

every time i get Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:a307c7d49d5c7c691916f2e6bdbb10bc698bbd58498de12c4db992c4755a7018": rpc error: code = Unknown desc = context canceled

error code = unknown & desc = canceled , doesn't matter if i use PVC or host path for my user data storage

Has anyone else had this error, any help would be much appreciated.

I have this error after the update from Angelfish to Bluefin 22.12 version, I can get it up and running with docker image pull [name]+version

 

[Migsi]

Chiming in, same issue on a <10Mbit/s connection since upgrade to Bluefin, only Nextcloud affected. It fails to pull some dependency containers, but when performed manually as described above, it works just fine. When I temporarily switch the whole home network over to a mobile connection with >10MB/s it just works fine and takes seconds to download and get up. So this must be related to network speed at least partially.
I'd consider letting user choose a global timeout value to mitigate this.

EDIT: A quick search on the bugtracker revealed this: https://ixsystems.atlassian.net/jir... "NAS" AND text ~ "GPU" ORDER BY created DESC but I can't figure where the related ticket about this timeout is.

 

[Flynn84]

Chiming in, same issue on a <10Mbit/s connection since upgrade to Bluefin, only Nextcloud affected. It fails to pull some dependency containers, but when performed manually as described above, it works just fine. When I temporarily switch the whole home network over to a mobile connection with >10MB/s it just works fine and takes seconds to download and get up. So this must be related to network speed at least partially.
I'd consider letting user choose a global timeout value to mitigate this.

EDIT: A quick search on the bugtracker revealed this: https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues/NAS-119121?jql=project = "NAS" AND text ~ "GPU" ORDER BY created DESC but I can't figure where the related ticket about this timeout is.

yep, happened to me as well on a pretty much identically slow connection.

Similar or identical problem is also documented in this thread, thrown error seems to be slightly different though. jira ticket is not public due to containing user info so no update on when this will get fixed unfortunately.

 

[morganL]

yep, happened to me as well on a pretty much identically slow connection.

Similar or identical problem is also documented in this thread, thrown error seems to be slightly different though. jira ticket is not public due to containing user info so no update on when this will get fixed unfortunately.

If you know its a slow connection issue and timeout is too short, please report another bug... if you are on 22.12.2

 

[Flynn84]

I just installed TrueNAS yesterday so I was on the latest stable version but I just switched over to UnRAID since they have ZFS as well now to see how that compares to TrueNAS since I found the k3s stack a little overkill for what I was trying to do and UnRAID supports GPU passthrough to a Jellyfin instance (I know that TrueNAS technically supports that too, but not with only a single GPU in the system). UnRAID is not exactly super intuitive either though. Would prefer TrueNAS but there seem to be quite a few really annoying bugs still present and some features missing I would like to have.

I will still report the bug I encountered though.

 

[blacksteel75]

I have this error after the update from Angelfish to Bluefin 22.12 version, I can get it up and running with docker image pull [name]+version

I had this same problem for several images failing on image pull, preventing some of (but not all) my apps from starting.

Doing a 'docker image pull' for those that failed from the command line resolved it, at least temporarily.

 

[victort]

I had this same problem for several images failing on image pull, preventing some of (but not all) my apps from starting.

Doing a 'docker image pull' for those that failed from the command line resolved it, at least temporarily.

Same experience for me. Someone should file a bug report about this…

 

[blacksteel75]

Same experience for me. Someone should file a bug report about this…

i did file a bug a few weeks ago on this topic. here's the ticket if you want to follow it: https://ixsystems.atlassian.net/browse/NAS-122115

 

[harshil12387]

i ended up installing cloudflared from truechart. this worked for me