Almost all apps not deploying

[Zofoor]

Hi all!
I have errors with almost all apps, that are not deploying as expected. I was running TrueNAS Scale 22.02.2.1, tried to upgrade to 22.02.3 but didn't helped.

I hope to have some good hints on how to fix it.

All failing apps have similar errors.
For example, uptime-kuma:

Code:

2022-09-05 15:35:00
Error: ErrImagePull
2022-09-05 15:37:50
Failed to pull image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699": rpc error: code = Unknown desc = failed to register layer: rename /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/tmp/write-set-3762648111 /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/sha256/f362fb2f6e405a735c27dbeaf2c3d53818b8ab8cc738b35d646ecb6b59ca65a4: file exists
2022-09-05 15:34:33
Pulling image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699"
2022-09-05 15:35:01
Error: ImagePullBackOff
2022-09-05 15:35:01
Back-off pulling image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699"
2022-09-05 15:36:32
Failed to pull image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699": rpc error: code = Unknown desc = failed to register layer: rename /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/tmp/write-set-217400353 /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/sha256/f362fb2f6e405a735c27dbeaf2c3d53818b8ab8cc738b35d646ecb6b59ca65a4: file exists
2022-09-05 15:35:39
Failed to pull image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699": rpc error: code = Unknown desc = failed to register layer: rename /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/tmp/write-set-3194575161 /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/sha256/f362fb2f6e405a735c27dbeaf2c3d53818b8ab8cc738b35d646ecb6b59ca65a4: file exists
2022-09-05 15:35:00
Failed to pull image "tccr.io/truecharts/uptime-kuma:v1.17.1@sha256:27b2535717c614886535029f9a177dcd8e1e91a54a2402c5e02f2d8bd8184699": rpc error: code = Unknown desc = failed to register layer: rename /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/tmp/write-set-1049853744 /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/sha256/f362fb2f6e405a735c27dbeaf2c3d53818b8ab8cc738b35d646ecb6b59ca65a4: file exists
2022-09-05 15:34:33
Started container prepare
2022-09-05 15:34:33
Created container prepare
2022-09-05 15:34:33
Container image "tccr.io/truecharts/multi-init:v0.0.1@sha256:4c2caebee117b055f379377fd0fd306f2ee0e6697fd47d9364073e0e9e6f2e02" already present on machine
2022-09-05 15:34:33
Add eth0 [172.16.0.180/16] from ix-net

Successfully assigned ix-uptime-kuma/uptime-kuma-58c54b574b-zsjck to ix-truenas
2022-09-05 15:34:32
Created pod: uptime-kuma-58c54b574b-zsjck
2022-09-05 15:34:32
Scaled up replica set uptime-kuma-58c54b574b to 1

I tried to upgrade few dockers and this fixed some apps, but not all.

The system is:
AMD 5950x with 64Gb ECC ram (ECC ram recognized)
Apps are on an professional nvme drive (volume of one disk. Nothing to care about with the exception of vaultwarden).

There may have been a problem of lack of free disk space in this volume (the issue happened during my holidays).

Any hints?

 

[morganL]

It might also be a slow network issue.. can you verify your download speed.

 

[Zofoor]

It might also be a slow network issue.. can you verify your download speed.

Download at ~80 mbits/sec using speedtest.

Jellyfin log:

Code:

2022-09-06 12:22:35
(combined from similar events): Error: Error response from daemon: exit status 2: "/usr/sbin/zfs fs snapshot nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756@455048802" => cannot open 'nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756': dataset does not exist usage: snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
2022-09-06 12:20:49
Error: Error response from daemon: exit status 2: "/usr/sbin/zfs fs snapshot nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756@135297912" => cannot open 'nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756': dataset does not exist usage: snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
2022-09-06 12:20:48
Container image "tccr.io/truecharts/jellyfin:10.8.4@sha256:e13124f441178fc8068ea038bf8c470c05840cd503bbd2b611ad99a885437897" already present on machine
2022-09-06 12:20:48
Error: Error response from daemon: exit status 2: "/usr/sbin/zfs fs snapshot nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756@127604800" => cannot open 'nvme_volume/ix-applications/docker/1aa8809304538c183e44d9a995606d0253062d467027829f692562a55667f756': dataset does not exist usage: snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
2022-09-06 12:20:47
Started container prepare
2022-09-06 12:20:47
Created container prepare
2022-09-06 12:20:46
Container image "tccr.io/truecharts/multi-init:v0.0.1@sha256:4c2caebee117b055f379377fd0fd306f2ee0e6697fd47d9364073e0e9e6f2e02" already present on machine
2022-09-06 12:20:46
Add eth0 [172.16.0.194/16] from ix-net

Successfully assigned ix-jellyfin/jellyfin-b7554f557-284bq to ix-truenas
2022-09-06 12:20:45
Created pod: jellyfin-b7554f557-284bq

Pod Logs shows:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"jellyfin\" in pod \"jellyfin-b7554f557-284bq\" is waiting to start: CreateContainerError","reason":"BadRequest","code":400}

Transmission v3.00_12.0.30 log:

Code:

2022-09-06 12:14:28
Back-off restarting failed container
2022-09-06 12:14:27
Started container prepare
2022-09-06 12:14:27
Created container prepare
2022-09-06 12:14:26
Container image "tccr.io/truecharts/multi-init:v0.0.1@sha256:4c2caebee117b055f379377fd0fd306f2ee0e6697fd47d9364073e0e9e6f2e02" already present on machine
2022-09-06 12:14:26
Add eth0 [172.16.0.192/16] from ix-net

Successfully assigned ix-transmission/transmission-6d85c94f98-dc58v to ix-truenas
2022-09-06 12:14:26
Created pod: transmission-6d85c94f98-dc58v

Transmission v3.00_12.0.32 (so I tried to upgrade it) log:

Code:

2022-09-06 12:18:14
Back-off pulling image "tccr.io/truecharts/transmission:v3.00@sha256:cf61bde05f265653ce12b0ef42e25ac25abcc51ff84c9e5528ef75fea8330eda"
2022-09-06 12:18:13
Error: ErrImagePull
2022-09-06 12:18:13
Failed to pull image "tccr.io/truecharts/transmission:v3.00@sha256:cf61bde05f265653ce12b0ef42e25ac25abcc51ff84c9e5528ef75fea8330eda": rpc error: code = Unknown desc = failed to register layer: rename /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/tmp/write-set-3765705431 /mnt/nvme_volume/ix-applications/docker/image/zfs/layerdb/sha256/88d50848a6c430edd82f62cb4312b615def83c4675c9a330aa87d7fee300d7e2: file exists
2022-09-06 12:17:53
Pulling image "tccr.io/truecharts/transmission:v3.00@sha256:cf61bde05f265653ce12b0ef42e25ac25abcc51ff84c9e5528ef75fea8330eda"
2022-09-06 12:17:53
Started container prepare
2022-09-06 12:17:53
Created container prepare
2022-09-06 12:17:52
Container image "tccr.io/truecharts/multi-init:v0.0.1@sha256:4c2caebee117b055f379377fd0fd306f2ee0e6697fd47d9364073e0e9e6f2e02" already present on machine
2022-09-06 12:17:52
Add eth0 [172.16.0.193/16] from ix-net

Successfully assigned ix-transmission/transmission-54d5b468c9-z9m7z to ix-truenas
2022-09-06 12:17:51
Created pod: transmission-54d5b468c9-z9m7z
2022-09-06 12:17:51
Scaled up replica set transmission-54d5b468c9 to 1
2022-09-06 12:17:51
Deleted pod: transmission-6d85c94f98-dc58v

Using Pod Logs I see this error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"transmission\" in pod \"transmission-54d5b468c9-z9m7z\" is waiting to start: trying and failing to pull image","reason":"BadRequest","code":400}