Trying to create any Application hangs on "Deploying" and fails to pull

[Southpaw1496]

Hi

I've been setting up TrueNAS SCALE over the past couple of days, and it's mostly been going well, except for one thing: I can never get Applications to work. I first tried the TrueCharts Jellyfin application, and after that failed, I tried the official PiHole application (2022.10_1.0.7) as a test, but that failed too.

Here's the Application Log from the PiHole application:

Code:

2022-10-17 21:07:02
Pulling image "pihole/pihole:2022.10"
2022-10-17 21:07:02
Add eth0 [172.16.0.54/16] from ix-net
Successfully assigned ix-pihole/pihole-6f7cddf448-kmr5d to ix-truenas
2022-10-17 21:06:58
Created pod: pihole-6f7cddf448-kmr5d
2022-10-17 21:06:58
Scaled up replica set pihole-6f7cddf448 to 1
2022-10-17 21:06:58
MountVolume.SetUp failed for volume "kube-api-access-d858w" : object "ix-pihole"/"kube-root-ca.crt" not registered
2022-10-17 21:06:58
Job completed
2022-10-17 21:06:58
Started container pre-install-job
2022-10-17 21:06:57
Created container pre-install-job
2022-10-17 21:06:54
Successfully pulled image "alpine:latest" in 37.673630463s
2022-10-17 21:06:17
Pulling image "alpine:latest"
2022-10-17 21:06:17
Add eth0 [172.16.0.53/16] from ix-net
Successfully assigned ix-pihole/pihole-preinstall-job-wtkm8 to ix-truenas
2022-10-17 21:06:13
Created pod: pihole-preinstall-job-wtkm8
2022-10-17 21:08:23
Failed to pull image "pihole/pihole:2022.10": rpc error: code = Unknown desc = context canceled
2022-10-17 21:08:24
Back-off pulling image "pihole/pihole:2022.10"
2022-10-17 21:08:24
Error: ImagePullBackOff
2022-10-17 21:08:23
Error: ErrImagePull
2022-10-17 21:07:02
Pulling image "pihole/pihole:2022.10"
2022-10-17 21:07:02
Add eth0 [172.16.0.54/16] from ix-net
Successfully assigned ix-pihole/pihole-6f7cddf448-kmr5d to ix-truenas
2022-10-17 21:06:58
Created pod: pihole-6f7cddf448-kmr5d
2022-10-17 21:06:58
Scaled up replica set pihole-6f7cddf448 to 1
2022-10-17 21:06:58
MountVolume.SetUp failed for volume "kube-api-access-d858w" : object "ix-pihole"/"kube-root-ca.crt" not registered
2022-10-17 21:06:58
Job completed
2022-10-17 21:06:58
Started container pre-install-job
2022-10-17 21:06:57
Created container pre-install-job
2022-10-17 21:06:54
Successfully pulled image "alpine:latest" in 37.673630463s
2022-10-17 21:06:17
Pulling image "alpine:latest"
2022-10-17 21:06:17
Add eth0 [172.16.0.53/16] from ix-net
Successfully assigned ix-pihole/pihole-preinstall-job-wtkm8 to ix-truenas
2022-10-17 21:06:13
Created pod: pihole-preinstall-job-wtkm8

I set the Application up with these settings (most if not all are default):

Code:

Application Name: pihole
Version: 1.0.7
Storage User ID: 568
Storage Group ID: 568
Admin password: REDACTED
Configure timezone: 'Europe/London' timezone
Pihole Storage:
Configuration Volume:
Enable Custom Host Path for Pihole Configuration Volume: false
DNSMASQ Volume for pihole:
Enable Custom Host Path for Pihole DNSMASQ Volume: false
Web Port for pihole: 20720
DNS TCP Port for pihole: 20721
DNS UDP Port for pihole: 20721
Enable DHCP: false
Enable Host Network: false
Enable Pod resource limits: false

I'm running TrueNAS-SCALE-22.02.4, and it is definitely connected to the internet. I have tried manually overriding my DNS to 1.1.1.1, and I have also tried pulling the Docker image manually when I tried setting up Jellyfin, but that also failed, though with a different error:

Code:

2022-10-17 21:36:37
Applied LoadBalancer DaemonSet ix-jellyfin/svclb-jellyfin
2022-10-17 21:36:44
LoadBalancer Ingress IP addresses updated: 192.168.0.128
2022-10-17 21:36:43
Started container lb-tcp-8096
2022-10-17 21:36:43
Created container lb-tcp-8096
2022-10-17 21:36:41
Container image "rancher/klipper-lb:v0.3.4" already present on machine
2022-10-17 21:36:41
Add eth0 [172.16.0.55/16] from ix-net
Successfully assigned ix-jellyfin/svclb-jellyfin-rn8mm to ix-truenas
2022-10-17 21:36:37
Created pod: svclb-jellyfin-rn8mm
2022-10-17 21:36:37
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-10-17 21:36:37
Created pod: jellyfin-bf8c66846-m87vq
2022-10-17 21:36:37
Scaled up replica set jellyfin-bf8c66846 to 1

I'm at my wits end with this issue. I don't want to switch back to "vanilla" Linux, because the SMB shares work so well on TrueNAS, but I feel like I'm running out of options. I've left both containers in their hanging state for now.

 

[justjasch]

Have u set networksettings under Apps -> settings advanced ?

the important ones are "Route v4 Gateway" (your Router Address) and "Route v4 Interface" (your Truenas Lan interface)

 

[Southpaw1496]

Have u set networksettings under Apps -> settings advanced ?

the important ones are "Route v4 Gateway" (your Router Address) and "Route v4 Interface" (your Truenas Lan interface)

View attachment 59259

I have now, unfortunately the error remains the same (I deleted the application and restarted the server)

 

[justjasch]

can u show your network and kubernotes settings + your pihole install settings

 

[Southpaw1496]

Kubernetes settings:



PiHole settings (left default whenever possible as it was only for testing):



Network Settings:



If there's a better way then screenshots to get all this information, I'm all ears

 

[justjasch]

ok,

plz. lets try 2 things, first add your default Gateway under Global Conf.
under Kuberbnetes settings set the Node IP (the drop down should have you Truenas ip avail. .
then plz restart and try again with pi hole install

 

[Southpaw1496]

ok,

plz. lets try 2 things, first add your default Gateway under Global Conf.
under Kuberbnetes settings set the Node IP (the drop down should have you Truenas ip avail. .
then plz restart and try again with pi hole install

0.0.0.0 is the only available Node IP in the dropdown.

I hardcoded the gateway (which was correct anyway) but still no joy.

 

[justjasch]

this is strange there should be you ip address as option.
ok on other thing an App pool is set?

under apps settings a Pool ist set and under storage u can see Ixapplication Dataset on this Pool?

 

[Southpaw1496]

I can, yes

 

[justjasch]

ok thats strange lets reset docker stuff (there is some stuff on it 1GB+ used).

under Apps Settings unset Pool-> then under Storage delete IX-Apps Dataset -> go back Apps now he promts for a Pool -> choose one base images will be fetched.
then go to Manage Docker Images and update the Base images .

then try a gain install with default.

 

[Southpaw1496]

PiHole experienced the same error after I did this, however, I also tried installing IPFS, which installed successfully, though it took a little longer than I thought it should.

Could it be to do with the image size or something?

 

[justjasch]

maybe, pihole took like 10-15sec for me with 100mbit inet uplink.
Have u already running a pihole on Rasberry or something?(maybe some dns request get blocked)

 

[Southpaw1496]

It's definitely an issue with TrueNAS, but my end goal is installing Jellyfin anyway. The computer I'm running at the moment is the only spare one I have.

Since yesterday, I've completely reinstalled TrueNAS to no avail. I'm going to try reconfiguring the network, and if that doesn't work I'll try installing the beta.

 

[Southpaw1496]

This evening it suddenly started working, and I have no idea why, but I'm not complaining.

For anyone from the future, I'm sorry that this may not be the solution you needed, but I will leave you with a little advice. You'll get a lot more useful results if you Google the error in relation to Kubernetes instead of TrueNAS, for example, I found this article. Remember to prepend every kubectl command with k3s, so for example k3s kubectl get pods.

Good luck!

 

[Southpaw1496]

It stopped working again, and it's still not working perfectly, but here's what I did to work around it.

First, I ran k3s kubectl get pods --all-namespaces to see what namespace the Jellyfin pod was in. Then, as it was in the ix-jellyfin namespace, I ran k3s kubectl get events -n ix-jellyfin --watch to check the event log for that namespace. I noticed that the image pulls were failing with the error Failed to pull image "<url>": rpc error: code = Unknown desc = context canceled, which I found indicated a timeout, so I copied the URL and pulled it manually using `docker pull`. It took far longer than it should have, which is likely why it was failing, but Jellyfin then launched, as the image was present on the system.

Images taking longer than they should to pull is another issue, which I will solve later, but I will link any solutions I find here.

 

[Heavy]

I'd also report this error to ix-systems if you can. I had a similar issue this morning and created a bug report. Its the only way issues like this will get solved.
Truenas Scale GUI > System Settings > General > File ticket

 

[Southpaw1496]

If you have already filed a bug report, what use is me filing another?

 

[Heavy]

If you have also filed a bug report, what use is me filing another?

??
More data to look over to hone in on the issue
Your issue could be caused by something slightly(or totally) different

Or don't, and live with bugs, idc

 

[Southpaw1496]

If I file an issue that is the same as yours, it will be marked as a duplicate and I will have wasted both my time and the time of the bug tracker triage team.

I do intend to file a bug report, but only after ensuring that it will be completely original.

 

[Southpaw1496]

I have updated to the Bluefin beta train and from my very limited testing, the issue with pulling Docker images has been resolved.

Edit: This is not the case, and I have filed a bug report under NAS-118987