After latest Truenas 22.12.3 update and app updates ingress is no longer working

[mvdheijkant]

After the latest update from Truenas Scale 22.12.2 to 22.12.3, but also updating all installed apps, ingress (traefik) is no longer working.
The error message when updating the configuration is as follows:

Error: [EINVAL] values.ingress.main.tls: Item#0 is not valid per list types: [EINVAL] tlsEntry.hosts: Item#0 is not valid per list types: [host] Not a string

What I found was that Traefik settings App Configuration, Expert Mode, ingressClass and isDefaultClass where disabled so I enabled them again.
But also every app that was configured with Ingress had TLS-Settings, Certificate Hosts, Host <domainname> and Use TrueNAS SCALE Certificate (Deprecated) <my domain certificate> disabled

Certificate Hosts, Host setting seems to be causing the error, although the text is the same as before "heimdall.mydomain.nl".
I'm not sure if setting "Use TrueNAS SCALE Certificate (Deprecated)" has indirect cause. But I don't know what the alternative is.

 

[morganL]

Has anyone else had success or failure updating with Traefik?

I doubt ist hardware related, but please follow forum rules and describe. If you've done that and no-one else has positive or negative experience, the suggest you report a bug if you have the time to help us troubleshoot.

Can you rollback to 22.12.2 and get things working again?

 

[victort]

Truecharts has deprecated the option to use Scale Certificate.

You now need to use clusterissuer + traefik to get certs working.

They have tutorials on their site for it. It’s fairly simple to configure. Any mention to cert-manager is no longer valid, the new name of the app is clusterissuer

 

[danb35]

It’s fairly simple to configure.

Is it? Because those tutorials say to use the cert-manager app, and I can't find any app by that name.

 

[victort]

Is it? Because those tutorials say to use the cert-manager app, and I can't find any app by that name.

Lol.

They changed the name to clusterissuer

 

[mvdheijkant]

Still can't seem to make it work. If anyone has a suggestion, or added documentation.
I hope this description makes any sense.

I've installed clusterissuer as is shown on https://truecharts.org/charts/enterprise/clusterissuer/how-to/
What is not explained, is if setting "SelfSigned Issuer" should or should not be enabled
tried both.

Also I still use App Configuration
Expert Mode
IngressClass=Enable
isDefaultClass=Enable
tried both: servicetype "loadbalancer"/Cluster ip, endpoint 9000

Everything else is default.

Opening the traefik website at https://traefik.domain.nl give a 404 error
I don't have a clear logging for traefik, but for Heimdall it looks like this
both traefik and heimdall have a valid certificate but give a 404 error

2023-06-20 20:14:54 Order completed successfully
2023-06-20 20:14:54 Certificate fetched from issuer successfully
2023-06-20 20:14:54 The certificate has been successfully issued
2023-06-20 20:14:53 Domain "heimdall.domain.nl" verified with "DNS-01" validation
2023-06-20 20:14:31 Created container heimdall
2023-06-20 20:14:31 Started container heimdall
2023-06-20 20:14:29 Scaled up replica set heimdall-c7946bfb6 to 1
2023-06-20 20:14:29 Created pod: heimdall-c7946bfb6-csvr7
2023-06-20 20:14:29 Successfully assigned ix-heimdall/heimdall-c7946bfb6-csvr7 to ix-truenas
2023-06-20 20:14:29 Add eth0 [172.16.1.56/16] from ix-net
2023-06-20 20:14:29 Container image "tccr.io/truecharts/heimdall:2.5.6@sha256:093de204da5125c16f2c60227950844705f51186f748ba15e78f7ce57d1dfb1b" already present on machine
2023-06-20 20:13:41 Presented challenge using DNS-01 challenge mechanism

 

[victort]

So it looks like certmanager/clusterissuer is working properly.

Have you tried reinstalling Traefik?

Also, I dont know if this matters, but for apps using Postgres databases, it is apparently mandatory to install CNPG app from the operators train.

Also don’t touch any setting when installing Traefik except for the ports.

 

[mvdheijkant]

Thanks for your suggestion.
Reinstalling Traefik didn't work. didn't touch any other settings.
Installing cloudnative-PG failed with the error message below.
I don't know if installing cloudnative-PG will solve the traefik issue, but for this error I created a bug at the cloudnative-PG Github.

[EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "backups.postgresql.cnpg.io" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "cloudnative-pg"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ix-cloudnative-pg"

 

[victort]

Thanks for your suggestion.
Reinstalling Traefik didn't work. didn't touch any other settings.
Installing cloudnative-PG failed with the error message below.
I don't know if installing cloudnative-PG will solve the traefik issue, but for this error I created a bug at the cloudnative-PG Github.

[EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "backups.postgresql.cnpg.io" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "cloudnative-pg"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ix-cloudnative-pg"

Did you follow the instructions to delete the databases?

The instructions are under the CNPG article.

 

[mvdheijkant]

No, but I found them when you mentioned it.
This is becoming too complex for my taste.
I hope at some point "they" will automate this during installing cloudnative-PG.
Or hope some traefik update will fix the problem.

 

[emil-kirilov]

Hey, guys!

I'm not sure if this will help you, however, I just installed Heimdall in a custom jail on my TrueNAS Core 13.

Perhaps my script can help you?

 

[danb35]

I'm not sure if this will help you,

It seems highly unlikely, as (1) this thread has nothing to do with TrueNAS CORE, (2) this thread has nothing to do with Heimdall, and (3) there's already a much more comprehensive script to install Heimdall in a CORE jail in the resources section here.

 

[emil-kirilov]

It seems highly unlikely, as (1) this thread has nothing to do with TrueNAS CORE, (2) this thread has nothing to do with Heimdall, and (3) there's already a much more comprehensive script to install Heimdall in a CORE jail in the resources section here.

You are right, danb35. I overlooked 1) and 2). I am sorry. If it were possible, I would delete my post.
I am aware of 3), and it didn't help me. Maybe that's me being a noob again.
Nevertheless, I will be more careful when posting in the future.

 

[danb35]

I am aware of 3), and it didn't help me.

I'd be interested in hearing why not, as it's my goal for the scripts to be as easy to use as possible--if you're willing, I'd appreciate a post in the discussion thread for that resource describing how it didn't work for you.