SOLVED TrueNAS Scale Cluster DNS not working

li_chang

Dabbler
Joined
May 31, 2017
Messages
35
Hi, after upgrading from SCALE-22.02-RC.1-1 to SCALE 22.02-RC.2, the apps installed via Truecharts cannot resolve hostnames anymore. I use docker exec and try nslookup and it fails. The DNS IP is 172.16.0.10 as default. In Network->Global Configuration->DNS I use cloudflare DNS (1.1.1.1), if this matters. By the way, apps that installed manually (not via Truecharts) work as expected (their DNS is same as global configuration mentioned above). Has anyone faced this issue as well? Thanks.


1641745131451.png



related issues:


 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I would suggest asking TrueCharts discord.... if it's only TrueCharts apps exhibiting the issue.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
Most likely @li_chang is trying the short DNS name syntax, which we have removed support for, as it was causing issues for some users.
It's adviced to follow the linking guide on our Website. We're pretty certain that works, as all of our staff use that.
 

FrostyCat

Explorer
Joined
Jan 4, 2022
Messages
79
@li_chang can you shows us an example of what you're trying to resolve?

If it's a Docker container name it won't work since Kubernetes uses a different DNS convention and 172.16.0.10 is the CoreDNS resolver IP address. It will work just fine with stuff like <service-name>.<namespace>.svc[.cluster.local] but not with Docker container names.
 

li_chang

Dabbler
Joined
May 31, 2017
Messages
35
Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.
We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.
 

li_chang

Dabbler
Joined
May 31, 2017
Messages
35
We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.
Thank you @truecharts ! Yeah that's weird given that all other discussions are about internal resolve.. anyway, i will file ticket on discord later!

--

update: it seems that it is TrueNAS-related issue: I installed AdGuardHome APP as DNS (let's say IP is 10.0.0.2) and TrueNAS uses DHCP to acquire IP and DNS. However, despite there are DNS (1.1.1.1) manually set in Network->Global Configuration->DNS fields, in TrueNAS /etc/resolv.conf there is only DNS from DHCP (10.0.0.2)...
 
Last edited:

iragoiko

Cadet
Joined
Jul 8, 2022
Messages
4
Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.
Best to start a new thread with your specific issues and settings... what troubleshooting have you done?
 

phin

Dabbler
Joined
Aug 28, 2018
Messages
11
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.
Useful workaround... if you'd like to "report-a-bug" we can better document and resolve the issue.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.

This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.
 

phin

Dabbler
Joined
Aug 28, 2018
Messages
11
This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.
I am in process of putting together a bug report. However, I am using this work around at this time as I do require functioning apps. I am not a fan of utilizing a bandaid and as soon as I am able to get and test a downstream fix, i will absolutely be utilizing it going forward.
 

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
New person here, How do I access Phin's issue to add my reports on this same issue.
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
New person here, How do I access Phin's issue to add my reports on this same issue.
The Jira ticket is here:

But engineering closed it stating that they could not reproduce the problem. You would have to open a new ticket in Jira and I would recommend putting a link to that problem in your description.
 

mraw435

Cadet
Joined
Sep 18, 2022
Messages
8
@Alphonse Did you ever resolve this?

I previously used your workaround and got my apps to work but since the recent truecharts updates - i do not see the steps you took as an option and I am back to everything being broken. Do have any ideas? I posted a separate thread, here:

Sorry dont mean to highjack another users thread!
 

mstinaff

Dabbler
Joined
Jan 21, 2014
Messages
34
I think I'm running into this same issue and i think this Roger guy figured it out

TL;DR
when CoireDNS cant resolve DNS internally it just asks google at 8.8.8.8 and then gives up

Sure would love an approved method of feeding CoreDNS my local DNS server
 
Top