Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SOLVED TrueNAS Scale Cluster DNS not working

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

li_chang

Dabbler
Joined
May 31, 2017
Messages
21
Hi, after upgrading from SCALE-22.02-RC.1-1 to SCALE 22.02-RC.2, the apps installed via Truecharts cannot resolve hostnames anymore. I use docker exec and try nslookup and it fails. The DNS IP is 172.16.0.10 as default. In Network->Global Configuration->DNS I use cloudflare DNS (1.1.1.1), if this matters. By the way, apps that installed manually (not via Truecharts) work as expected (their DNS is same as global configuration mentioned above). Has anyone faced this issue as well? Thanks.


1641745131451.png



related issues:


 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
1,599
I would suggest asking TrueCharts discord.... if it's only TrueCharts apps exhibiting the issue.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
578
Most likely @li_chang is trying the short DNS name syntax, which we have removed support for, as it was causing issues for some users.
It's adviced to follow the linking guide on our Website. We're pretty certain that works, as all of our staff use that.
 

FrostyCat

Dabbler
Joined
Jan 4, 2022
Messages
32
@li_chang can you shows us an example of what you're trying to resolve?

If it's a Docker container name it won't work since Kubernetes uses a different DNS convention and 172.16.0.10 is the CoreDNS resolver IP address. It will work just fine with stuff like <service-name>.<namespace>.svc[.cluster.local] but not with Docker container names.
 

li_chang

Dabbler
Joined
May 31, 2017
Messages
21
Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
578
Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.
We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.
 

li_chang

Dabbler
Joined
May 31, 2017
Messages
21
We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.
Thank you @truecharts ! Yeah that's weird given that all other discussions are about internal resolve.. anyway, i will file ticket on discord later!

--

update: it seems that it is TrueNAS-related issue: I installed AdGuardHome APP as DNS (let's say IP is 10.0.0.2) and TrueNAS uses DHCP to acquire IP and DNS. However, despite there are DNS (1.1.1.1) manually set in Network->Global Configuration->DNS fields, in TrueNAS /etc/resolv.conf there is only DNS from DHCP (10.0.0.2)...
 
Last edited:

iragoiko

Cadet
Joined
Jul 8, 2022
Messages
4
Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
1,599
Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.
Best to start a new thread with your specific issues and settings... what troubleshooting have you done?
 

phin

Cadet
Joined
Aug 28, 2018
Messages
9
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
1,599
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.
Useful workaround... if you'd like to "report-a-bug" we can better document and resolve the issue.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
578
I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.

This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.
 

phin

Cadet
Joined
Aug 28, 2018
Messages
9
This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.
I am in process of putting together a bug report. However, I am using this work around at this time as I do require functioning apps. I am not a fan of utilizing a bandaid and as soon as I am able to get and test a downstream fix, i will absolutely be utilizing it going forward.
 
Top