SOLVED TrueNAS Scale Cluster DNS not working

[li_chang]

Hi, after upgrading from SCALE-22.02-RC.1-1 to SCALE 22.02-RC.2, the apps installed via Truecharts cannot resolve hostnames anymore. I use docker exec and try nslookup and it fails. The DNS IP is 172.16.0.10 as default. In Network->Global Configuration->DNS I use cloudflare DNS (1.1.1.1), if this matters. By the way, apps that installed manually (not via Truecharts) work as expected (their DNS is same as global configuration mentioned above). Has anyone faced this issue as well? Thanks.

related issues:

https://jira.ixsystems.com/browse/NAS-112421

K8s DNS Issue

I have recently updated to 20.08. I had originally used all docker containers, and everything works fine. Recently I moved to TrueCharts for most my apps. I noticed any container I set up was having an issue connecting externally. After working with them on the issue we found out what was going...

www.truenas.com

 

[morganL]

I would suggest asking TrueCharts discord.... if it's only TrueCharts apps exhibiting the issue.

 

[truecharts]

Most likely @li_chang is trying the short DNS name syntax, which we have removed support for, as it was causing issues for some users.
It's adviced to follow the linking guide on our Website. We're pretty certain that works, as all of our staff use that.

 

[FrostyCat]

@li_chang can you shows us an example of what you're trying to resolve?

If it's a Docker container name it won't work since Kubernetes uses a different DNS convention and 172.16.0.10 is the CoreDNS resolver IP address. It will work just fine with stuff like <service-name>.<namespace>.svc[.cluster.local] but not with Docker container names.

 

[li_chang]

Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.

 

[truecharts]

Hi @truecharts @FrostyCat The app was trying to resolve common hostnames like google.com, and it was unable to do so. There is only 172.16.0.10 in /etc/resolv.conf. @truecharts Could you please provide the link to the guide? Because I've rollbacked to SCALE-22.02-RC.1-1, if there is any information needed, please let me know and I will boot to latest release! Thanks.

We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.

 

[li_chang]

We where assuming you meant internal resolve...

Anyway, if you need any help with our Apps. Please file a support ticket with us directly on discord. We don't offer active support here, byond simple answers.

Thank you @truecharts ! Yeah that's weird given that all other discussions are about internal resolve.. anyway, i will file ticket on discord later!

--

update: it seems that it is TrueNAS-related issue: I installed AdGuardHome APP as DNS (let's say IP is 10.0.0.2) and TrueNAS uses DHCP to acquire IP and DNS. However, despite there are DNS (1.1.1.1) manually set in Network->Global Configuration->DNS fields, in TrueNAS /etc/resolv.conf there is only DNS from DHCP (10.0.0.2)...

 

[iragoiko]

Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.

 

[morganL]

Just landed in truenas community. I am currently having the same issue as the one described above. Could you share how did you solve? My DNS is working perfectly outside cointainers but inside cluster it is not responding.

Best to start a new thread with your specific issues and settings... what troubleshooting have you done?

 

[phin]

I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.

 

[morganL]

I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.

Useful workaround... if you'd like to "report-a-bug" we can better document and resolve the issue.

 

[truecharts]

I too am also having issues with DNS from containers. However, i did find a workaround, but it takes a moment for it to be utilized so it doesnt always work when first deploying the container.

I have networking set to simple and it seems that from withen a container I cannot not resolve any DNS, at all. I tested this via a curl command to google.com and was met with a "Could not resolve host" error.

If i modify the settings for the said application, and set the networking to Expert and leave host network off (didn't try it on) I can then assign my application a network IP, gateway, route and dns servers. I modify the DNSPOLICY to none and put in my main DNS server on my network and it works.

So the issue seems to lay somewhere in the kubernetes layer. For whatever reason the default assigned DNS servers from kubernetes is not routing traffic upstream if its unable to resolve the host from its own DNS.

I am willing to provide more information if this doesnt make sense, or if more is required.

Thank you.

EDIT: This issue happens with BOTH Offical and Truecharts applications.

This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.

 

[phin]

This hack should NOT be used with our apps.
Users should never tick an "expert" checkbox, unless they have relatively high kubernetes experience.

With your solution, you completely bypass the kubernetes network stack, introducing all sorts of potential (security) issues and problems.
Users should also not go tweak the DNSPolicy, unless they know what they are doing. This for example completely invalidates our app linking guide.

The correct solution is to report these issues to the iX-Systems bugtracker like @morganL advices. If this was reported with debugs attached the previous times it was reported, this bug would most likely already been fixed by the excellent developers over at iX-Systems.
However: if users do not file bugreports and start using janky workarounds, issues will keep existing.

I am in process of putting together a bug report. However, I am using this work around at this time as I do require functioning apps. I am not a fan of utilizing a bandaid and as soon as I am able to get and test a downstream fix, i will absolutely be utilizing it going forward.

 

[phin]

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

I created the following report.

 

[Alphonse]

New person here, How do I access Phin's issue to add my reports on this same issue.

 

[NickF]

New person here, How do I access Phin's issue to add my reports on this same issue.

The Jira ticket is here:

[NAS-112421] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

But engineering closed it stating that they could not reproduce the problem. You would have to open a new ticket in Jira and I would recommend putting a link to that problem in your description.

 

[phin]

@Alphonse Did you ever resolve this?

 

[Alphonse]

@Alphonse Did you ever resolve this?

I thought I did, as I was assigning an IP address to my bridge, which was a 2nd ethernet jack, and that solved it for a little bit, but now the problem seems to have returned.

 

[mraw435]

@Alphonse Did you ever resolve this?

I previously used your workaround and got my apps to work but since the recent truecharts updates - i do not see the steps you took as an option and I am back to everything being broken. Do have any ideas? I posted a separate thread, here:

Help: Unable to get DNS queries

Hi all - I recently set up my first go with TrueNas Scale late last week and have an issue with DNS queries. My initial problem started when I downloaded Plex and then being unable to claim my server. Steps taken so far: 1.) Used plex claim toke 2.) I do have configure host network checked 3.)...

www.truenas.com

Sorry dont mean to highjack another users thread!

 

[mstinaff]

I think I'm running into this same issue and i think this Roger guy figured it out

Pod DNS Problems

I’ve got an extra instance of CoreDNS running in my cluster, serving *.k3s.differentpla.net, with LoadBalancer and Ingress names registered in it, and it’s working fine for queries to the cluster. It’s not working fine for queries inside the cluster. What’s up with that?

blog.differentpla.net

CoreDNS Customization

While debugging pod DNS problems, I discovered that CoreDNS allows customization by importing extra zone files from a config map. I’m going to use that to forward queries for k3s.differentpla.net to my custom CoreDNS instance.

blog.differentpla.net

TL;DR
when CoireDNS cant resolve DNS internally it just asks google at 8.8.8.8 and then gives up

Sure would love an approved method of feeding CoreDNS my local DNS server