K8s DNS Issue

[NightShaman]

I have recently updated to 20.08.

I had originally used all docker containers, and everything works fine. Recently I moved to TrueCharts for most my apps. I noticed any container I set up was having an issue connecting externally. After working with them on the issue we found out what was going on. When the app uses DNS it fails. So when Sabnzbd was set up and using us.newsgroup.com it would fail. But if I used xxx.xxx.xxx.xxx it would connect with no issue.

So it seems my K8s ClusterFirst DNS is broken. Anyone have any suggestions on how to fix it?

Thanks in advance!

 

[NightShaman]

Oooops 21.08

 

[brando56894]

I don't have the solution for you, but you probably need to change something in the Kubernetes settings, which you can access by going to Apps -> Settings (upper righthand corner).

 

[NightShaman]

I don't have the solution for you, but you probably need to change something in the Kubernetes settings, which you can access by going to Apps -> Settings (upper righthand corner).

Thanks for the reply. I had changed the interfaces and everything.
The only thing I hadn't changed was the actual cluster addresses. It looks like that would re-initialize the cluster. So maybe I will try that tonight. I have become a pro at re-installing everything.

 

[NightShaman]

That didn't work. ¯\_(ツ)_/¯

 

[majinjing3]

Disable ipv6 may be work

 

[NightShaman]

Disable ipv6 may be work

Thanks, I don't use IPV6

 

[slm4996]

Your DNS ndots config is off or your DNS server is forwarding domains externally incorrectly.

Either add a DNS config for the app as ndots with a value of 1, or just append an extra "." at the end of any fully qualified domains your pod needs to reach externally. "us.newsgroup.com" would become "us.newsgroup.com."

More info:

Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances

pracucci.com

 

[stavros-k]

Your DNS ndots config is off or your DNS server is forwarding domains externally incorrectly.

Either add a DNS config for the app as ndots with a value of 1, or just append an extra "." at the end of any fully qualified domains your pod needs to reach externally. "us.newsgroup.com" would become "us.newsgroup.com."

More info:

Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances

pracucci.com

While appending a dot, might work, It's not viable to edit every apps source code to append a dot to every url/link/whatever they use.

Also editing config files in scale is a no go, as it's going to be reseted on upgrade or even reboot.

Best case is opening a jira ticket and let iX tackle this
So noone will ever deal with this again!

 

[truecharts]

Slight clearification:
We've helped this user and pointed him here (or towards iX's @waqarahmed ), as this is not the default or expected behavior.
Generally speaking our users are able to reach outside domains using ClusterFirst DNS without any issues.

@slm4996 Thank you for your reference we'll take that into account for future improvements! :)
Feel free to throw is more bones at our Discord or Github, both of which are available on https://truecharts.org

 

[slm4996]

No need to edit core configuration, just use the Advanced DNS portion of the App editor. I fixed this on my DNS server, but if that isn't an option or is being figured out then this is a good workaround.

 

[NightShaman]

Sorry I was out of town for a few days. It does look like the ndots adding the . at the end of the address and it worked as it should.

Thanks!

 

[NightShaman]

No need to edit core configuration, just use the Advanced DNS portion of the App editor. I fixed this on my DNS server, but if that isn't an option or is being figured out then this is a good workaround.

How did you fix this on your DNS Server? I am currently using Pi-Hole for DNS

 

[slm4996]

How did you fix this on your DNS Server? I am currently using Pi-Hole for DNS

I believe on Pi-Hole / AdGuard Home / DNSMasq, you may be able to fix it by adding a regex block filter for nested sub-domains of your domain. For example, if your domain is "example.com" try "/^.*[\w-]+\.[\w-]+\.example.com/"

 

[truecharts]

How did you fix this on your DNS Server? I am currently using Pi-Hole for DNS

This is not a failure in your DNS server, it's a failure in SCALE's integrated kubernetes DNS server, please file a bugreport at iX JIRA, as requested by us earlier, to make sure this doesn't happen again (to you and other users) and be sure to attach a debuglog from your TrueNAS SCALE Machine. This should not have happened with TrueCharts or with any App, using default settings.

 

[slm4996]

This is not a failure in your DNS server, it's a failure in SCALE's integrated kubernetes DNS server, please file a bugreport at iX JIRA, as requested by us earlier, to make sure this doesn't happen again (to you and other users) and be sure to attach a debuglog from your TrueNAS SCALE Machine. This should not have happened with TrueCharts or with any App, using default settings.

This could actually be both Kubernetes and DNS Server. An easy way to test is to make sure you have a search suffix domain specified on your computer then use nslookup:

nslookup example.com
nslookup example.com.
nslookup www.example.com
nslookup www.example.com.

And check that all give you the correct IP and FQDN with search domain NOT appended. If search domain gets appended to test 1 and 3 or the IP is wrong, then your DNS Server is misconfigured.

This usually occurs when when dns is configured with one or more search domains, but most is mostly just a nuisance unless your root domain has a public A or CNAME entry.

FYI,
You can also workaround this on SCALE by not adding a search domain in the NAS general network settings and/ or using public/external DNS servers.

 

[NightShaman]

This is not a failure in your DNS server, it's a failure in SCALE's integrated kubernetes DNS server, please file a bugreport at iX JIRA, as requested by us earlier, to make sure this doesn't happen again (to you and other users) and be sure to attach a debuglog from your TrueNAS SCALE Machine. This should not have happened with TrueCharts or with any App, using default settings.

I did.
[NAS-112192] ClusterFirst DNS not connecting to external addresses. - iX - Bug Tracker (Jira) (ixsystems.com)

 

[NightShaman]

Also tried suggestions above, and nothing worked.

Since it is easy for me to rebuild everything, I did a fresh install of 21.08
Used to have 2 bonded networks only set up one.
removed the old xi-applications dataset so it would recreate it.
still the same

 

[NightShaman]

Sooo I changed the search domain from my reg domain (which is a public domain) to localdomain and everything works. Thanks @slm4996

 

[appliance]

thanks for this I can run Plex finally. I have TrueNAS-SCALE-22.02.1, and adding ndots was the fix for Plex chart.