Sometimes it’s DNSvmk8s.guru, October 29, 2024
https://vmk8s.guru/2024/10/29/sometimes-its-dns/
other times it’s not…
As mentioned in the article here, this post is about a weird issue we experienced. The issue was that we would randomly have (what appeared to be) a DNS resolution issue. We would have resolution issues for between 10 and 15 minutes, and it would resolve without intervention.
Some of the errors we got during this whole process are as follows:
application.example.domain.com: No address associated with the hostname
application.example.domain.com: 500 Internal server errorThe above were normally together, and then we got these also.
Failed to resolve 'application.example.domain.com.' [A(1)] after 2 queriesWe also got NXDOMAIN errors.
[INFO] IP:42889 - 7796 "A IN application.example.domain.com. udp 72 false 512" NXDOMAIN qr,aa,rd 165 0.000130923sLet’s dive deeper into this.
The setup for the original issue.
- You are running 2 EKS clusters in the same aws account.
- Both clusters share the same Private Hosted Zone in Route53 (example.domain.com).
- Route53 Resolver Rules are pushed to the account (and all aws accounts for the Org) from a networking aws account.
- You use DHCP Options to push down nameservers to your hosts. The
/etc/resolv.conffile looks something like the below.
sh-4.2$ cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search internaldomain.com af-south-1.compute.internal
options timeout:2 attempts:5
nameserver 100.100.100.2
nameserver 10.10.10.10
nameserver 10.10.10.11- An api-gateway service in cluster-1 would make a call to an api-gateway service in cluster-2 via the ingress application.example.domain.com. This would then route calls to the relevant services in cluster-2.
The issue that we experienced was that randomly the calls to application.example.domain.com in cluster-2 would just start failing. This would last for 10-15 minutes, and then resolve. The issue would happen during a quieter time of the day, so it wasn’t related to volume/load.
Initially we thought the issue could be due to coreDNS having issues. Both cluster-1 and cluster-2 were very busy clusters, with quite a few nodes in them. On investigating, we could not see coreDNS usage out of the norm. We could also see that all internal calls in both clusters were working as expected. At the same time we checked to see if we were maybe hitting some ENI limits on the EC2 instances where coreDNS was running, but we were not. As a precaution we increased the number of coreDNS pods on both clusters. This was to just reduce the impact on the EC2 ENI’s, and also see if it had any impact on the problem. i.e. Does it move from a complete failure on resolution, to intermittent failure on resolution. None of this resolved the problem.
Because cluster-1 was only making 1 external call, which was to cluster-2 we decided to modify some settings in coreDNS, to try and help resolve/work around the issue, and see if it could potentially isolate where the issue was happening. These are the changes we made (note, this is not the whole file, but just the snippets/plugins that were changed).
cache 300 {
success 50000
denial 25000
}The above was done to see if this would reduce the outage time by increasing the caching. This made no difference. i.e. The outage was still happening randomly, and still for 10-15 minutes. This was done on both clusters.
For cluster-1 we then put in an entry for the specific URL that we called in cluster-2.
application.example.domain.com {
errors
log
cache 900
forward . 100.100.100.2
}The reason we did this to force the lookup to use the VPC DNS for the aws account as we suspected that it was maybe routing a different way. This also didn’t seem to resolve the issue. i.e. The outage was still happening randomly, and still for 10-15 minutes.
At this point the application team, also added DNS caching at the java level. Unfortunately, this also did not resolve the issue.
This was proving to be a very interesting issue, and we started look deeper at our aws account setup, and specifically at Route53, Active Directory and DHCP. The focus was on Route53 and DHCP as we were still not convinced that everything was resolving using the VPC DNS in the account, even with the change we put in the corefile for application.example.domain.com.
As part of our account setup, we have a DHCP options set that we push down to each account. It adds the following nameservers to each server.
nameserver 10.10.10.10
nameserver 10.10.10.11
They are added to the resolv.conf file, so we end up with 3 (as shown at the top of the page) as the VPC DNS nameserver is in the file also.
The decision was made to remove the DHCP option set that we push down to the account. This would then force everything in the account to only use the VPC DNS, as it would be the only entry in the resolv.conf file.
Just a note here, as mentioned in the article here, we could have forced the traffic to use the VPC dns by updating the forward plugin policy in the corefile. This however may not have made a difference to the below section.
While investigating the issue a bit more, and whilst on a call with aws support, we noticed that we didn’t have a System Resolver Rule in Route53 for example.domain.com. We only had a Forward rule for domain.com. We then created the System Resolver Rule for example.domain.com. The reason for the System Resolver Rule is to keep all resolution for example.domain.com in the Private Hosted Zone for this account, rather than forwarding it to the DNS Resolvers in the Network account where domain.com is hosted.
It was at about the same time we added the above System Resolver Rule that we made the decision to get rid of cluster-2, and add a nodegroup to cluster-1 where we could then run all the services from cluster-2. This would then keep all the traffic internal to the cluster, and would ultimately be more efficient.
With making the last 2 changes, the issue did go away, but we couldn’t be sure if the System Resolver Rule would have resolved the issue or not.
Now we were running with just cluster-1, and things were good. No more random issues… Until, we had to start calling another service in another cluster, in a different aws account. After a while, we encountered the same random resolution failure to this new service. The issue was a little different though, in that it appeared as though other accounts/services could still resolve this new service. In the original issue, there was only 1 account/service making calls to application.example.domain.com.
After the 2nd time this issue happened, we added the fix for the corefile forward plugin. As shown below (this was done on all EKS clusters).
forward . /etc/resolv.conf {
policy sequential
}Since making the above change, we haven’t experienced the issue again. Not to say this is the fix, as there could have been some other application changes (or even library changes in a dependency) that were done which could have resolved the issue. Another thing we decided to do was to make sure that all our accounts are setup the same from a DNS, DHCP, Route53 perspective. Having the accounts setup in different ways could also play a role in having some weird/interesting issues.
While investigating this issue, we also played around with the ndots settings. We did this at an application level rather than cluster wide, just so we didn’t risk potentially breaking some unknown calls that would break the boundaries of the ndots values we went with.
For those of you that are not sure what the ndots setting does, let me try explain. The value of the ndots setting is to tell your lookup at which point to consider a lookup as a FQDN. i.e. If your ndots value is 5 (the default for k8s), then if you search for www.exampleaddress.com the lookup will go through each search domain you have (including the kubernetes search domains), before it finally resolves. See extract from coreDNS logs.
[INFO] IP:42889 - 7796 "A IN www.exampleaddress.com.svc.cluster.local. udp 72 false 512" NXDOMAIN qr,aa,rd 165 0.000130923s
[INFO] IP:50539 - 20153 "A IN www.exampleaddress.com.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000101822s
[INFO] IP:36554 - 26889 "A IN www.exampleaddress.com.searchdomain.com. udp 74 false 512" NXDOMAIN qr,rd,ra 189 0.002975576s
[INFO] IP:60848 - 63513 "A IN www.exampleaddress.com. udp 54 false 512" NOERROR qr,rd,ra 163 0.000408049sIf you however change your ndots value to 2 for your deployment, then this will assume that anything with 2 or more dots in the lookup is an FQDN, and it will ignore all the search domains. So if you do a lookup for www.exampleaddress.com you will just see the successful call in your coreDNS logs.
This does however mean that you need to make sure that you always use an FQDN in your lookup, because if your lookup is something like www.prod.exampleaddress.subdomain your lookup will fail as this will not search through your search domains as it expects www.prod.exampleaddress.subdomain to be an FQDN that resolves.
We played around with a few options on the ndots, but in the end we decided to go with specifying the FQDN with a . at the end of the lookup. This is in fact the only time a lookup is considered a FQDN, and it your ndots is 5, but you specify a . at the end of your lookup, it will consider it an FQDN and ignore the search domains. So we do all our lookups within applications using an FQDN like “www.exampleaddress.com.”.
The benefits of doing this, or even changing your ndots for your deployment to something lower than 5, is the amount of work that coreDNS needs to do.
You should even do this when calling services within your cluster. So use “service.namespace.svc.cluster.local.”.
Just some other information that came out while still digging into this after all of the above changes after we had a similar issue from an on-prem server to a cluster in the cloud.
Our DNS setup was not optimal. We had some strange forwarding rules in place, and other rules were missing servers in the configs. With that in mind, this was very likely the cause of our strange issues. Below is an image with some descriptions of the worst possible flow of a DNS request for a POD running in a cluster, calling a POD running in another cluster in the same account. You will notice that the flow goes from 1 AWS account to another AWS account to On-Prem back to the 2nd AWS account only then to resolve and send the resolution to the 1st AWS account.
Note: Open images in a new tab to see full size.
After all the changes we made the flow is now like the below image. The flow of the DNS request does not leave the account at all.
