CoreDNS defaults to a random routing policy for upstream nameservers rather than the sequential order typically used by standard operating systems. Configured upstream targets in the forward plugin receive queries at random instead of acting as primary and secondary failovers, which can break split horizon DNS setups requiring specific internal nameserver priority.
CoreDNS Forward Policy Guide to Fix Upstream Nameserversvmk8s.guru, February 21, 2024
https://vmk8s.guru/2024/02/21/coredns-forward-policy-guide-to-fix-upstream-nameservers/
Have you ever wondered how CoreDNS chooses an upstream nameserver to do DNS lookups against? I never did, until recently that is. I (and many others) just assumed this would work like most OSes work. This is not the case though.
Let’s take a look at this a bit more.
The setup.
- You are running an EKS cluster in AWS.
- There is an externally hosted
example.comdomain that is publicly available on Route53. - You have a VPC endpoint shared with you from another AWS account for the externally hosted
example.comdomain. - You are not allowed to hit the external
example.comdomain, so need to go via an internally hosted (in a Private Hosted Zone)example.comrecord. This points to the shared VPC endpoint. - You use DHCP Options to push down nameservers to your hosts. The
/etc/resolv.conffile looks something like the below.
sh-4.2$ cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search internaldomain.com af-south-1.compute.internal
options timeout:2 attempts:5
nameserver 100.100.100.2
nameserver 10.10.10.10
nameserver 10.10.10.11In the above code snippet the first nameserver (100.100.100.2) is the VPC DNS nameserver. i.e. This should be internal to your VPC, and should always resolve to the internally hosted example.com entry. The reason for this is because (Linux in this case) uses a sequential policy to choose the nameserver (from /etc/resolv.conf) to perform the lookups against. i.e. It will always use the first one in the list, unless that one has a problem.
When we were testing from the EKS node, we always hit the correct example.com, but our containers running on the EKS cluster would sometimes resolve the public example.com, and sometimes our internal example.com. This was very weird to say the least.
We know that CoreDNS will forward to the hosts /etc/resolv.conf file based on the following in the Corefile.
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
ready :8181
forward . /etc/resolv.conf
cache 30
loop
log
reload
loadbalance
}So one would think that this would also always use the first entry in the /etc/resolv.conf of the host. One would be wrong though.
The default policy for the forward CoreDNS plugin is random.
From the plugin page. Found at https://coredns.io/plugins/forward/
policyspecifies the policy to use for selecting upstream servers. The default israndom.
randomis a policy that implements random upstream selection.round_robinis a policy that selects hosts based on round robin ordering.sequentialis a policy that selects hosts based on sequential ordering.”
This means that CoreDNS will randomly choose 1 of the nameservers in the hosts /etc/resolv.conf file to use for the lookup of a specific query. This could be the same one for a few queries, or it could be a different one for each query. This explained the randomness of our results from our containers.
Luckily, there is a fix for this (as seen above). You can specify the policy as sequential. The default is as below.
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
ready :8181
forward . /etc/resolv.conf
cache 30
loop
log
reload
loadbalance
}What we changed it to.
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf {
policy sequential
}
cache 30
loop
reload
loadbalance
}Once we made the above change, and CoreDNS reloaded with the new config we now always hit the internally hosted example.com entry.
Interestingly enough the above fix would have solved, or at least helped solve another issue we had. I’ll write up a post about that soon.
Why CoreDNS uses random as the default I cannot answer. I also don’t know why AWS themselves don’t change the default for the CoreDNS they install as part of an EKS cluster.
One could also remove the additional nameservers from /etc/resolve.conf but some people may have a reason to have these in there.