apiserver pod is not able to resolve internal DNS: Name or service not known

[viovanov]

Run minikube, then create a pod (using apiserver's image), then run:

kubectl exec -it test-pod -- sh -c "ping kubernetes"

The output correctly says:

PING kubernetes.default.svc.cluster.local (10.96.0.1) 56(84) bytes of data.

When trying to do the same thing on the api server:

kubectl exec -it kube-apiserver-minikube --namespace kube-system -- /bin/sh -c "ping kubernetes"

We get:

ping: kubernetes: Name or service not known

This happens because the resolv.conf in the apiserver pod is not pointing to the DNS service.
Why is this the case? Admission controllers (webhook servers) don't work if the apiserver can't resolve services.

[afbjorklund]

This is a known issue where the resolv.conf only contains the external DNS, not the internal DNS.

In my cause the default one is 10.0.2.3 (for eth0 host 10.0.2.15), while internal one is 10.96.0.10

It also does not have a search domain:

$ nslookup kubernetes
Server:    10.0.2.3
Address 1: 10.0.2.3

nslookup: can't resolve 'kubernetes'
$ nslookup kubernetes 10.96.0.10
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes'
$ nslookup kubernetes.default.svc.cluster.local 10.96.0.10
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

And doesn't know its own IP address.

$ nslookup minikube
Server:    10.0.2.3
Address 1: 10.0.2.3

Name:      minikube
Address 1: 127.0.1.1 minikube

It could have chosen eth0 (10.0.2.15) or eth1 (192.168.99.100),
but decided to go for lo (127.0.1.1) when it came to /etc/hosts.

The API server just inherits these files, so the same root cause.

[josh-ferrell-sas]

I wanted to add on to this. Creating an AuditSink with a webhook url set to a service FQDN(i.e. twistlock.kube-system.svc.cluster.local) will fail and present in the kubernetes apiserver logs.

E0520 13:30:02.470263 1 metrics.go:96] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: Post [urladdress] dial tcp: lookup twistlock-console on 10.19.1.23:53: no such host

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale