AWX unable to resolve local domain servers: Customizing Kubernetes k3s CoreDNS

In my last post, I created a container group for linking AWX with my domain Kerberos for authentication against Windows hosts. It turned out my AWX POD was unable to lookup any of my Windows domain servers. Simple testing showed it could reach the host on the right port.

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup hosta.contoso.com

Obviously instead of hosta.contoso.com I was using an actual host in my actual domain. I thought my next step was to create another Container group linking my Linux hosts’ /etc/resolve file with my Execution environment, but that would not work. After some googling I saw some other were having similar issues and they resolved by updating Kubernetes CoreDNS to forward all queries for my local domain to one of my local domain DNS servers.

To play it safe, I copied my existing CoreDNS configuration by running the following:

kubectl -n kube-system get configmap coredns -o yaml

I saved the output of that to a file called coredns-custom.yml and added a forwarder section for my internal domain.

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    contoso.com:53 {
       errors
       cache 30
       forward . 10.5.1.53
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
    10.5.1.8 localhost.localdomain
kind: ConfigMap
metadata:
  annotations:
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4yQwWrzMBCEX0Xs2fEf20nsX9BDybH02lMva2kdq1Z2g6SkBJN3L8IUCiVtbyNGOzvfzoAn90IhOmHQcKmgAIsJQc+wl0CD8wQaSr1t1PzKSilFIUiIix4JfRoXHQjtdZHTuafAlCgq488xUSi9wK2AybEFDXvhwR2e8QQFHCnh50ZkloTJCcf8lP6NTIqUyuCkNJiSp9LJP5czoLjryztTWB0uE2iYmvjFuVSFenJsHx6tFf41gvGY6Y0Eshz/9D2e0OSZfIJVvMZExwzusSf/I9SIcQQNvaG6a+r/XVdV7abBddPtsN9W66Eedi0N7aberM22zaHf6t0tcPsIAAD//8Ix+PfoAQAA
    objectset.rio.cattle.io/id: ""
    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
    objectset.rio.cattle.io/owner-name: coredns
    objectset.rio.cattle.io/owner-namespace: kube-system
  creationTimestamp: "2023-01-24T18:28:23Z"
  labels:
    objectset.rio.cattle.io/hash: bce283298811743a0386ab510f2f67ef74240c57
  name: coredns
  namespace: kube-system

Now you can apply the new forwarder to Kubernetes CoreDNS with the follwoing command:

kubectl apply -f coredns-custom.yml

#You can test it applied and worked by running:

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup hosta.contoso.com

Now my Kubernetes DNS was resolving as expected and in turn so was AWX!